eq-layer 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- eq_layer-0.1.0/.gitignore +8 -0
- eq_layer-0.1.0/LICENSE +21 -0
- eq_layer-0.1.0/PKG-INFO +527 -0
- eq_layer-0.1.0/README.md +496 -0
- eq_layer-0.1.0/THIRD_PARTY_DATA.md +130 -0
- eq_layer-0.1.0/eq_layer/__init__.py +68 -0
- eq_layer-0.1.0/eq_layer/ab_eval.py +302 -0
- eq_layer-0.1.0/eq_layer/actions.py +242 -0
- eq_layer-0.1.0/eq_layer/affect.py +227 -0
- eq_layer-0.1.0/eq_layer/auditor.py +321 -0
- eq_layer-0.1.0/eq_layer/cli.py +174 -0
- eq_layer-0.1.0/eq_layer/coarse_discourse.py +107 -0
- eq_layer-0.1.0/eq_layer/data/intent_train.jsonl +299 -0
- eq_layer-0.1.0/eq_layer/dbdc3.py +197 -0
- eq_layer-0.1.0/eq_layer/dialogbank_iso.py +272 -0
- eq_layer-0.1.0/eq_layer/dialogue_reference.py +393 -0
- eq_layer-0.1.0/eq_layer/intent.py +199 -0
- eq_layer-0.1.0/eq_layer/intent_belief.py +276 -0
- eq_layer-0.1.0/eq_layer/interaction.py +314 -0
- eq_layer-0.1.0/eq_layer/policies.py +295 -0
- eq_layer-0.1.0/eq_layer/preferences.py +125 -0
- eq_layer-0.1.0/eq_layer/response_experiment.py +606 -0
- eq_layer-0.1.0/eq_layer/steer.py +281 -0
- eq_layer-0.1.0/eq_layer/tracker.py +111 -0
- eq_layer-0.1.0/eq_layer/trained_affect.py +246 -0
- eq_layer-0.1.0/eq_layer/trained_breakdown.py +143 -0
- eq_layer-0.1.0/eq_layer/trained_disagreement.py +148 -0
- eq_layer-0.1.0/eq_layer/trained_intent.py +291 -0
- eq_layer-0.1.0/eq_layer/trained_subtext.py +436 -0
- eq_layer-0.1.0/pyproject.toml +70 -0
eq_layer-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 eq-layer contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
eq_layer-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,527 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: eq-layer
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Inspectable intent, affect, repair, uncertainty, and factored-action control for LLM responses
|
|
5
|
+
Project-URL: Homepage, https://github.com/Furox-Art/eq-layer
|
|
6
|
+
Project-URL: Repository, https://github.com/Furox-Art/eq-layer
|
|
7
|
+
Project-URL: Issues, https://github.com/Furox-Art/eq-layer/issues
|
|
8
|
+
Author: Furox-Art
|
|
9
|
+
License: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: affective-computing,agent-skills,agents,dialogue-management,intent-classification,llm,llm-steering
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Intended Audience :: Science/Research
|
|
15
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
21
|
+
Requires-Python: >=3.10
|
|
22
|
+
Provides-Extra: dev
|
|
23
|
+
Requires-Dist: build<2,>=1.2; extra == 'dev'
|
|
24
|
+
Requires-Dist: pytest; extra == 'dev'
|
|
25
|
+
Requires-Dist: ruff; extra == 'dev'
|
|
26
|
+
Requires-Dist: scikit-learn<2,>=1.5; extra == 'dev'
|
|
27
|
+
Requires-Dist: twine<7,>=6; extra == 'dev'
|
|
28
|
+
Provides-Extra: ml
|
|
29
|
+
Requires-Dist: scikit-learn<2,>=1.5; extra == 'ml'
|
|
30
|
+
Description-Content-Type: text/markdown
|
|
31
|
+
|
|
32
|
+
# eq-layer
|
|
33
|
+
|
|
34
|
+
A control layer that gives an LLM a *decision* about how to speak before it speaks — instead of asking it to be more empathetic in a prompt.
|
|
35
|
+
|
|
36
|
+
## Why this exists
|
|
37
|
+
|
|
38
|
+
Asking a model to be more empathetic is style transfer. It produces *more* of the same flat warmth, which reads as uncanny rather than attentive. The failure modes are structural, not knowledge gaps:
|
|
39
|
+
|
|
40
|
+
- **Flat affect.** "That sounds really tough!" arrives at the same register regardless of what was said.
|
|
41
|
+
- **Escalation blindness.** The user is frustrated, then angry, then threatening. The model holds one tone for all three turns.
|
|
42
|
+
- **Validation without motion.** The feeling is acknowledged; the next step never comes.
|
|
43
|
+
- **Repair collapse.** After a correction the model either capitulates entirely or starts defending itself. There is no middle.
|
|
44
|
+
|
|
45
|
+
So this repo does not retrain the model. It adds a layer:
|
|
46
|
+
|
|
47
|
+
1. **Affect state** — `{valence, arousal, escalation_delta, stance, subtext}` as an explicit intermediate representation, tracked across turns. `escalation_delta` is the point: snapshot classification cannot see a trend.
|
|
48
|
+
2. **Intent belief state** — the learned classifier preserves the full posterior over `action_request/status_check/explanation/question/statement/unknown` instead of collapsing uncertainty to one confidence number. A transparent one-step Bayes-risk router chooses whether to act or clarify from that distribution. This is POMDP-inspired uncertainty handling, not a full POMDP.
|
|
49
|
+
3. **Conversation health state** — `repair` identifies observable user→assistant misalignment and its target; `interaction_quality` tracks structural conversation failure separately from emotion using current score, delta, repeated failures, unresolved repair, and clarification load.
|
|
50
|
+
4. **Policy selection** — affect and intent jointly choose a response-policy family (`mirror`, `direct`, `repair`, `hold`, `boundary`, ...). Selection is a discrete decision, not generation.
|
|
51
|
+
5. **Factored action compilation** — the selected policy is split into orthogonal controls: `task_move`, `social_move`, `repair_move`, and `realization`. The task is driven primarily by user intent, so affective adaptation cannot silently replace what the user actually asked for.
|
|
52
|
+
6. **Steering** — the factored action, source policy, repair state, interaction quality, and canonical request are injected into the decode path so the control decision actually lands in the tokens.
|
|
53
|
+
7. **Structural response audit** — the generated response is checked against inspectable control contracts such as question budget, required clarification, unsupported privacy promises, unknown-stance admissions, repair repetition, and low-verbosity overruns. This is not a factual truth checker.
|
|
54
|
+
|
|
55
|
+
## Prior art and novelty boundary
|
|
56
|
+
|
|
57
|
+
EQ-Layer does **not** claim that affect-aware dialogue policy, user modelling,
|
|
58
|
+
repair, intent routing, or adaptive dialogue management are new. Those ideas
|
|
59
|
+
have substantial pre-LLM precedent in TRAINS/grounding work, adaptive TOOT,
|
|
60
|
+
RavenClaw, user-tailored generation, affect-sensitive AutoTutor, affective
|
|
61
|
+
POMDP dialogue management, and related systems.
|
|
62
|
+
|
|
63
|
+
The defensible research question is narrower: whether an explicit,
|
|
64
|
+
model-agnostic control layer can improve a **fixed general-purpose language
|
|
65
|
+
model** by separating intent, affect, conversational evidence, policy selection,
|
|
66
|
+
and generation steering while keeping those intermediate decisions inspectable.
|
|
67
|
+
|
|
68
|
+
See `docs/PRIOR_ART_AND_NOVELTY.md` for the historical map, claims to avoid,
|
|
69
|
+
and architecture lessons carried forward into this project.
|
|
70
|
+
|
|
71
|
+
## Status
|
|
72
|
+
|
|
73
|
+
Early. The harness runs; the numbers are development signals, not a real-world EQ benchmark.
|
|
74
|
+
|
|
75
|
+
There is no standard benchmark for EQ. `eval/` is a first attempt at one, not a finished measurement. If you improve the scorer, that contribution matters as much as adding a policy.
|
|
76
|
+
|
|
77
|
+
## Agent Skills: Codex, Claude, Copilot, Cursor, Antigravity, and Grok
|
|
78
|
+
|
|
79
|
+
EQ-Layer is packaged as a portable Agent Skill in `skills/eq-layer/` and as an
|
|
80
|
+
Agent Plugins 1.0 package via `plugin.json`. Repository discovery shims are
|
|
81
|
+
included for the major agent ecosystems:
|
|
82
|
+
|
|
83
|
+
| Agent | Repository path |
|
|
84
|
+
| --- | --- |
|
|
85
|
+
| OpenAI Codex | `.codex/skills/eq-layer/` |
|
|
86
|
+
| Anthropic Claude Code | `.claude/skills/eq-layer/` |
|
|
87
|
+
| GitHub Copilot | `.github/skills/eq-layer/` |
|
|
88
|
+
| Cursor | `.cursor/skills/eq-layer/` |
|
|
89
|
+
| Google Antigravity | `.agents/skills/eq-layer/` |
|
|
90
|
+
| xAI Grok Build | `.grok/skills/eq-layer/` |
|
|
91
|
+
|
|
92
|
+
For a user-level install across all supported discovery roots:
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
python tools/install_agent_skills.py --target all
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
The skill includes a local routing script and reports whether it used the
|
|
99
|
+
lightweight portable path or the full learned research pipeline.
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
printf '%s' '[{"role":"user","content":"devam et"}]' \
|
|
103
|
+
| python skills/eq-layer/scripts/route.py --pretty
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
See `skills/eq-layer/references/VENDOR_COMPATIBILITY.md` for verified discovery
|
|
107
|
+
paths and source documentation.
|
|
108
|
+
|
|
109
|
+
## Quick start
|
|
110
|
+
|
|
111
|
+
Install the learned intent tracker:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
pip install "eq-layer[ml]"
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
```python
|
|
118
|
+
from eq_layer import ConversationTracker, Selector, Steer, TrainedAffect, TrainedIntent, TrainedSubtext, compose_action
|
|
119
|
+
|
|
120
|
+
messages = [
|
|
121
|
+
{"role": "user", "content": "Randevumu üç kez değiştirdiler"},
|
|
122
|
+
]
|
|
123
|
+
affect = TrainedAffect.load("artifacts/emobank_affect.joblib")
|
|
124
|
+
subtext = TrainedSubtext.load("artifacts/xdailydialog_subtext.joblib")
|
|
125
|
+
tracker = ConversationTracker(affect=affect, subtext=subtext)
|
|
126
|
+
tracked = tracker.infer(messages)
|
|
127
|
+
state = tracked.state
|
|
128
|
+
intent = TrainedIntent.from_bundled().infer(messages)
|
|
129
|
+
selection = Selector().select(state, intent)
|
|
130
|
+
action = compose_action(
|
|
131
|
+
selection.policy,
|
|
132
|
+
intent,
|
|
133
|
+
repair=tracked.repair,
|
|
134
|
+
interaction_quality=tracked.interaction_quality,
|
|
135
|
+
)
|
|
136
|
+
print(selection.policy.name, action.to_dict())
|
|
137
|
+
|
|
138
|
+
prompt = Steer.build(
|
|
139
|
+
selection.policy,
|
|
140
|
+
intent,
|
|
141
|
+
repair=tracked.repair,
|
|
142
|
+
interaction_quality=tracked.interaction_quality,
|
|
143
|
+
).apply_to_prompt(messages[-1]["content"])
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
Run the seeded cases. Exits non-zero on any mismatch or any policy that no
|
|
147
|
+
case can reach:
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
python eval/run.py
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
Run the intent benchmark separately:
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
python eval/intent_eval.py
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
It compares the learned adapter with the heuristic fallback and reports
|
|
160
|
+
five-fold cross-validation on the bundled corpus.
|
|
161
|
+
|
|
162
|
+
Train the affect regressor from the pinned official EmoBank release:
|
|
163
|
+
|
|
164
|
+
```bash
|
|
165
|
+
python tools/train_emobank_affect.py --output artifacts/emobank_affect.joblib
|
|
166
|
+
python eval/emobank_affect_eval.py
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
EmoBank is not bundled into this repository. The training/evaluation path uses
|
|
170
|
+
the upstream train/dev/test split and records its pinned commit and
|
|
171
|
+
CC-BY-SA-4.0 provenance in `THIRD_PARTY_DATA.md`.
|
|
172
|
+
|
|
173
|
+
Train the learned dialogue-signal tracker from pinned XDailyDialog files:
|
|
174
|
+
|
|
175
|
+
```bash
|
|
176
|
+
python tools/train_xdailydialog_subtext.py --output artifacts/xdailydialog_subtext.joblib
|
|
177
|
+
python eval/xdailydialog_subtext_eval.py
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
XDailyDialog provides supervised dialogue-act and basic-emotion labels. EQ-Layer
|
|
181
|
+
does **not** pretend that `challenge`, `demand`, or `exhaustion` are direct
|
|
182
|
+
upstream labels: those are conservative compositions of learned dialogue
|
|
183
|
+
signals with learned V/A state. See `THIRD_PARTY_DATA.md` for the license-chain
|
|
184
|
+
caution on the English data.
|
|
185
|
+
|
|
186
|
+
## Intent is a control signal, not mind-reading
|
|
187
|
+
|
|
188
|
+
The default intent path is now learned rather than keyword-selected.
|
|
189
|
+
`TrainedIntent` fits a balanced logistic-regression classifier over combined
|
|
190
|
+
word and character TF-IDF features. The bundled corpus currently contains
|
|
191
|
+
Turkish and English examples for `action_request`, `status_check`,
|
|
192
|
+
`explanation`, `question`, `statement`, and `unknown`.
|
|
193
|
+
|
|
194
|
+
The classifier emits a full posterior distribution, exposed as
|
|
195
|
+
`IntentBelief`. Production routing no longer discards that distribution behind
|
|
196
|
+
a single confidence/margin threshold. `decide_intent_action` computes the
|
|
197
|
+
expected interaction loss of each response move and clarification, then chooses
|
|
198
|
+
the minimum-risk action. For example, ambiguity between a status check and a
|
|
199
|
+
generic question can still be answered because the semantic mismatch cost is
|
|
200
|
+
small, while ambiguity between executing an action and merely responding to a
|
|
201
|
+
statement can trigger clarification.
|
|
202
|
+
|
|
203
|
+
The current loss matrix is **hand-specified and transparent, not learned or
|
|
204
|
+
claimed optimal**. It encodes relative interaction costs. EQ-Layer therefore
|
|
205
|
+
audits routing sensitivity rather than treating one matrix as ground truth:
|
|
206
|
+
`eval/intent_risk_sensitivity.py` sweeps clarification cost and performs
|
|
207
|
+
one-at-a-time ±20% perturbations of every non-zero loss cell, reporting action
|
|
208
|
+
flips overall and by source stratum. This is a local robustness diagnostic, not
|
|
209
|
+
loss-matrix optimization, and it must not be tuned against blinded human
|
|
210
|
+
preference outcomes. The old confidence and margin fields remain serialized for
|
|
211
|
+
backwards compatibility and diagnostics; they no longer choose the learned
|
|
212
|
+
intent route.
|
|
213
|
+
|
|
214
|
+
### Intent-risk sensitivity audit
|
|
215
|
+
|
|
216
|
+
The production control path has also been run over all 120 external held-out
|
|
217
|
+
cases without generating LLM responses. Under one-at-a-time ±20% perturbations
|
|
218
|
+
of every non-zero loss cell, 7,332 / 7,440 routing decisions were unchanged
|
|
219
|
+
(**98.55% local decision stability**).
|
|
220
|
+
|
|
221
|
+
That result does **not** validate the loss matrix. Scaling the entire
|
|
222
|
+
clarification-cost row changes routing substantially:
|
|
223
|
+
|
|
224
|
+
- production clarification rate: **45.0%**
|
|
225
|
+
- clarification cost ×0.75: **69.17%**
|
|
226
|
+
- clarification cost ×1.25: **19.17%**
|
|
227
|
+
- clarification cost ×1.50: **9.17%**
|
|
228
|
+
|
|
229
|
+
The `task_repair` stratum is especially uncertain: 26 / 30 cases route to
|
|
230
|
+
clarification (**86.67%**) with mean fused intent entropy 0.8954. This may
|
|
231
|
+
reflect genuine ambiguity in repair turns, but it is not used to tune the
|
|
232
|
+
matrix. The current loss table remains hand-specified and uncalibrated.
|
|
233
|
+
|
|
234
|
+
Machine-readable audit:
|
|
235
|
+
`eval/results/intent_risk_heldout_120_audit.json`.
|
|
236
|
+
|
|
237
|
+
Explicit constraints such as `brief`, `scope_limited`, and `no_guess`
|
|
238
|
+
remain deterministic because they are control requirements, not semantic
|
|
239
|
+
labels.
|
|
240
|
+
|
|
241
|
+
`HeuristicIntent` remains available as a zero-dependency fallback. It does not
|
|
242
|
+
manufacture a learned posterior and is no longer the adapter used by the main
|
|
243
|
+
evaluation harness.
|
|
244
|
+
|
|
245
|
+
Affect can still override intent during sustained escalation. A clear action
|
|
246
|
+
request should drive the response in a calm turn; it should not erase a
|
|
247
|
+
multi-turn escalation signal.
|
|
248
|
+
|
|
249
|
+
The bundled data is small and synthetic. `eval/intent_eval.py` therefore
|
|
250
|
+
reports both a fixed held-out score and five-fold cross-validation; neither is
|
|
251
|
+
presented as evidence of production-level semantic understanding.
|
|
252
|
+
|
|
253
|
+
## Verified development baselines
|
|
254
|
+
|
|
255
|
+
The current CI-verified EmoBank regression baseline uses the official upstream
|
|
256
|
+
train/dev/test split (8062 / 1000 / 1000). On the untouched test split:
|
|
257
|
+
|
|
258
|
+
| dimension | MAE | train-mean baseline MAE | Spearman rho |
|
|
259
|
+
| --- | ---: | ---: | ---: |
|
|
260
|
+
| Valence | 0.2125 | 0.2449 | 0.5460 |
|
|
261
|
+
| Arousal | 0.1782 | 0.1904 | 0.3120 |
|
|
262
|
+
| Dominance | 0.1494 | 0.1563 | 0.2573 |
|
|
263
|
+
|
|
264
|
+
The regressor beats the train-mean baseline on MAE for all three dimensions,
|
|
265
|
+
but Arousal and Dominance rank correlations are still modest. The machine-readable
|
|
266
|
+
record is `eval/results/emobank_affect_baseline.json`.
|
|
267
|
+
|
|
268
|
+
### Dialogue-signal baseline
|
|
269
|
+
|
|
270
|
+
The learned dialogue-signal tracker was verified on the pinned XDailyDialog
|
|
271
|
+
English train/dev/test files (83035 / 8025 / 7716 utterances). On the untouched
|
|
272
|
+
test file:
|
|
273
|
+
|
|
274
|
+
| task | accuracy | macro-F1 | majority accuracy |
|
|
275
|
+
| --- | ---: | ---: | ---: |
|
|
276
|
+
| dialogue act | 0.8025 | 0.7274 | 0.4557 |
|
|
277
|
+
| basic emotion | 0.8249 | 0.4310 | 0.8166 |
|
|
278
|
+
|
|
279
|
+
The dialogue-act signal is substantially above the majority baseline. The
|
|
280
|
+
emotion signal is much weaker: macro-F1 is only 0.431 and accuracy is only
|
|
281
|
+
slightly above the majority baseline on test (and below it on dev). EQ-Layer
|
|
282
|
+
therefore treats emotion as supporting evidence rather than standalone proof of
|
|
283
|
+
subtext. The machine-readable result is
|
|
284
|
+
`eval/results/xdailydialog_signal_baseline.json`.
|
|
285
|
+
|
|
286
|
+
### Higher-level subtext audits
|
|
287
|
+
|
|
288
|
+
Direct higher-level supervision was investigated rather than assumed:
|
|
289
|
+
|
|
290
|
+
| source / target | verified result | routing decision |
|
|
291
|
+
| --- | --- | --- |
|
|
292
|
+
| DialogBank correction | 4 direct examples | **NO-GO** — insufficient supervision |
|
|
293
|
+
| DialogBank disagreement | 1 direct example | **NO-GO** — insufficient supervision |
|
|
294
|
+
| Coarse Discourse disagreement | test ROC-AUC 0.8111, AP 0.1420, F1 0.2278 | **NO-GO** — ranking signal exists, but precision/recall are insufficient for `challenge` routing |
|
|
295
|
+
| DBDC3 hard breakdown | test ROC-AUC 0.5057, AP 0.2555 | **NO-GO** — effectively chance-level text-only generalisation |
|
|
296
|
+
|
|
297
|
+
These results are preserved rather than optimized away. The corresponding
|
|
298
|
+
machine-readable records are
|
|
299
|
+
`eval/results/dialogbank_label_inventory.json`,
|
|
300
|
+
`eval/results/coarse_disagreement_no_go.json`, and
|
|
301
|
+
`eval/results/dbdc3_breakdown_no_go.json`.
|
|
302
|
+
|
|
303
|
+
|
|
304
|
+
## Evidence provenance
|
|
305
|
+
|
|
306
|
+
A subtext label is not enough by itself. `SubtextDecision.evidence_level`
|
|
307
|
+
records how the label was obtained:
|
|
308
|
+
|
|
309
|
+
- `verified` — caller/external annotation supplied the state.
|
|
310
|
+
- `direct_learned` — the upstream training target directly matches the emitted label.
|
|
311
|
+
- `derived` — multiple learned signals are composed into a higher-level control label.
|
|
312
|
+
- `structural` — explicit conversational form or marker triggered the label.
|
|
313
|
+
- `fallback` — learned evidence was too weak and an auditable fallback was used.
|
|
314
|
+
|
|
315
|
+
For example, XDailyDialog directly supervises `question`, but it does not
|
|
316
|
+
directly supervise EQ-Layer's `challenge` or `exhaustion`; those remain
|
|
317
|
+
`derived`. `StanceDecision` uses the same principle: factual
|
|
318
|
+
`user_right/user_wrong` stays `unknown` unless externally verified.
|
|
319
|
+
|
|
320
|
+
## Blind response-level evaluation
|
|
321
|
+
|
|
322
|
+
Component metrics do not establish that EQ-Layer improves the final response.
|
|
323
|
+
The repository therefore includes a condition-blind paired human-evaluation
|
|
324
|
+
harness. The complete pre-registered protocol is in `eval/AB_PROTOCOL.md`.
|
|
325
|
+
|
|
326
|
+
Generate paired responses with the **same base model** in both arms:
|
|
327
|
+
|
|
328
|
+
```bash
|
|
329
|
+
python eval/ab_generate.py heldout.jsonl pairs.jsonl manifest.json \
|
|
330
|
+
--model-command "python model_adapter.py" \
|
|
331
|
+
--model-id SAME_BASE_MODEL_VERSION \
|
|
332
|
+
--affect-model artifacts/emobank_affect.joblib \
|
|
333
|
+
--subtext-model artifacts/xdailydialog_subtext.joblib \
|
|
334
|
+
--temperature 0 \
|
|
335
|
+
--seed 42 \
|
|
336
|
+
--git-commit <eq-layer-commit>
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
The configured command receives JSON on stdin and returns either plain response
|
|
340
|
+
text or JSON containing `text`/`response`. Secrets belong in environment
|
|
341
|
+
variables, not command arguments. `eval/cases.jsonl` is rejected by default
|
|
342
|
+
because it is a development set, and gold annotations are ignored unless an
|
|
343
|
+
explicit oracle-analysis flag is supplied.
|
|
344
|
+
|
|
345
|
+
|
|
346
|
+
Prepare pairs as JSONL:
|
|
347
|
+
|
|
348
|
+
```json
|
|
349
|
+
{"id":"case-001","context":[{"role":"user","content":"..."}],"baseline":"...","eq":"..."}
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
Blind and randomize them:
|
|
353
|
+
|
|
354
|
+
```bash
|
|
355
|
+
python eval/ab_prepare.py pairs.jsonl ballot.jsonl key.json --seed 42
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
Keep `key.json` away from raters. For every case, rate A/B/tie on:
|
|
359
|
+
|
|
360
|
+
- `intent_fidelity`
|
|
361
|
+
- `appropriateness`
|
|
362
|
+
- `actionability`
|
|
363
|
+
- `non_patronizing`
|
|
364
|
+
- `overall`
|
|
365
|
+
|
|
366
|
+
Then score a single rater:
|
|
367
|
+
|
|
368
|
+
```bash
|
|
369
|
+
python eval/ab_score.py rated_ballot.jsonl key.json
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
Or score multiple raters on the same blinded cases:
|
|
373
|
+
|
|
374
|
+
```bash
|
|
375
|
+
python eval/ab_score_multi.py key.json rater1.jsonl rater2.jsonl rater3.jsonl
|
|
376
|
+
```
|
|
377
|
+
|
|
378
|
+
The single-rater scorer reports EQ wins/losses/ties, non-tie win rate, Wilson
|
|
379
|
+
95% confidence intervals, and an exact two-sided sign test. The multi-rater
|
|
380
|
+
scorer additionally aggregates **case-level majority preference** and reports
|
|
381
|
+
**Fleiss' kappa** for inter-rater agreement; it does not treat every rater vote
|
|
382
|
+
as an independent case.
|
|
383
|
+
|
|
384
|
+
The harness does **not** manufacture a result: response generation and human
|
|
385
|
+
ratings must come from a real pre-registered comparison.
|
|
386
|
+
|
|
387
|
+
### Passive response auditor
|
|
388
|
+
|
|
389
|
+
A generated EQ response is also inspected by `ResponseAuditor`. The A/B
|
|
390
|
+
generation pipeline records the result under `generation.response_audit` and
|
|
391
|
+
the manifest records `response_auditor_mode=passive-metadata-only`.
|
|
392
|
+
|
|
393
|
+
The auditor currently checks only structurally observable control violations:
|
|
394
|
+
|
|
395
|
+
- empty response;
|
|
396
|
+
- question-budget overrun;
|
|
397
|
+
- missing required clarification;
|
|
398
|
+
- unsupported absolute privacy promise;
|
|
399
|
+
- `stance=unknown` combined with an explicit "you are right / I was wrong"
|
|
400
|
+
admission;
|
|
401
|
+
- near-duplicate recent assistant responses;
|
|
402
|
+
- repeated prior answer while a repair is active;
|
|
403
|
+
- large low-verbosity overruns.
|
|
404
|
+
|
|
405
|
+
It does **not** decide whether factual claims are true, whether the user's
|
|
406
|
+
position is correct, or whether advice is substantively good. Hard audit
|
|
407
|
+
failures currently request regeneration in metadata only; automatic
|
|
408
|
+
regeneration is deliberately disabled in the A/B experiment so the evaluation
|
|
409
|
+
protocol is not silently changed.
|
|
410
|
+
|
|
411
|
+
## Selection
|
|
412
|
+
|
|
413
|
+
Policies declare preconditions, and the most constrained applicable policy
|
|
414
|
+
wins. There is no first-match rule, so adding a policy cannot silently change
|
|
415
|
+
what a looser one does to a state you did not think about.
|
|
416
|
+
|
|
417
|
+
Two consequences worth arguing about:
|
|
418
|
+
|
|
419
|
+
- **Implicit tie-breaking is forbidden.** Selection ranks by specificity, then
|
|
420
|
+
explicit `priority`. If two applicable policies still tie, the selector
|
|
421
|
+
raises an ambiguity error instead of silently picking whichever was declared
|
|
422
|
+
first.
|
|
423
|
+
- **`stance` is not guessed.** Whether the user is right is a semantic
|
|
424
|
+
judgement, so the tracker returns `unknown` unless a case or caller supplies
|
|
425
|
+
one. Policies that need it declare `user_is_right` or `stance_unknown`, so
|
|
426
|
+
"I could not tell" stays visible in the output instead of becoming a
|
|
427
|
+
confident wrong answer.
|
|
428
|
+
|
|
429
|
+
Cases can carry `annotated` fields (`stance`, `subtext`) for signals that
|
|
430
|
+
keywords genuinely cannot recover — exhaustion is the current example.
|
|
431
|
+
|
|
432
|
+
## Layout
|
|
433
|
+
|
|
434
|
+
```
|
|
435
|
+
eq_layer/
|
|
436
|
+
policies.py joint affect/intent policy taxonomy + selector
|
|
437
|
+
actions.py task/social/repair/realization factorization
|
|
438
|
+
auditor.py structural post-generation control-contract audit
|
|
439
|
+
interaction.py structural repair lifecycle + interaction-quality state
|
|
440
|
+
affect.py affect-state tracker interface + zero-dependency structural fallback
|
|
441
|
+
trained_affect.py learned VAD regressor + multi-turn affect adapter
|
|
442
|
+
trained_subtext.py learned dialogue-act/emotion signals + conservative subtext derivation
|
|
443
|
+
trained_disagreement.py experimental direct-disagreement model; verified no-go for routing
|
|
444
|
+
trained_breakdown.py experimental text-only breakdown model; verified no-go for routing
|
|
445
|
+
tracker.py composes affect, subtext, stance, repair and interaction quality
|
|
446
|
+
intent.py intent state + zero-dependency fallback
|
|
447
|
+
intent_belief.py posterior belief + transparent Bayes-risk router
|
|
448
|
+
trained_intent.py learned TF-IDF + logistic-regression posterior adapter
|
|
449
|
+
data/intent_train.jsonl bundled training corpus
|
|
450
|
+
steer.py policy + intent → decode path, and the scorer
|
|
451
|
+
response_experiment.py same-model baseline-vs-EQ generation engine
|
|
452
|
+
eval/
|
|
453
|
+
cases.jsonl development cases, not final A/B evidence
|
|
454
|
+
AB_PROTOCOL.md pre-registered response-level evaluation protocol
|
|
455
|
+
ab_generate.py paired response generation
|
|
456
|
+
ab_prepare.py A/B blinding
|
|
457
|
+
ab_score.py single-rater scoring
|
|
458
|
+
ab_score_multi.py multi-rater scoring
|
|
459
|
+
run.py
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
## Scoring
|
|
463
|
+
|
|
464
|
+
Three metrics, deliberately narrow:
|
|
465
|
+
|
|
466
|
+
- **specificity** — does the response name the concrete thing the user actually said, or a category of difficulty?
|
|
467
|
+
- **genericness** — count of empty validation phrases. Penalised on purpose: these are cheap under human raters, so models learn to produce them unless a cost is attached.
|
|
468
|
+
- **escalation latency** — how many turns pass after tension rises before the register changes.
|
|
469
|
+
|
|
470
|
+
If genericness is not penalised, a model will score well by saying nothing in particular.
|
|
471
|
+
|
|
472
|
+
## Affect and conversational signals are now partly learned
|
|
473
|
+
|
|
474
|
+
`TrainedAffect` learns continuous Valence, Arousal and Dominance from EmoBank,
|
|
475
|
+
then converts Valence to `[-1, 1]` and Arousal to `[0, 1]` for policy state.
|
|
476
|
+
Multi-turn `escalation_delta` is computed from consecutive learned arousal
|
|
477
|
+
predictions rather than keyword counts.
|
|
478
|
+
|
|
479
|
+
`TrainedSubtext` separately learns dialogue-act and basic-emotion signals from
|
|
480
|
+
XDailyDialog. Those supervised signals are then combined with V/A state to
|
|
481
|
+
derive higher-level control labels conservatively:
|
|
482
|
+
|
|
483
|
+
- learned `question` -> `question`
|
|
484
|
+
- learned `directive` + independent heat -> `demand`
|
|
485
|
+
- learned `question` + anger/disgust + high arousal -> `challenge`
|
|
486
|
+
- learned sadness + negative low-arousal V/A -> `exhaustion` / `resignation`
|
|
487
|
+
|
|
488
|
+
`correction` and disclosure boundaries remain structural because the chosen
|
|
489
|
+
upstream data does not directly supervise those categories.
|
|
490
|
+
|
|
491
|
+
`stance=user_right/user_wrong` is **not** inferred from generic dialogue.
|
|
492
|
+
Whether a user is factually or procedurally correct requires external evidence.
|
|
493
|
+
`AnnotationStanceResolver` therefore emits `unknown` unless a verifier or
|
|
494
|
+
annotation supplies a stance. This is intentional rather than a missing
|
|
495
|
+
confidence threshold.
|
|
496
|
+
|
|
497
|
+
The remaining limitations are:
|
|
498
|
+
|
|
499
|
+
- **`escalating_past_n` excludes the current turn.** The question is whether
|
|
500
|
+
the preceding turns were rising, so one polite message after three hostile
|
|
501
|
+
ones does not reset the register.
|
|
502
|
+
- **`question` vs `challenge`** turns on whether escalation markers are
|
|
503
|
+
present, not on punctuation. `Sözleşme kaç gün geçerli?` is a question;
|
|
504
|
+
`Sen de mi?!` is a challenge.
|
|
505
|
+
- **Exhaustion/resignation still lacks direct supervision.** The current label
|
|
506
|
+
is a `derived` composition of learned sadness plus low-valence/low-arousal
|
|
507
|
+
state. It must not be described as a directly learned exhaustion classifier.
|
|
508
|
+
- **First-word matching strips punctuation.** `Neden?` must match `neden` or
|
|
509
|
+
the one-word repeated question — the exact shape `escalating` exists to
|
|
510
|
+
catch — silently never fires.
|
|
511
|
+
|
|
512
|
+
The remaining hard gap is direct, transferable supervision for higher-level
|
|
513
|
+
states such as correction, exhaustion/resignation, and factual stance. Two
|
|
514
|
+
plausible direct proxies — disagreement detection and text-only breakdown
|
|
515
|
+
detection — were tested and retained as no-go results instead of being wired
|
|
516
|
+
into policy routing. The tracker exposes where labels are direct, derived,
|
|
517
|
+
structural, fallback, or externally verified.
|
|
518
|
+
|
|
519
|
+
## Contributing
|
|
520
|
+
|
|
521
|
+
Cases are the main entry point. Open one for a failure mode you can describe concretely — especially the ones where two policies seem equally right.
|
|
522
|
+
|
|
523
|
+
Policy changes go through an RFC first: `.github/RFC_TEMPLATE.md`.
|
|
524
|
+
|
|
525
|
+
## License
|
|
526
|
+
|
|
527
|
+
MIT. See `LICENSE`.
|