@akalsey/sapience-feedback 0.1.1 → 0.3.0

package/README.md

@@ -32,7 +32,12 @@ openclaw plugins install npm:@akalsey/sapience-feedback
     "sapience-feedback": {
       "logPath": "~/.openclaw/sapience/feedback.md",
       "calibrationPath": "~/.openclaw/sapience/calibration.json",
-      "memoryEnabled": true
+      "memoryEnabled": true,
+      "semanticDetection": {
+        "enabled": true,
+        "minLength": 8,
+        "minConfidence": 0.6
+      }
     }
   }
 }
@@ -40,62 +45,51 @@ openclaw plugins install npm:@akalsey/sapience-feedback
 
 All settings are optional — defaults are used if omitted.
 
+**`semanticDetection`** controls the LLM-based classifier. When enabled (the default), every user message above `minLength` characters is classified by the agent's default inference provider. Set `enabled: false` to fall back to regex-only matching (useful if you want zero LLM cost on routine chat).
+
 ---
 
 ## What it detects
 
-The plugin scans every message you send for three types of signals:
+Every user message is analyzed by the agent's default inference provider (using `api.runtime.llm.complete` — no separate provider configuration required). The classifier returns structured signals in one of three categories. No trigger words or special syntax — speak normally.
 
 ### Corrections
 
-Phrases that express "don't do that" or "do it differently":
-
-- `don't update OKRs for other teams without asking`
-- `stop doing that`
-- `never push to main without a PR`
-- `use the company template, not the default`
-- `you shouldn't have done that without checking`
+Anything that tells the agent it did something wrong or should do it differently. The classifier picks up direct phrasing ("don't push to main"), rhetorical questions ("did you check the password manager first?"), and implicit critiques ("is there something wrong with the passwords you have?").
 
 **Effect:** Confidence on the matching domain/action-class drops by 0.3.
 
 ### Confirmations
 
-Phrases that express "yes, that was right":
-
-- `yes exactly`
-- `good call`
-- `perfect, keep doing that`
-- `that's exactly right`
+Anything that reinforces what the agent just did — agreement, praise, "keep doing that".
 
 **Effect:** Confidence on the matching domain/action-class increases by 0.1.
 
 ### Tier adjustments
 
-Explicit instructions about how much autonomy you want:
-
-- `just do it` / `you don't need to ask` → bumps toward **Act**
-- `always ask me first` / `ask me before touching X` → bumps toward **Ask**
+Instructions about how much autonomy the agent should have. "Just do it" or "stop asking" bumps toward **Act**. "Always check first" or "ask me before doing X" bumps toward **Ask**.
 
 **Effect:** Tier for matching domain/action-class is updated directly.
 
+If the LLM is unavailable (no `api.runtime.llm` exposed, or the call fails), the plugin falls back to a regex matcher covering the common phrasings. The regex layer is intentionally conservative and misses paraphrases — semantic detection is the primary path.
+
 ---
 
-## Domain detection
+## Explicit feedback: the `/feedback` command
 
-The plugin extracts domains from context in your message:
+When you want to leave feedback without ambiguity, use the slash command:
 
-| Text contains | Domain |
-|---------------|--------|
-| `github`, `PR`, `push` | `github` |
-| `Salesforce` | `salesforce` |
-| `PostHog` | `posthog` |
-| `Slack` | `slack` |
-| `slides`, `deck` | `slides` |
-| `OKR` | `okr-system` |
-| `Linear` | `linear` |
-| (nothing matched) | `general` |
+```
+/feedback always look at the password manager before asking me for credentials
+```
+
+The command runs the same classifier and then records the result as a `manual` signal. If the classifier finds no clear signal, the message is still logged as a generic correction in the `general` domain — manual feedback is never discarded.
+
+---
+
+## Domain detection
 
-Domain detection is keyword-based, not semantic. Be explicit when giving feedback: "don't update Salesforce records" works better than "don't do that" (which routes to `general`).
+The LLM extracts a domain slug from the content of your message: `github`, `credentials`, `okr-system`, `salesforce`, etc. When the LLM can't identify anything specific, it returns `general`. The regex fallback uses a fixed keyword table and is more likely to bucket things into `general`.
 
 ---
 
@@ -134,8 +128,8 @@ The plugin only scans messages you send (role: `user`), not the agent's response
 **Calibration not updating**
 Check that `calibration.json` exists and has an entry for the domain you're correcting. Feedback only updates *existing* entries — it doesn't create new ones. New domains are created by `sapience` when it first routes a proposal in that domain.
 
-**Domain matching to "general" when it shouldn't**
-Add more specific keywords to your feedback message. "Don't do that" → "Don't update the Salesforce contact without asking."
+**Feedback getting misclassified or missed**
+Raise the bar with `semanticDetection.minConfidence` if the classifier is too noisy; lower it if real feedback is being dropped. To force a recording, use `/feedback <text>` — manual entries bypass the confidence threshold.
 
-**Too many false positives in the feedback log**
-The pattern matching is intentionally broad. If casual phrases are being captured incorrectly, check the log and note which patterns are misfiring. You can't currently tune the patterns without modifying `src/feedback-parser.ts`.
+**LLM cost concerns**
+Every user message above `minLength` characters incurs one classifier call. To disable, set `semanticDetection.enabled: false` — the plugin will fall back to the regex matcher with no LLM calls.

package/dist/src/calibration-bridge.js

@@ -17,10 +17,10 @@ async function saveProfile(profile, path) {
 export async function applyFeedbackToProfile(signal, calibrationPath) {
     const profile = await loadProfile(calibrationPath);
     if (profile.length === 0)
-        return;
+        return { status: "orphaned" };
     const idx = profile.findIndex(e => e.domain === signal.domain && e.action_class === signal.action_class);
     if (idx === -1)
-        return;
+        return { status: "orphaned" };
     const entry = profile[idx];
     let updated;
     if (signal.type === "confirmation") {
@@ -33,7 +33,14 @@ export async function applyFeedbackToProfile(signal, calibrationPath) {
         updated = { ...entry, tier: signal.suggested_tier, last_calibrated: new Date().toISOString() };
     }
     else {
-        return;
+        return { status: "noop" };
     }
     await saveProfile(profile.map((e, i) => i === idx ? updated : e), calibrationPath);
+    return {
+        status: "applied",
+        old_confidence: entry.confidence,
+        new_confidence: updated.confidence,
+        old_tier: entry.tier,
+        new_tier: updated.tier,
+    };
 }

package/dist/src/events.js

@@ -0,0 +1,12 @@
+import { appendFile, mkdir } from "fs/promises";
+import { dirname } from "path";
+export async function appendEvent(eventsPath, event) {
+    try {
+        const full = { ...event, ts: event.ts ?? new Date().toISOString() };
+        await mkdir(dirname(eventsPath), { recursive: true });
+        await appendFile(eventsPath, JSON.stringify(full) + "\n", "utf-8");
+    }
+    catch {
+        // Observability must never break the host plugin.
+    }
+}

package/dist/src/feedback-handler.js

@@ -0,0 +1,77 @@
+import { parseMessage } from "./feedback-parser.js";
+import { classifyWithLlm } from "./llm-classifier.js";
+import { appendFeedback } from "./log-writer.js";
+import { applyFeedbackToProfile } from "./calibration-bridge.js";
+import { appendEvent } from "./events.js";
+import { generateId } from "./utils.js";
+export function shouldClassify(text, config) {
+    const trimmed = text.trim();
+    if (trimmed.length < config.semanticDetection.minLength)
+        return false;
+    if (trimmed.startsWith("```") || trimmed.startsWith("~~~"))
+        return false;
+    return true;
+}
+export async function classifyMessage(text, config, llm) {
+    if (!shouldClassify(text, config))
+        return [];
+    if (config.semanticDetection.enabled && llm) {
+        const signals = await classifyWithLlm(text, llm, {
+            minConfidence: config.semanticDetection.minConfidence,
+        });
+        if (signals.length > 0)
+            return signals;
+    }
+    return parseMessage(text).map(s => ({ ...s, source: "regex" }));
+}
+export function buildMetaPointer(signal) {
+    return `Before working on ${signal.domain} / ${signal.action_class}: check feedback log — correction recorded: "${signal.raw_text.slice(0, 80)}"`;
+}
+export async function persistSignal(signal, ctx) {
+    const metaPointer = signal.type === "correction" ? buildMetaPointer(signal) : undefined;
+    const entry = {
+        id: generateId(),
+        detected_at: new Date().toISOString(),
+        signal,
+        meta_pointer: metaPointer,
+    };
+    await appendFeedback(entry, ctx.config.logPath);
+    const result = await applyFeedbackToProfile(signal, ctx.config.calibrationPath);
+    if (metaPointer && ctx.config.memoryEnabled && ctx.memoryAdd) {
+        await ctx.memoryAdd({
+            content: metaPointer,
+            metadata: { tags: ["feedback", "behavioral-correction", signal.domain], source: "feedback" },
+        });
+    }
+    await appendEvent(ctx.config.eventsPath, {
+        plugin: "feedback",
+        type: "signal_detected",
+        signal_type: signal.type,
+        domain: signal.domain,
+        action_class: signal.action_class,
+        source: signal.source ?? "regex",
+    });
+    if (result.status === "orphaned") {
+        await appendEvent(ctx.config.eventsPath, {
+            plugin: "feedback",
+            type: "signal_orphaned",
+            signal_type: signal.type,
+            domain: signal.domain,
+            action_class: signal.action_class,
+        });
+    }
+    else if (result.status === "applied") {
+        await appendEvent(ctx.config.eventsPath, {
+            plugin: "feedback",
+            type: "calibration_change",
+            domain: signal.domain,
+            action_class: signal.action_class,
+            old_confidence: result.old_confidence,
+            new_confidence: result.new_confidence,
+            old_tier: result.old_tier,
+            new_tier: result.new_tier,
+            source: "feedback",
+        });
+    }
+    return entry;
+}

package/dist/src/llm-classifier.js

@@ -0,0 +1,134 @@
+const VALID_TYPES = new Set(["correction", "confirmation", "tier_adjustment"]);
+const VALID_TIERS = new Set(["act", "propose", "ask", "explore"]);
+const SYSTEM_PROMPT = `You analyze ONE chat message a developer sent to their AI coding agent. Decide whether it contains behavioral feedback the agent should learn from.
+
+Three signal types:
+
+1. "correction" — the user is correcting the agent, pointing out a mistake, redirecting an approach, expressing frustration with a recent action, or telling it to use a different tool/source. This includes leading questions like "did you check X first?" or "is there something wrong with X?" — these are corrections phrased as rhetorical questions.
+
+2. "confirmation" — the user is reinforcing what the agent just did: praising, agreeing, asking it to keep behaving that way.
+
+3. "tier_adjustment" — the user is changing the agent's autonomy level:
+   - suggested_tier "act": user wants less asking ("just do it", "stop asking", "go ahead without me")
+   - suggested_tier "ask": user wants more checking ("always ask first", "never do that without checking", "check X before doing Y")
+
+A message is NOT feedback if it is: a fresh task request, a technical question about code or systems, a code snippet, conversational filler, or status unrelated to the agent's behavior.
+
+When unsure, prefer empty output — false positives are worse than misses.
+
+For each detected signal extract:
+- "type": one of "correction" | "confirmation" | "tier_adjustment"
+- "domain": short kebab-case slug for the subject area. Use what fits: "github", "credentials", "okr-system", "salesforce", "slack", "slides", "linear", "posthog", or invent a clear one. Use "general" only if nothing specific applies.
+- "action_class": short slug; "general" if nothing more specific
+- "suggested_tier": "act" | "propose" | "ask" | "explore" — only set for tier_adjustment, otherwise null
+- "confidence": 0..1, how confident you are this is genuine feedback
+
+Respond with ONLY a single JSON object, no prose, no code fences:
+
+{"signals":[{"type":"...","domain":"...","action_class":"general","suggested_tier":null,"confidence":0.0}]}
+
+If no feedback: {"signals":[]}
+
+Examples:
+
+Message: "did you look in the password manager before asking me for credentials"
+Response: {"signals":[{"type":"correction","domain":"credentials","action_class":"general","suggested_tier":null,"confidence":0.9}]}
+
+Message: "you need to always look at your password manager before asking me for credentials"
+Response: {"signals":[{"type":"tier_adjustment","domain":"credentials","action_class":"general","suggested_tier":"ask","confidence":0.92}]}
+
+Message: "is there something wrong with the passwords you have"
+Response: {"signals":[{"type":"correction","domain":"credentials","action_class":"general","suggested_tier":null,"confidence":0.8}]}
+
+Message: "you have credentials in the password manager"
+Response: {"signals":[{"type":"correction","domain":"credentials","action_class":"general","suggested_tier":null,"confidence":0.75}]}
+
+Message: "yes that's exactly what I wanted, keep doing that"
+Response: {"signals":[{"type":"confirmation","domain":"general","action_class":"general","suggested_tier":null,"confidence":0.95}]}
+
+Message: "what does this regex match"
+Response: {"signals":[]}
+
+Message: "write a function that reverses a linked list"
+Response: {"signals":[]}`;
+export function buildClassifierMessages(text) {
+    return [
+        { role: "system", content: SYSTEM_PROMPT },
+        { role: "user", content: `Message: ${JSON.stringify(text)}\nResponse:` },
+    ];
+}
+function stripFences(raw) {
+    const trimmed = raw.trim();
+    const fence = trimmed.match(/^```(?:json)?\s*([\s\S]*?)\s*```$/);
+    if (fence)
+        return fence[1].trim();
+    return trimmed;
+}
+function extractJsonObject(raw) {
+    const stripped = stripFences(raw);
+    try {
+        return JSON.parse(stripped);
+    }
+    catch {
+        const first = stripped.indexOf("{");
+        const last = stripped.lastIndexOf("}");
+        if (first === -1 || last <= first)
+            return null;
+        try {
+            return JSON.parse(stripped.slice(first, last + 1));
+        }
+        catch {
+            return null;
+        }
+    }
+}
+function normalizeSignal(raw, originalText) {
+    if (typeof raw.type !== "string" || !VALID_TYPES.has(raw.type))
+        return null;
+    const type = raw.type;
+    const domain = typeof raw.domain === "string" && raw.domain.length > 0 ? raw.domain : "general";
+    const action_class = typeof raw.action_class === "string" && raw.action_class.length > 0 ? raw.action_class : "general";
+    const confidence = typeof raw.confidence === "number" ? raw.confidence : undefined;
+    const signal = {
+        type,
+        domain,
+        action_class,
+        message: originalText,
+        raw_text: originalText,
+        source: "llm",
+    };
+    if (confidence !== undefined)
+        signal.confidence = confidence;
+    if (type === "tier_adjustment" && typeof raw.suggested_tier === "string" && VALID_TIERS.has(raw.suggested_tier)) {
+        signal.suggested_tier = raw.suggested_tier;
+    }
+    return signal;
+}
+export async function classifyWithLlm(text, client, options = {}) {
+    const minConfidence = options.minConfidence ?? 0;
+    let result;
+    try {
+        result = await client.complete({
+            messages: buildClassifierMessages(text),
+            maxTokens: options.maxTokens ?? 256,
+            temperature: options.temperature ?? 0.1,
+            purpose: options.purpose ?? "sapience-feedback.classify",
+        });
+    }
+    catch {
+        return [];
+    }
+    const parsed = extractJsonObject(result.text);
+    if (!parsed || !Array.isArray(parsed.signals))
+        return [];
+    const signals = [];
+    for (const raw of parsed.signals) {
+        const normalized = normalizeSignal(raw, text);
+        if (!normalized)
+            continue;
+        if (normalized.confidence !== undefined && normalized.confidence < minConfidence)
+            continue;
+        signals.push(normalized);
+    }
+    return signals;
+}

package/dist/src/service.js

@@ -1,20 +1,24 @@
 // src/service.ts
 import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
 import { DEFAULT_CONFIG } from "./types.js";
-import { resolveDataPath, generateId } from "./utils.js";
-import { parseMessage } from "./feedback-parser.js";
-import { appendFeedback } from "./log-writer.js";
-import { applyFeedbackToProfile } from "./calibration-bridge.js";
+import { resolveDataPath } from "./utils.js";
+import { classifyMessage, persistSignal } from "./feedback-handler.js";
 function mergeConfig(raw, workspaceDir) {
+    const rawSemantic = raw.semanticDetection ?? {};
     return {
         ...DEFAULT_CONFIG,
         ...raw,
         logPath: resolveDataPath(raw.logPath, workspaceDir, DEFAULT_CONFIG.logPath),
         calibrationPath: resolveDataPath(raw.calibrationPath, workspaceDir, DEFAULT_CONFIG.calibrationPath),
+        eventsPath: resolveDataPath(raw.eventsPath, workspaceDir, DEFAULT_CONFIG.eventsPath),
+        semanticDetection: { ...DEFAULT_CONFIG.semanticDetection, ...rawSemantic },
     };
 }
-function buildMetaPointer(signal) {
-    return `Before working on ${signal.domain} / ${signal.action_class}: check feedback log — correction recorded: "${signal.raw_text.slice(0, 80)}"`;
+function getLlmClient(api) {
+    const llm = api?.runtime?.llm;
+    if (!llm || typeof llm.complete !== "function")
+        return null;
+    return { complete: (params) => llm.complete(params) };
 }
 export default definePluginEntry({
     id: "sapience-feedback",
@@ -23,28 +27,16 @@ export default definePluginEntry({
     register(api) {
         const workspaceDir = api.runtime.agent.resolveAgentWorkspaceDir(api.pluginConfig);
         const config = mergeConfig(api.pluginConfig, workspaceDir);
+        const llm = getLlmClient(api);
+        const memoryAdd = api.memory?.add ? (params) => api.memory.add(params) : undefined;
         if (api.session?.onMessage) {
             api.session.onMessage(async (message) => {
                 if (message.role !== "user")
                     return;
                 try {
-                    const signals = parseMessage(message.content);
+                    const signals = await classifyMessage(message.content, config, llm);
                     for (const signal of signals) {
-                        const metaPointer = signal.type === "correction" ? buildMetaPointer(signal) : undefined;
-                        const entry = {
-                            id: generateId(),
-                            detected_at: new Date().toISOString(),
-                            signal,
-                            meta_pointer: metaPointer,
-                        };
-                        await appendFeedback(entry, config.logPath);
-                        await applyFeedbackToProfile(signal, config.calibrationPath);
-                        if (metaPointer && config.memoryEnabled) {
-                            await api.memory?.add({
-                                content: metaPointer,
-                                metadata: { tags: ["feedback", "behavioral-correction", signal.domain], source: "feedback" },
-                            });
-                        }
+                        await persistSignal(signal, { config, memoryAdd });
                     }
                 }
                 catch {
@@ -52,5 +44,44 @@ export default definePluginEntry({
                 }
             });
         }
+        if (typeof api.registerCommand === "function") {
+            api.registerCommand({
+                name: "feedback",
+                description: "Record explicit feedback for the agent to learn from. Usage: /feedback <your feedback>",
+                acceptsArgs: true,
+                handler: async (ctx) => {
+                    const text = (ctx.args ?? "").trim();
+                    if (!text) {
+                        return { text: "Usage: /feedback <your feedback>\n\nExample: /feedback always check the password manager before asking me for credentials" };
+                    }
+                    try {
+                        let signals = await classifyMessage(text, config, llm);
+                        if (signals.length === 0) {
+                            signals = [{
+                                    type: "correction",
+                                    domain: "general",
+                                    action_class: "general",
+                                    message: text,
+                                    raw_text: text,
+                                    source: "manual",
+                                }];
+                        }
+                        else {
+                            signals = signals.map(s => ({ ...s, source: "manual" }));
+                        }
+                        for (const signal of signals) {
+                            await persistSignal(signal, { config, memoryAdd });
+                        }
+                        const summary = signals.map(s => s.type === "tier_adjustment" && s.suggested_tier
+                            ? `${s.type} → ${s.suggested_tier} (${s.domain})`
+                            : `${s.type} (${s.domain})`).join(", ");
+                        return { text: `Recorded ${signals.length} feedback signal(s): ${summary}` };
+                    }
+                    catch (err) {
+                        return { text: `Failed to record feedback: ${err instanceof Error ? err.message : String(err)}` };
+                    }
+                },
+            });
+        }
     },
 });

package/dist/src/types.js

@@ -1,5 +1,11 @@
 export const DEFAULT_CONFIG = {
     logPath: "sapience/feedback.md",
     calibrationPath: "sapience/calibration.json",
+    eventsPath: "sapience/events.jsonl",
     memoryEnabled: true,
+    semanticDetection: {
+        enabled: true,
+        minLength: 8,
+        minConfidence: 0.6,
+    },
 };

package/openclaw.plugin.json

@@ -1,7 +1,7 @@
 {
   "id": "sapience-feedback",
   "name": "Sapience Feedback",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Persists behavioral corrections and confirmations from session messages into the sapience calibration profile",
   "activation": { "onStartup": true },
   "configSchema": {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@akalsey/sapience-feedback",
-  "version": "0.1.1",
+  "version": "0.3.0",
   "type": "module",
   "main": "./dist/index.js",
   "exports": {