open-multi-agent-kit 0.80.6 → 0.80.8

package/examples/extensions/aside-computer-use/README.md

@@ -0,0 +1,114 @@
+# OMK ↔ Aside Computer-Use Bridge
+
+A security-gated extension that lets the OMK agent drive an [Aside](https://aside.com)
+Chromium browser via `aside mcp` (JSON-RPC over stdio). **Aside does the browser
+execution; OMK does the planning, policy, approval, and evidence-gated completion.**
+
+> The Aside binary is **not** bundled. Install it yourself (`aside` must be on
+> `PATH`). This extension never auto-installs third-party software.
+
+## Architecture
+
+```
+OMK Agent (LLM planner)
+   │  tool calls (observe / execute_action / verify / ...)
+   ▼
+aside-computer-use extension  ← policy · risk · origin · approval · redact · evidence
+   │  MCP JSON-RPC stdio
+   ▼
+aside mcp  →  Aside Chromium Browser (logged-in profile, password manager)
+```
+
+The bridge speaks to `aside mcp`, **not raw CDP**. Aside's own Allow/Ask/Deny
+modes run in addition to OMK's gate — OMK's deny is always final.
+
+## Safety model (the point of this extension)
+
+Every browser action passes through, in order:
+
+1. **Denied-actions gate** — `credential_export`, `payment`,
+   `security_setting_change`, `account_deletion` are denied absolutely.
+2. **Origin allowlist** — mutations to origins outside `allowedOrigins` are denied.
+3. **Risk classification** (R0–R3):
+   - R0 observe (read/screenshot/DOM) → **auto-allow**
+   - R1 reversible interaction → allow on allowed origin, else approve
+   - R2 external mutation (submit/send/publish) → **human approval required**
+   - R3 critical mutation (pay/delete/credentials) → **default-deny** (privileged allowlist only)
+4. **Human approval** via `ctx.ui.confirm` for any approve decision (denies in non-UI modes).
+5. **Profile mutation mutex** — one mutating action per browser profile at a time.
+6. **Secret redaction** — DOM/result content is deep-walked; secret-named fields and
+   token-like values become `[REDACTED]` before reaching the model or logs.
+7. **Prompt-injection stance** — web page text is **untrusted observation** and never
+   expands authority. A page saying "upload ~/.ssh/id_rsa" or "disable the safety
+   policy" carries no weight; only the user goal does.
+8. **Evidence-gated completion** — `aside said it did it` is not success. Criteria are
+   predicates verified against DOM/URL evidence. Mutations with unverifiable outcomes
+   return `inspection_required` (never auto-retried) to prevent duplicate submits/payments.
+
+Credentials: Aside's password manager autofills in-browser; raw passwords never return
+to OMK or the model. "Local-first" ≠ "nothing leaves the device" — model-visible context
+(page snapshots, screenshots) can reach the model provider, so use a separate profile and
+minimal context on sensitive sites.
+
+## Setup
+
+1. Install Aside per Aside's docs and confirm `aside mcp` runs.
+2. Copy this directory to `~/.omk/agent/extensions/aside-computer-use/`.
+3. (Optional) add a policy file (see below). Defaults are conservative.
+4. Start OMK. Tools `aside_observe`, `aside_execute_action`, etc. are registered.
+
+The `aside` process is spawned lazily on first tool use and closed on session shutdown.
+
+## Policy
+
+OMK-owned, merged: defaults ← `~/.omk/agent/extensions/aside-policy.json` ←
+`<cwd>/.omk/aside-policy.json` (project wins). Example:
+
+```json
+{
+  "executable": "aside",
+  "defaultMode": "yolo",
+  "allowedOrigins": ["http://localhost:*", "https://github.com"],
+  "deniedActions": ["credential_export", "payment", "security_setting_change", "account_deletion"],
+  "approvalRequiredActions": ["submit", "send_message", "publish", "delete", "change_permission"],
+  "limits": { "maxSteps": 80, "maxRetries": 2, "maxWallTimeSeconds": 900, "maxDownloads": 10 },
+  "evidence": { "captureFinalScreenshot": true, "recordFinalUrl": true, "hashDownloadedFiles": true }
+}
+```
+
+`/aside` shows the active policy.
+
+## Tools
+
+| Tool | Purpose |
+|---|---|
+| `aside_observe` | R0 read-only page state (URL/title/text) |
+| `aside_plan_action` | Preview an action's risk band + authorization **without** executing |
+| `aside_execute_action` | Gate + execute one action (risk → origin → approval → lock → redact → evidence) |
+| `aside_verify` | Check success criteria against the current observation |
+| `aside_take_screenshot` | R0 capture, save to evidence dir with sha256 |
+| `aside_download_artifact` | Gated download, hash for integrity |
+| `aside_start_task` | observe → scripted plan → gated execute loop |
+| `aside_close_task` | Gracefully close the Aside MCP process |
+
+The agent (LLM) is the planner. It calls these tools; the extension applies the full
+safety gate on every action — matching Algorithm 2's loop while fitting OMK's tool-call model.
+
+## Module layout
+
+| File | Role |
+|---|---|
+| `index.ts` | Entry: load policy, lazy client, register tools + `/aside`, shutdown close |
+| `policy.ts` | `AsidePolicy`, defaults, load + merge |
+| `risk-classifier.ts` | action → R0..R3 band (pure) |
+| `risk-authorize.ts` | (action, risk, policy) → allow/approve/deny (pure) |
+| `url-origin.ts` | origin resolution + glob/port-wildcard matching (pure) |
+| `schema-fingerprint.ts` | MCP tool inputSchema drift detection (pure) |
+| `evidence.ts` | `redactSecrets`, file hashing, evidence builders |
+| `session-binding.ts` | OMK↔Aside binding store + per-profile mutex |
+| `mcp-client.ts` | JSON-RPC stdio client (initialize/list/call, lazy spawn, graceful close) |
+| `controller.ts` | Algorithm 2 policy-gated loop (mockable ports) |
+| `tools.ts` | 8-tool facade over client + gate |
+
+Pure safety modules have unit tests (`packages/coding-agent/test/aside-*.test.ts`); the
+controller is tested with mock ports — no real Aside binary or browser required.

package/examples/extensions/aside-computer-use/action-scoring.ts

@@ -0,0 +1,122 @@
+/**
+ * Deterministic candidate action scoring for Aside planner proposals.
+ *
+ * Pure module — no I/O, no controller/tool coupling.
+ */
+
+import type { ActionScoringConfidence, BrowserAction, PlannedActionCandidate, RiskLevel } from "./types.ts";
+
+const RISK_THRESHOLDS: Readonly<Record<RiskLevel, number>> = {
+	R0: 0.5,
+	R1: 0.65,
+	R2: 0.85,
+	R3: 0.95,
+};
+
+const AMBIGUOUS_DELTA = 0.1;
+
+export interface ScoredActionCandidate {
+	readonly candidate: PlannedActionCandidate;
+	readonly total: number;
+}
+
+export interface ActionCandidateSelection {
+	readonly status: ActionScoringConfidence;
+	readonly selected?: ScoredActionCandidate;
+	readonly scored: readonly ScoredActionCandidate[];
+	readonly reason: string;
+}
+
+function clampScore(value: number | undefined): number {
+	if (typeof value !== "number" || !Number.isFinite(value)) return 0;
+	return Math.max(0, Math.min(1, value));
+}
+
+function stableStringify(value: unknown): string {
+	if (value === undefined) return "undefined";
+	if (value === null || typeof value !== "object") {
+		if (typeof value === "bigint") return value.toString();
+		return JSON.stringify(value) ?? String(value);
+	}
+	if (Array.isArray(value)) return `[${value.map((item) => stableStringify(item)).join(",")}]`;
+	const record = value as Readonly<Record<string, unknown>>;
+	return `{${Object.keys(record)
+		.sort()
+		.map((key) => `${JSON.stringify(key)}:${stableStringify(record[key])}`)
+		.join(",")}}`;
+}
+
+function actionSortKey(action: BrowserAction): string {
+	return [
+		action.kind,
+		action.url ?? "",
+		action.description,
+		action.asideTool ?? "",
+		stableStringify(action.asideArgs ?? {}),
+	].join("\u0000");
+}
+
+function compareScoredCandidates(left: ScoredActionCandidate, right: ScoredActionCandidate): number {
+	if (left.total !== right.total) return right.total - left.total;
+	const leftKey = actionSortKey(left.candidate.action);
+	const rightKey = actionSortKey(right.candidate.action);
+	if (leftKey < rightKey) return -1;
+	if (leftKey > rightKey) return 1;
+	return 0;
+}
+
+/** Score one planned action candidate using the fixed weighted formula. */
+export function scoreActionCandidate(input: PlannedActionCandidate): number {
+	const total =
+		0.3 * clampScore(input.goalProgress) +
+		0.2 * clampScore(input.observationSupport) +
+		0.15 * clampScore(input.selectorCertainty) +
+		0.1 * clampScore(input.policyFit) +
+		0.1 * clampScore(input.reversibility) +
+		0.1 * clampScore(input.toolReliability) +
+		0.05 * clampScore(input.evidenceGain) -
+		clampScore(input.ambiguityPenalty) -
+		clampScore(input.repeatPenalty) -
+		clampScore(input.riskPenalty);
+	return clampScore(total);
+}
+
+/** Minimum acceptable score for a risk band. */
+export function thresholdForRisk(risk: RiskLevel): number {
+	return RISK_THRESHOLDS[risk];
+}
+
+/**
+ * Select the best safe candidate, or require inspection when evidence is weak
+ * or the top candidates are too close to choose deterministically.
+ */
+export function selectActionCandidate(candidates: readonly PlannedActionCandidate[]): ActionCandidateSelection {
+	const scored = candidates
+		.map((candidate) => ({ candidate, total: scoreActionCandidate(candidate) }))
+		.sort(compareScoredCandidates);
+
+	const top = scored[0];
+	if (!top) {
+		return { status: "inspection_required", scored, reason: "no action candidates available" };
+	}
+
+	const threshold = thresholdForRisk(top.candidate.risk);
+	if (top.total < threshold) {
+		return {
+			status: "inspection_required",
+			scored,
+			reason: `top action score ${top.total.toFixed(3)} is below ${top.candidate.risk} threshold ${threshold.toFixed(2)}`,
+		};
+	}
+
+	const runnerUp = scored[1];
+	if (runnerUp && top.total - runnerUp.total < AMBIGUOUS_DELTA) {
+		return {
+			status: "inspection_required",
+			scored,
+			reason: `ambiguous top candidates: score delta ${(top.total - runnerUp.total).toFixed(3)}`,
+		};
+	}
+
+	return { status: "selected", selected: top, scored, reason: "top action exceeds risk threshold and ambiguity gap" };
+}

package/examples/extensions/aside-computer-use/assertions.ts

@@ -0,0 +1,380 @@
+import type {
+	AssertionVerification,
+	AssertionVerificationStatus,
+	AssertionVerificationSummary,
+	Observation,
+	SuccessAssertion,
+	SuccessCriterion,
+} from "./types.ts";
+
+export interface VerifyAssertionsOptions {
+	readonly minConfidence?: number;
+	readonly allowLowConfidenceFallback?: boolean;
+	readonly tokenOverlapThreshold?: number;
+}
+
+type CriterionInput = string | SuccessCriterion;
+
+const DEFAULT_MIN_CONFIDENCE = 0.7;
+const DEFAULT_TOKEN_OVERLAP_THRESHOLD = 0.6;
+
+export function parseCriterionToAssertion(criterion: CriterionInput): SuccessAssertion {
+	const source = typeof criterion === "string" ? criterion : criterion.description;
+	const description = normalizeWhitespace(source);
+	const id = typeof criterion === "string" ? `assert_${stableHash(description)}` : criterion.id;
+
+	const domPrefix = "dom:";
+	if (description.startsWith(domPrefix)) {
+		const domCriterion = description.slice(domPrefix.length);
+		const separatorIndex = domCriterion.lastIndexOf("=");
+		if (separatorIndex > 0) {
+			const target = normalizeWhitespace(domCriterion.slice(0, separatorIndex));
+			const expected = normalizeExpected(domCriterion.slice(separatorIndex + 1));
+			if (target.length > 0) {
+				return createAssertion(id, description, "element_value", 0.95, { target, expected });
+			}
+		}
+	}
+
+	const visiblePrefix = "visible:";
+	if (description.startsWith(visiblePrefix)) {
+		const target = normalizeWhitespace(description.slice(visiblePrefix.length));
+		if (target.length > 0) {
+			return createAssertion(id, description, "element_visible", 0.95, { target });
+		}
+	}
+
+	const urlPrefix = "url:";
+	if (description.startsWith(urlPrefix)) {
+		return createAssertion(id, description, "url", 0.8, {
+			expected: normalizeExpected(description.slice(urlPrefix.length)),
+		});
+	}
+
+	const titlePrefix = "title:";
+	if (description.startsWith(titlePrefix)) {
+		return createAssertion(id, description, "title", 0.7, {
+			expected: normalizeExpected(description.slice(titlePrefix.length)),
+		});
+	}
+
+	const textPrefix = "text:";
+	if (description.startsWith(textPrefix)) {
+		return createAssertion(id, description, "text", 0.7, {
+			expected: normalizeExpected(description.slice(textPrefix.length)),
+		});
+	}
+
+	const absentPrefix = "absent:";
+	if (description.startsWith(absentPrefix)) {
+		return createAssertion(id, description, "negative_text_absent", 0.7, {
+			expected: normalizeExpected(description.slice(absentPrefix.length)),
+		});
+	}
+
+	const quotedPhrase = getQuotedPhrase(description);
+	if (quotedPhrase !== undefined) {
+		return createAssertion(id, description, "text", 0.7, { expected: quotedPhrase });
+	}
+
+	const tokens = tokenize(description);
+	return createAssertion(id, description, "token_overlap", 0.45, {
+		expected: description,
+		tokens,
+	});
+}
+
+export function verifyAssertions(
+	assertions: readonly SuccessAssertion[],
+	observation: Observation,
+	options: VerifyAssertionsOptions = {},
+): AssertionVerificationSummary {
+	const minConfidence = options.minConfidence ?? DEFAULT_MIN_CONFIDENCE;
+	const allowLowConfidenceFallback = options.allowLowConfidenceFallback ?? false;
+	const results = assertions.map((assertion) => verifyAssertion(assertion, observation, options));
+	const requiredResults = results.filter((result) => result.assertion.required);
+
+	if (requiredResults.length === 0) {
+		return {
+			status: "inconclusive",
+			confidence: 0,
+			assertions: results,
+		};
+	}
+
+	const confidence = Math.min(...requiredResults.map((result) => result.confidence));
+	const hasFailure = requiredResults.some((result) => result.status === "fail");
+	if (hasFailure) {
+		return {
+			status: "fail",
+			confidence,
+			assertions: results,
+		};
+	}
+
+	const hasInconclusive = requiredResults.some((result) => result.status === "inconclusive");
+	const hasLowConfidence = requiredResults.some((result) => {
+		const fallbackAllowed = result.assertion.kind === "token_overlap" && allowLowConfidenceFallback;
+		return result.status === "pass" && result.confidence < minConfidence && !fallbackAllowed;
+	});
+
+	return {
+		status: hasInconclusive || hasLowConfidence ? "inconclusive" : "pass",
+		confidence,
+		assertions: results,
+	};
+}
+
+function createAssertion(
+	id: string,
+	description: string,
+	kind: SuccessAssertion["kind"],
+	confidence: number,
+	fields: {
+		readonly target?: string;
+		readonly expected?: string;
+		readonly tokens?: readonly string[];
+	},
+): SuccessAssertion {
+	return {
+		id,
+		kind,
+		description,
+		required: true,
+		confidence,
+		...(fields.target !== undefined ? { target: fields.target } : {}),
+		...(fields.expected !== undefined ? { expected: fields.expected } : {}),
+		...(fields.tokens !== undefined ? { tokens: fields.tokens } : {}),
+	};
+}
+
+function verifyAssertion(
+	assertion: SuccessAssertion,
+	observation: Observation,
+	options: VerifyAssertionsOptions,
+): AssertionVerification {
+	switch (assertion.kind) {
+		case "element_value":
+			return verifyElementValue(assertion, observation);
+		case "element_visible":
+			return verifyElementVisible(assertion, observation);
+		case "url":
+			return verifyContains(assertion, observation.url, "url");
+		case "title":
+			return verifyContains(assertion, observation.title ?? "", "title");
+		case "text":
+			return verifyContains(assertion, observation.text ?? "", "text");
+		case "negative_text_absent":
+			return verifyNegativeTextAbsent(assertion, observation);
+		case "token_overlap":
+			return verifyTokenOverlap(assertion, observation, options);
+	}
+}
+
+function verifyElementValue(assertion: SuccessAssertion, observation: Observation): AssertionVerification {
+	if (assertion.target === undefined || assertion.expected === undefined) {
+		return inconclusive(assertion, "element value assertion is missing a selector or expected value", 0);
+	}
+
+	const element = findElement(observation, assertion.target);
+	if (element === undefined) {
+		return fail(assertion, `selector ${assertion.target} was not observed`, assertion.confidence);
+	}
+
+	const actual = normalizeForMatch(element.value);
+	const expected = normalizeForMatch(assertion.expected);
+	if (actual === expected) {
+		return pass(
+			assertion,
+			`selector ${assertion.target} matched expected value`,
+			assertion.confidence,
+			element.value,
+		);
+	}
+
+	return fail(assertion, `selector ${assertion.target} value did not match`, assertion.confidence, element.value);
+}
+
+function verifyElementVisible(assertion: SuccessAssertion, observation: Observation): AssertionVerification {
+	if (assertion.target === undefined) {
+		return inconclusive(assertion, "visible assertion is missing a selector", 0);
+	}
+
+	const element = findElement(observation, assertion.target);
+	if (element === undefined) {
+		return fail(assertion, `selector ${assertion.target} was not observed`, assertion.confidence);
+	}
+
+	return pass(assertion, `selector ${assertion.target} was observed`, assertion.confidence, element.value);
+}
+
+function verifyContains(assertion: SuccessAssertion, actualValue: string, label: string): AssertionVerification {
+	if (assertion.expected === undefined || assertion.expected.length === 0) {
+		return inconclusive(assertion, `${label} assertion is missing expected text`, 0);
+	}
+	if (actualValue.length === 0) {
+		return inconclusive(assertion, `${label} was not present in the observation`, assertion.confidence);
+	}
+	if (containsNormalized(actualValue, assertion.expected)) {
+		return pass(assertion, `${label} contained expected text`, assertion.confidence, actualValue);
+	}
+	return fail(assertion, `${label} did not contain expected text`, assertion.confidence, actualValue);
+}
+
+function verifyNegativeTextAbsent(assertion: SuccessAssertion, observation: Observation): AssertionVerification {
+	if (assertion.expected === undefined || assertion.expected.length === 0) {
+		return inconclusive(assertion, "negative text assertion is missing expected text", 0);
+	}
+
+	const searchableText = observationText(observation);
+	if (searchableText.length === 0) {
+		return inconclusive(assertion, "no textual evidence was present in the observation", assertion.confidence);
+	}
+	if (containsNormalized(searchableText, assertion.expected)) {
+		return fail(assertion, "forbidden text was present", assertion.confidence, assertion.expected);
+	}
+	return pass(assertion, "forbidden text was absent", assertion.confidence);
+}
+
+function verifyTokenOverlap(
+	assertion: SuccessAssertion,
+	observation: Observation,
+	options: VerifyAssertionsOptions,
+): AssertionVerification {
+	const tokens = assertion.tokens ?? tokenize(assertion.expected ?? assertion.description);
+	if (tokens.length === 0) {
+		return inconclusive(assertion, "token-overlap assertion has no comparable tokens", 0);
+	}
+
+	const observedTokens = new Set(tokenize(observationSearchText(observation)));
+	const overlapCount = tokens.filter((token) => observedTokens.has(token)).length;
+	const overlapRatio = overlapCount / tokens.length;
+	const matchedValue = `${overlapCount}/${tokens.length} tokens`;
+
+	if (!(options.allowLowConfidenceFallback ?? false)) {
+		return {
+			assertion,
+			status: "inconclusive",
+			confidence: assertion.confidence,
+			reason: "low-confidence fallback requires explicit allowLowConfidenceFallback",
+			matchedValue,
+		};
+	}
+
+	const threshold = options.tokenOverlapThreshold ?? DEFAULT_TOKEN_OVERLAP_THRESHOLD;
+	if (overlapRatio >= threshold) {
+		return pass(assertion, "fallback token overlap met threshold", assertion.confidence, matchedValue);
+	}
+	return fail(assertion, "fallback token overlap missed threshold", assertion.confidence, matchedValue);
+}
+
+function findElement(
+	observation: Observation,
+	selector: string,
+): { readonly selector: string; readonly value: string } | undefined {
+	return observation.dom?.find((element) => element.selector === selector);
+}
+
+function observationText(observation: Observation): string {
+	return [observation.title, observation.text, ...(observation.dom ?? []).map((element) => element.value)]
+		.filter((value): value is string => value !== undefined && value.length > 0)
+		.join("\n");
+}
+
+function observationSearchText(observation: Observation): string {
+	return [observation.url, observationText(observation)].join("\n");
+}
+
+function pass(
+	assertion: SuccessAssertion,
+	reason: string,
+	confidence: number,
+	matchedValue?: string,
+): AssertionVerification {
+	return verification(assertion, "pass", reason, confidence, matchedValue);
+}
+
+function fail(
+	assertion: SuccessAssertion,
+	reason: string,
+	confidence: number,
+	matchedValue?: string,
+): AssertionVerification {
+	return verification(assertion, "fail", reason, confidence, matchedValue);
+}
+
+function inconclusive(
+	assertion: SuccessAssertion,
+	reason: string,
+	confidence: number,
+	matchedValue?: string,
+): AssertionVerification {
+	return verification(assertion, "inconclusive", reason, confidence, matchedValue);
+}
+
+function verification(
+	assertion: SuccessAssertion,
+	status: AssertionVerificationStatus,
+	reason: string,
+	confidence: number,
+	matchedValue?: string,
+): AssertionVerification {
+	return {
+		assertion,
+		status,
+		confidence: clamp01(confidence),
+		reason,
+		...(matchedValue !== undefined ? { matchedValue } : {}),
+	};
+}
+
+function normalizeExpected(value: string): string {
+	const normalized = normalizeWhitespace(value);
+	return getQuotedPhrase(normalized) ?? normalized;
+}
+
+function getQuotedPhrase(value: string): string | undefined {
+	if (value.length < 2) {
+		return undefined;
+	}
+	const exactFirst = value[0];
+	const exactLast = value[value.length - 1];
+	if ((exactFirst === '"' && exactLast === '"') || (exactFirst === "'" && exactLast === "'")) {
+		return normalizeWhitespace(value.slice(1, -1));
+	}
+	const embedded = value.match(/["']([^"']+)["']/u);
+	return embedded ? normalizeWhitespace(embedded[1]) : undefined;
+}
+
+function containsNormalized(haystack: string, needle: string): boolean {
+	const normalizedNeedle = normalizeForMatch(needle);
+	return normalizedNeedle.length > 0 && normalizeForMatch(haystack).includes(normalizedNeedle);
+}
+
+function normalizeForMatch(value: string): string {
+	return normalizeWhitespace(value).toLowerCase();
+}
+
+function normalizeWhitespace(value: string): string {
+	return value.trim().replace(/\s+/gu, " ");
+}
+
+function tokenize(value: string): readonly string[] {
+	return normalizeForMatch(value).match(/[a-z0-9]+/gu) ?? [];
+}
+
+function clamp01(value: number): number {
+	if (!Number.isFinite(value)) {
+		return 0;
+	}
+	return Math.min(1, Math.max(0, value));
+}
+
+function stableHash(value: string): string {
+	let hash = 0x811c9dc5;
+	for (let index = 0; index < value.length; index += 1) {
+		hash ^= value.charCodeAt(index);
+		hash = Math.imul(hash, 0x01000193) >>> 0;
+	}
+	return hash.toString(16).padStart(8, "0");
+}