sentience-ts 0.10.3

package/package.json

@@ -0,0 +1,58 @@
+{
+  "name": "sentience-ts",
+  "version": "0.10.3",
+  "description": "TypeScript SDK for Sentience AI Agent Browser Automation",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "scripts": {
+    "build": "tsc",
+    "test": "jest",
+    "prepare": "npm run build",
+    "prepublishOnly": "npm test && npm run build",
+    "example:hello": "ts-node examples/hello.ts",
+    "example:basic": "ts-node examples/basic-agent.ts",
+    "cli": "ts-node src/cli.ts"
+  },
+  "bin": {
+    "sentience": "./dist/cli.js"
+  },
+  "dependencies": {
+    "playwright": "^1.40.0",
+    "turndown": "^7.2.2",
+    "zod": "^3.22.0"
+  },
+  "devDependencies": {
+    "@types/jest": "^29.5.14",
+    "@types/node": "^20.0.0",
+    "@types/turndown": "^5.0.3",
+    "jest": "^29.0.0",
+    "ts-jest": "^29.0.0",
+    "ts-node": "^10.9.0",
+    "typescript": "^5.0.0"
+  },
+  "files": [
+    "dist",
+    "spec",
+    "README.md",
+    "LICENSE"
+  ],
+  "keywords": [
+    "browser-automation",
+    "playwright",
+    "ai-agent",
+    "web-automation",
+    "sentience"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/SentienceAPI/sentience-ts.git"
+  },
+  "bugs": {
+    "url": "https://github.com/SentienceAPI/sentience-ts/issues"
+  },
+  "homepage": "https://github.com/SentienceAPI/sentience-ts#readme",
+  "license": "MIT",
+  "engines": {
+    "node": ">=20.0.0"
+  }
+}

package/spec/README.md

@@ -0,0 +1,72 @@
+# Sentience API Specification
+
+This directory contains the **single source of truth** for the API contract between the Chrome extension and SDKs.
+
+## Files
+
+- **`snapshot.schema.json`** - JSON Schema for snapshot response validation
+- **`SNAPSHOT_V1.md`** - Human-readable snapshot API contract
+- **`sdk-types.md`** - SDK-level type definitions (ActionResult, WaitResult, TraceStep)
+
+## Purpose
+
+These specifications ensure:
+1. **Consistency**: Both Python and TypeScript SDKs implement the same contract
+2. **Validation**: SDKs can validate extension responses
+3. **Type Safety**: Strong typing in both languages
+4. **Documentation**: Clear reference for developers
+
+## Usage
+
+### For SDK Developers
+
+1. **Read** `SNAPSHOT_V1.md` for human-readable contract
+2. **Use** `snapshot.schema.json` for JSON Schema validation
+3. **Reference** `sdk-types.md` for SDK-level types
+
+### For Extension Developers
+
+1. **Ensure** extension output matches `snapshot.schema.json`
+2. **Update** schema when adding new fields
+3. **Version** schema for breaking changes
+
+## Versioning
+
+- **v1.0.0**: Initial stable version (Day 1)
+- Future versions: Increment major version for breaking changes
+- SDKs should validate version and handle compatibility
+
+## Validation
+
+Both SDKs should validate extension responses:
+
+**Python**:
+```python
+import jsonschema
+from spec.snapshot.schema import load_schema
+
+schema = load_schema()
+jsonschema.validate(snapshot_data, schema)
+```
+
+**TypeScript**:
+```typescript
+import Ajv from 'ajv';
+import schema from './spec/snapshot.schema.json';
+
+const ajv = new Ajv();
+const validate = ajv.compile(schema);
+validate(snapshot_data);
+```
+
+## Testing
+
+- Validate against real extension output
+- Test with edge cases (empty pages, many elements, errors)
+- Verify type coercion and defaults
+
+---
+
+**Last Updated**: Day 1 Implementation  
+**Status**: ✅ Stable
+

package/spec/SNAPSHOT_V1.md

@@ -0,0 +1,208 @@
+# Sentience Snapshot API Contract v1
+
+**Version**: 1.0.0  
+**Last Updated**: [Current Date]  
+**Status**: Stable
+
+This document defines the **single source of truth** for the snapshot data structure returned by `window.sentience.snapshot()`. Both Python and TypeScript SDKs must implement this contract exactly.
+
+## Overview
+
+The snapshot API returns a structured representation of the current page state, including:
+- All interactive elements with semantic roles
+- Element positions (bounding boxes)
+- Importance scores (AI-optimized ranking)
+- Visual cues (primary actions, colors, clickability)
+- Optional screenshot
+
+## Response Structure
+
+### Top-Level Object
+
+```typescript
+{
+  status: "success" | "error",
+  timestamp?: string,           // ISO 8601
+  url: string,
+  viewport?: { width: number, height: number },
+  elements: Element[],
+  screenshot?: string,          // Base64 data URL
+  screenshot_format?: "png" | "jpeg",
+  error?: string,              // If status is "error"
+  requires_license?: boolean   // If license required
+}
+```
+
+### Element Object
+
+```typescript
+{
+  id: number,                  // REQUIRED: Unique identifier (registry index)
+  role: string,                // REQUIRED: Semantic role
+  text: string | null,         // Text content, aria-label, or placeholder
+  importance: number,          // REQUIRED: Importance score (-300 to ~1800)
+  bbox: BBox,                  // REQUIRED: Bounding box
+  visual_cues: VisualCues,     // REQUIRED: Visual analysis
+  in_viewport: boolean,        // Is element visible in viewport
+  is_occluded: boolean,        // Is element covered by overlay
+  z_index: number              // CSS z-index (0 if auto)
+}
+```
+
+### BBox (Bounding Box)
+
+```typescript
+{
+  x: number,       // Left edge in pixels
+  y: number,       // Top edge in pixels
+  width: number,   // Width in pixels
+  height: number   // Height in pixels
+}
+```
+
+### VisualCues
+
+```typescript
+{
+  is_primary: boolean,                    // Visually prominent primary action
+  background_color_name: string | null,   // Named color from palette
+  is_clickable: boolean                   // Has pointer cursor or actionable role
+}
+```
+
+## Field Details
+
+### `id` (required)
+- **Type**: `integer`
+- **Description**: Unique element identifier, corresponds to index in `window.sentience_registry`
+- **Usage**: Used for actions like `click(id)`
+- **Stability**: May change between page loads (not persistent)
+
+### `role` (required)
+- **Type**: `string`
+- **Values**: `"button"`, `"link"`, `"textbox"`, `"searchbox"`, `"checkbox"`, `"radio"`, `"combobox"`, `"image"`, `"generic"`
+- **Description**: Semantic role inferred from HTML tag, ARIA attributes, and context
+- **Usage**: Primary filter for query engine
+
+### `text` (optional)
+- **Type**: `string | null`
+- **Description**: Text content extracted from element:
+  - `aria-label` if present
+  - `value` or `placeholder` for inputs
+  - `alt` for images
+  - `innerText` for other elements (truncated to 100 chars)
+- **Usage**: Text matching in query engine
+
+### `importance` (required)
+- **Type**: `integer`
+- **Range**: -300 to ~1800
+- **Description**: AI-optimized importance score calculated from:
+  - Role priority (inputs: 1000, buttons: 500, links: 100)
+  - Area score (larger elements score higher, capped at 200)
+  - Visual prominence (+200 for primary actions)
+  - Viewport/occlusion penalties (-500 off-screen, -800 occluded)
+- **Usage**: Ranking and filtering elements
+
+### `bbox` (required)
+- **Type**: `BBox` object
+- **Description**: Element position and size in viewport coordinates
+- **Coordinates**: Relative to viewport (0,0) at top-left
+- **Usage**: Spatial queries, visual grounding, click coordinates
+
+### `visual_cues` (required)
+- **Type**: `VisualCues` object
+- **Description**: Visual analysis results
+- **Fields**:
+  - `is_primary`: True if element is visually prominent primary action
+  - `background_color_name`: Nearest named color (32-color palette) or null
+  - `is_clickable`: True if element has pointer cursor or actionable role
+
+### `in_viewport` (optional)
+- **Type**: `boolean`
+- **Description**: True if element is visible in current viewport
+- **Default**: `true` (if not present, assume visible)
+
+### `is_occluded` (optional)
+- **Type**: `boolean`
+- **Description**: True if element is covered by another element
+- **Default**: `false` (if not present, assume not occluded)
+
+### `z_index` (optional)
+- **Type**: `integer`
+- **Description**: CSS z-index value (0 if "auto" or not set)
+- **Default**: `0`
+
+## Element Sorting
+
+Elements in the `elements` array are sorted by:
+1. **Primary sort**: `importance` (descending) - most important first
+2. **Secondary sort**: `bbox.y` (ascending) - top-to-bottom reading order (if limit applied)
+
+## Example Response
+
+```json
+{
+  "status": "success",
+  "timestamp": "2025-01-20T10:30:00Z",
+  "url": "https://example.com",
+  "viewport": {
+    "width": 1280,
+    "height": 800
+  },
+  "elements": [
+    {
+      "id": 42,
+      "role": "button",
+      "text": "Sign In",
+      "importance": 850,
+      "bbox": {
+        "x": 100,
+        "y": 200,
+        "width": 120,
+        "height": 40
+      },
+      "visual_cues": {
+        "is_primary": true,
+        "background_color_name": "blue",
+        "is_clickable": true
+      },
+      "in_viewport": true,
+      "is_occluded": false,
+      "z_index": 0
+    }
+  ]
+}
+```
+
+## Error Response
+
+```json
+{
+  "status": "error",
+  "error": "Headless mode requires a valid license key...",
+  "requires_license": true
+}
+```
+
+## SDK Implementation Requirements
+
+Both Python and TypeScript SDKs must:
+
+1. **Validate** snapshot response against this schema
+2. **Parse** all required fields correctly
+3. **Handle** optional fields gracefully (defaults)
+4. **Type-check** all fields (Pydantic for Python, TypeScript types for TS)
+5. **Preserve** field names exactly (no renaming)
+
+## Versioning
+
+- **v1.0.0**: Initial stable version
+- Future versions will increment major version for breaking changes
+- SDKs should validate version and handle compatibility
+
+## Related Documents
+
+- `snapshot.schema.json` - JSON Schema validation
+- Extension implementation: `sentience-chrome/injected_api.js`
+- WASM implementation: `sentience-chrome/src/lib.rs`
+

package/spec/sdk-types.md

@@ -0,0 +1,259 @@
+# SDK-Level Type Definitions
+
+**Purpose**: Define SDK-level types that extend the snapshot contract with action results, wait results, and trace steps.
+
+## Core Types
+
+### Snapshot
+```typescript
+interface Snapshot {
+  status: "success" | "error";
+  timestamp?: string;
+  url: string;
+  viewport?: { width: number; height: number };
+  elements: Element[];
+  screenshot?: string;
+  screenshot_format?: "png" | "jpeg";
+  error?: string;
+  requires_license?: boolean;
+}
+```
+
+### Element
+```typescript
+interface Element {
+  id: number;
+  role: string;
+  text: string | null;
+  importance: number;
+  bbox: BBox;
+  visual_cues: VisualCues;
+  in_viewport: boolean;
+  is_occluded: boolean;
+  z_index: number;
+}
+```
+
+### BBox
+```typescript
+interface BBox {
+  x: number;
+  y: number;
+  width: number;
+  height: number;
+}
+```
+
+### Viewport
+```typescript
+interface Viewport {
+  width: number;
+  height: number;
+}
+```
+
+### VisualCues
+```typescript
+interface VisualCues {
+  is_primary: boolean;
+  background_color_name: string | null;
+  is_clickable: boolean;
+}
+```
+
+## Action Types
+
+### ActionResult
+```typescript
+interface ActionResult {
+  success: boolean;
+  duration_ms: number;
+  outcome?: "navigated" | "dom_updated" | "no_change" | "error";
+  url_changed?: boolean;
+  snapshot_after?: Snapshot;  // Optional in Week 1, required in Week 2
+  error?: {
+    code: string;
+    reason: string;
+    recovery_hint?: string;
+  };
+}
+```
+
+**Fields**:
+- `success`: True if action completed successfully
+- `duration_ms`: Time taken in milliseconds
+- `outcome`: What happened after action (navigation, DOM update, no change, error)
+- `url_changed`: True if URL changed (for navigation detection)
+- `snapshot_after`: Post-action snapshot (optional in MVP, required for recorder)
+- `error`: Error details if action failed
+
+## Wait & Assert Types
+
+### WaitResult
+```typescript
+interface WaitResult {
+  found: boolean;
+  element?: Element;
+  duration_ms: number;
+  timeout: boolean;
+}
+```
+
+**Fields**:
+- `found`: True if element was found
+- `element`: Found element (if found)
+- `duration_ms`: Time taken to find (or timeout)
+- `timeout`: True if wait timed out
+
+### AssertionError
+```typescript
+class AssertionError extends Error {
+  predicate: string | object;
+  timeout: number;
+  element?: Element;
+}
+```
+
+## Trace Types (for Recorder)
+
+### Trace
+```typescript
+interface Trace {
+  version: string;           // "1.0.0"
+  created_at: string;        // ISO 8601
+  start_url: string;
+  steps: TraceStep[];
+}
+```
+
+### TraceStep
+```typescript
+interface TraceStep {
+  ts: number;                // Timestamp (milliseconds since start)
+  type: "navigation" | "click" | "type" | "press" | "wait" | "assert";
+  selector?: string;         // Semantic selector (inferred)
+  element_id?: number;       // Element ID
+  text?: string;             // For type actions (may be masked)
+  key?: string;             // For press actions
+  url?: string;             // For navigation
+  snapshot?: Snapshot;       // Optional: snapshot at this step
+}
+```
+
+**Step Types**:
+- `navigation`: `goto(url)`
+- `click`: Click on element
+- `type`: Type text into element
+- `press`: Press keyboard key
+- `wait`: Wait for element/predicate
+- `assert`: Assertion check
+
+## Query Types
+
+### QuerySelector
+```typescript
+// String DSL
+type QuerySelectorString = string;  // e.g., "role=button text~'Sign in'"
+
+// Structured object
+interface QuerySelectorObject {
+  role?: string;
+  text?: string | RegExp;
+  name?: string | RegExp;
+  clickable?: boolean;
+  isPrimary?: boolean;
+  importance?: number | { min?: number; max?: number };
+}
+
+type QuerySelector = QuerySelectorString | QuerySelectorObject;
+```
+
+## Python Equivalents (Pydantic)
+
+```python
+from pydantic import BaseModel
+from typing import Optional, List, Union
+from datetime import datetime
+
+class BBox(BaseModel):
+    x: float
+    y: float
+    width: float
+    height: float
+
+class Viewport(BaseModel):
+    width: float
+    height: float
+
+class VisualCues(BaseModel):
+    is_primary: bool
+    background_color_name: Optional[str]
+    is_clickable: bool
+
+class Element(BaseModel):
+    id: int
+    role: str
+    text: Optional[str]
+    importance: int
+    bbox: BBox
+    visual_cues: VisualCues
+    in_viewport: bool = True
+    is_occluded: bool = False
+    z_index: int = 0
+
+class Snapshot(BaseModel):
+    status: str  # "success" | "error"
+    timestamp: Optional[str] = None
+    url: str
+    viewport: Optional[Viewport] = None
+    elements: List[Element]
+    screenshot: Optional[str] = None
+    screenshot_format: Optional[str] = None
+    error: Optional[str] = None
+    requires_license: Optional[bool] = None
+
+class ActionResult(BaseModel):
+    success: bool
+    duration_ms: int
+    outcome: Optional[str] = None
+    url_changed: Optional[bool] = None
+    snapshot_after: Optional[Snapshot] = None
+    error: Optional[dict] = None
+
+class WaitResult(BaseModel):
+    found: bool
+    element: Optional[Element] = None
+    duration_ms: int
+    timeout: bool
+
+class TraceStep(BaseModel):
+    ts: int
+    type: str
+    selector: Optional[str] = None
+    element_id: Optional[int] = None
+    text: Optional[str] = None
+    key: Optional[str] = None
+    url: Optional[str] = None
+    snapshot: Optional[Snapshot] = None
+
+class Trace(BaseModel):
+    version: str
+    created_at: str
+    start_url: str
+    steps: List[TraceStep]
+```
+
+## Type Validation Rules
+
+1. **Required Fields**: Must be present and non-null
+2. **Optional Fields**: May be omitted or null
+3. **Type Coercion**: Numbers should be validated (no NaN, Infinity)
+4. **Enum Values**: String enums must match exactly
+5. **Array Bounds**: Elements array should be validated (not empty for success status)
+
+## Compatibility Notes
+
+- SDKs should handle missing optional fields gracefully
+- Default values should match extension behavior
+- Type coercion should be minimal (prefer validation errors)
+