@archetypeai/ds-cli 0.3.7 → 0.3.10

package/files/skills/newton-camera-frame-analysis/SKILL.md

@@ -0,0 +1,611 @@
+---
+name: newton-camera-frame-analysis
+description: Live webcam frame analysis using Newton's vision model via model.query (request/response). Captures frames from a webcam as base64 JPEG and sends them to Newton. Use for live camera analysis, scene description, presence detection, or visual Q&A. NOT for video file uploads — use /activity-monitor-lens-on-video for that.
+argument-hint: [question] [camera_index]
+allowed-tools: Bash(python *), Read
+---
+
+# Newton Camera Frame Analysis (Live Webcam → base64 → model.query)
+
+Capture live webcam frames, encode as base64 JPEG, and send to Newton's vision model via `model.query` (synchronous request/response). Supports Python (OpenCV) and JavaScript (getUserMedia + canvas).
+
+> **Frontend architecture:** When building a web UI for this skill, decompose into components (webcam input, status display, results view) rather than a monolithic page. Extract API logic into `$lib/api/`. See `@rules/frontend-architecture` for conventions and `@skills/create-dashboard` / `@skills/build-pattern` for layout and component patterns.
+
+**This skill is for LIVE WEBCAM input only.** For analyzing uploaded video files, use `/activity-monitor-lens-on-video` instead.
+
+| | This skill (camera-frame-analysis) | activity-monitor-lens-on-video |
+|---|---|---|
+| **Input** | Live webcam (base64 JPEG frames) | Uploaded video file |
+| **Who captures frames** | Client (Python cv2 / JS canvas) | Server (`video_file_reader`) |
+| **Event type** | `model.query` (request/response) | Server-driven, results via SSE |
+| **Response** | Direct in POST response | Async via SSE stream |
+| **Use case** | Real-time webcam Q&A | Batch video analysis |
+
+---
+
+## Model Parameters
+
+| Parameter | Default | Notes |
+|---|---|---|
+| `model_version` | `Newton::c2_4_7b_251215a172f6d7` | Newton model ID |
+| `template_name` | `image_qa_template_task` | Prompt template |
+| `instruction` | *(user-provided)* | System prompt guiding output format |
+| `focus` | *(user-provided)* | The question or what to look for |
+| `max_new_tokens` | `512` | Max response length |
+| `camera_buffer_size` | `1` | Single-frame buffer for webcam |
+| `min_replicas` / `max_replicas` | `1` / `1` | Scaling config |
+
+**IMPORTANT:** `instruction` and `focus` must be passed as parameters — not hardcoded. The values in the lens config (registration) and in each `model.query` event must be consistent. Pass the user's values into both.
+
+---
+
+## Python Implementation
+
+### Requirements
+
+- `archetypeai` Python package
+- `opencv-python` (`cv2`), `Pillow`
+- Environment variables: `ATAI_API_KEY` or `ARCHETYPE_API_KEY`
+
+### Quick Start
+
+```bash
+export ATAI_API_KEY=your_key_here
+python camera_frame_analysis.py "Describe what you see"
+
+# Custom question
+python camera_frame_analysis.py "Is anyone present?"
+
+# Different camera
+python camera_frame_analysis.py "Describe the scene" 1
+```
+
+### Parameters
+
+- **question** (positional, optional): What to analyze (default: "Describe what you see")
+- **camera_index** (positional, optional): Camera index (default: 0)
+
+### How It Works
+
+1. **Capture**: Opens webcam with OpenCV, reads a frame
+2. **Encode**: Converts frame to base64 JPEG (BGR → RGB → PIL → JPEG → base64)
+3. **Setup**: Registers Newton lens, creates session, waits for ready
+4. **Initialize**: Sends `session.modify` to initialize the processor
+5. **Query**: Sends base64 image as `model.query` event, gets response directly
+6. **Cleanup**: Destroys session
+
+### Webcam Capture → base64
+
+```python
+import cv2
+import base64
+import io
+from PIL import Image
+
+def capture_frame_base64(camera_index=0, jpeg_quality=80, resize=(640, 480)):
+    """Capture a webcam frame and return as raw base64 JPEG string."""
+    cap = cv2.VideoCapture(camera_index)
+    ret, frame = cap.read()
+    cap.release()
+
+    if not ret:
+        raise RuntimeError(f"Failed to capture from camera {camera_index}")
+
+    # Resize if needed
+    h, w = frame.shape[:2]
+    if (w, h) != resize:
+        frame = cv2.resize(frame, resize)
+
+    # BGR (OpenCV) → RGB → PIL → JPEG → base64
+    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+    pil_image = Image.fromarray(rgb_frame)
+
+    buffer = io.BytesIO()
+    pil_image.save(buffer, format="JPEG", quality=jpeg_quality)
+    raw_base64 = base64.b64encode(buffer.getvalue()).decode()
+
+    return raw_base64  # No "data:image/jpeg;base64," prefix
+```
+
+### Full Python Example
+
+```python
+import os
+import time
+from archetypeai.api_client import ArchetypeAI
+
+api_key = os.getenv("ATAI_API_KEY")
+client = ArchetypeAI(api_key)
+
+# --- User-provided values (NOT hardcoded) ---
+instruction = "Answer the following question about the image:"
+focus = "Describe what you see in this image."
+
+def build_lens_config(instruction: str, focus: str) -> dict:
+    """Build lens config with user-provided instruction and focus."""
+    return {
+        "lens_name": "camera-frame-capture-lens",
+        "lens_config": {
+            "model_pipeline": [
+                {"processor_name": "lens_camera_processor", "processor_config": {}}
+            ],
+            "model_parameters": {
+                "model_version": "Newton::c2_4_7b_251215a172f6d7",
+                "template_name": "image_qa_template_task",
+                "instruction": instruction,
+                "focus": focus,
+                "max_new_tokens": 512,
+                "camera_buffer_size": 1,
+                "min_replicas": 1,
+                "max_replicas": 1,
+            },
+        },
+    }
+
+def build_query_event(raw_base64: str, instruction: str, focus: str) -> dict:
+    """Build model.query event with the SAME instruction and focus as the lens config."""
+    return {
+        "type": "model.query",
+        "event_data": {
+            "model_version": "Newton::c2_4_7b_251215a172f6d7",
+            "template_name": "image_qa_template_task",
+            "instruction": instruction,
+            "focus": focus,
+            "max_new_tokens": 512,
+            "data": [{"type": "base64_img", "base64_img": raw_base64}],
+        },
+    }
+
+# 1. Register lens (pass user's instruction + focus)
+lens_config = build_lens_config(instruction, focus)
+lens = client.lens.register(lens_config)
+lens_id = lens["lens_id"]
+
+# 2. Create session
+session = client.lens.sessions.create(lens_id)
+session_id = session["session_id"]
+
+# 3. Wait for session ready
+for _ in range(60):
+    try:
+        status = client.lens.sessions.process_event(
+            session_id, {"type": "session.status"}
+        )
+        if status.get("session_status") in ["3", "LensSessionStatus.SESSION_STATUS_RUNNING"]:
+            break
+    except Exception:
+        pass
+    time.sleep(0.5)
+
+# 4. Initialize processor (REQUIRED)
+client.lens.sessions.process_event(session_id, {
+    "type": "session.modify",
+    "event_data": {"camera_buffer_size": 1}
+})
+
+# 5. Capture frame and send as model.query (same instruction + focus)
+raw_base64 = capture_frame_base64(camera_index=0)
+event = build_query_event(raw_base64, instruction, focus)
+
+response = client.lens.sessions.process_event(session_id, event)
+
+if response.get("type") == "model.query.response":
+    result = response["event_data"]["response"]
+    if isinstance(result, list):
+        result = result[0]
+    print(f"Answer: {result}")
+
+# 6. Cleanup
+client.lens.sessions.destroy(session_id)
+```
+
+---
+
+## Web / JavaScript Implementation
+
+Uses direct `fetch` calls to the Archetype AI REST API.
+
+### Requirements
+
+- Browser with `getUserMedia` support (webcam access)
+- HTTPS (required for camera access, except `localhost`)
+
+### API Reference
+
+| Operation | Method | Endpoint | Body |
+|-----------|--------|----------|------|
+| List lenses | GET | `/lens/metadata` | — |
+| Register lens | POST | `/lens/register` | `{ lens_config: config }` |
+| Delete lens | POST | `/lens/delete` | `{ lens_id }` |
+| Create session | POST | `/lens/sessions/create` | `{ lens_id }` |
+| Process event | POST | `/lens/sessions/events/process` | `{ session_id, event }` |
+| Destroy session | POST | `/lens/sessions/destroy` | `{ session_id }` |
+
+### Helpers: API wrappers
+
+```typescript
+const API_ENDPOINT = 'https://api.u1.archetypeai.app/v0.5'
+
+async function apiGet<T>(path: string, apiKey: string): Promise<T> {
+  const response = await fetch(`${API_ENDPOINT}${path}`, {
+    method: 'GET',
+    headers: { Authorization: `Bearer ${apiKey}` },
+  })
+  if (!response.ok) throw new Error(`API GET ${path} failed: ${response.status}`)
+  return response.json()
+}
+
+async function apiPost<T>(path: string, apiKey: string, body: unknown, timeoutMs = 5000): Promise<T> {
+  const controller = new AbortController()
+  const timeoutId = setTimeout(() => controller.abort(), timeoutMs)
+
+  try {
+    const response = await fetch(`${API_ENDPOINT}${path}`, {
+      method: 'POST',
+      headers: {
+        Authorization: `Bearer ${apiKey}`,
+        'Content-Type': 'application/json',
+      },
+      body: JSON.stringify(body),
+      signal: controller.signal,
+    })
+
+    if (!response.ok) {
+      const errorBody = await response.json().catch(() => ({}))
+      throw new Error(`API POST ${path} failed: ${response.status} - ${JSON.stringify(errorBody)}`)
+    }
+
+    return response.json()
+  } finally {
+    clearTimeout(timeoutId)
+  }
+}
+```
+
+### Step 1: Find or create the lens (clean up stale lenses)
+
+A stale lens from a previous run causes `"Input stream is unhealthy!"` errors. Always check for an existing lens with the same name and delete it before registering a fresh one.
+
+**Pass the user's `instruction` and `focus` into the lens config** — do not hardcode them.
+
+```typescript
+const LENS_NAME = 'camera-frame-capture-lens'
+
+// --- User-provided values (NOT hardcoded) ---
+const instruction = 'Answer the following question about the image:'
+const focus = 'Describe what you see in this image.'
+
+function buildLensConfig(instruction: string, focus: string) {
+  return {
+    lens_name: LENS_NAME,
+    lens_config: {
+      model_pipeline: [
+        { processor_name: 'lens_camera_processor', processor_config: {} },
+      ],
+      model_parameters: {
+        model_version: 'Newton::c2_4_7b_251215a172f6d7',
+        template_name: 'image_qa_template_task',
+        instruction,
+        focus,
+        max_new_tokens: 512,
+        camera_buffer_size: 1,
+        min_replicas: 1,
+        max_replicas: 1,
+      },
+    },
+  }
+}
+
+// Delete any existing lens with the same name to avoid stale state
+const existingLenses = await apiGet<Array<{ lens_id: string; lens_name: string }>>(
+  '/lens/metadata', apiKey
+)
+const staleLens = existingLenses.find(l => l.lens_name === LENS_NAME)
+if (staleLens) {
+  console.log('Deleting stale lens:', staleLens.lens_id)
+  await apiPost('/lens/delete', apiKey, { lens_id: staleLens.lens_id })
+}
+
+// Register fresh lens with user's instruction + focus
+const lensConfig = buildLensConfig(instruction, focus)
+const registeredLens = await apiPost<{ lens_id: string }>(
+  '/lens/register', apiKey, { lens_config: lensConfig }
+)
+const lensId = registeredLens.lens_id
+```
+
+### Step 2: Create session and wait for ready
+
+```typescript
+const session = await apiPost<{ session_id: string; session_endpoint: string }>(
+  '/lens/sessions/create', apiKey, { lens_id: lensId }
+)
+const sessionId = session.session_id
+
+// Wait for session to be ready (poll until status = running)
+async function waitForSessionReady(sessionId: string, maxWaitMs = 30000): Promise<boolean> {
+  const start = Date.now()
+  while (Date.now() - start < maxWaitMs) {
+    const status = await apiPost<{ session_status: string }>(
+      '/lens/sessions/events/process', apiKey,
+      { session_id: sessionId, event: { type: 'session.status' } },
+      10000
+    )
+    if (status.session_status === 'LensSessionStatus.SESSION_STATUS_RUNNING' ||
+        status.session_status === '3') {
+      return true
+    }
+    if (status.session_status === 'LensSessionStatus.SESSION_STATUS_FAILED' ||
+        status.session_status === '6') {
+      return false
+    }
+    await new Promise(r => setTimeout(r, 500))
+  }
+  return false
+}
+
+const isReady = await waitForSessionReady(sessionId)
+if (!isReady) throw new Error('Session failed to initialize')
+```
+
+### Step 3: Initialize the processor (REQUIRED for lens_camera_processor)
+
+This sends a `session.modify` event that triggers `update_lens_params()` which initializes `video_narrator_memory`. **Without this step, inference will fail.**
+
+```typescript
+await apiPost('/lens/sessions/events/process', apiKey, {
+  session_id: sessionId,
+  event: {
+    type: 'session.modify',
+    event_data: {
+      camera_buffer_size: 1,
+    },
+  },
+}, 30000) // 30s timeout for initialization
+```
+
+### Step 4: Start webcam and capture frames as base64
+
+#### 4a. Create a video element
+
+```html
+<!-- Visible preview (optional) -->
+<video id="webcam" autoplay playsinline muted></video>
+
+<!-- Or create it in JS (no visible preview) -->
+```
+
+```typescript
+// Option A: Reference an existing <video> element
+const video = document.getElementById('webcam') as HTMLVideoElement
+
+// Option B: Create a hidden video element in JS
+const video = document.createElement('video')
+video.autoplay = true
+video.playsInline = true  // Required for iOS
+video.muted = true
+```
+
+#### 4b. Request camera access and start the stream
+
+```typescript
+async function startCamera(
+  preferredWidth = 640,
+  preferredHeight = 480,
+  facingMode: 'user' | 'environment' = 'user',  // 'user' = front, 'environment' = back
+): Promise<MediaStream> {
+  const stream = await navigator.mediaDevices.getUserMedia({
+    video: {
+      width: { ideal: preferredWidth },
+      height: { ideal: preferredHeight },
+      facingMode,
+    },
+    audio: false,
+  })
+
+  video.srcObject = stream
+
+  // Wait until video is actually playing and has dimensions
+  await new Promise<void>((resolve) => {
+    video.onloadedmetadata = () => {
+      video.play()
+      resolve()
+    }
+  })
+
+  console.log(`Camera started: ${video.videoWidth}x${video.videoHeight}`)
+  return stream
+}
+
+const stream = await startCamera()
+```
+
+**Permission notes:**
+- Browser will show a permission prompt on first call
+- HTTPS is **required** (except `localhost`)
+- On mobile, `facingMode: 'environment'` selects the rear camera
+
+#### 4c. Capture a frame as base64 JPEG
+
+The flow is: **video element → canvas → toDataURL → base64 string**.
+
+```typescript
+function captureFrame(quality = 0.8): string | null {
+  if (!video.videoWidth || !video.videoHeight) return null
+
+  const canvas = document.createElement('canvas')
+  canvas.width = video.videoWidth
+  canvas.height = video.videoHeight
+
+  const ctx = canvas.getContext('2d')
+  if (!ctx) return null
+
+  // Draw current video frame onto canvas
+  ctx.drawImage(video, 0, 0)
+
+  // Convert to base64 JPEG — returns "data:image/jpeg;base64,/9j/4AAQ..."
+  return canvas.toDataURL('image/jpeg', quality)
+}
+```
+
+**Quality vs size tradeoffs:**
+| Quality | ~Size (640x480) | Use case |
+|---------|-----------------|----------|
+| `0.5` | ~20-30 KB | Fast continuous streaming |
+| `0.8` | ~40-60 KB | Good balance (recommended) |
+| `1.0` | ~80-120 KB | Maximum detail |
+
+#### 4d. Strip the data URI prefix before sending to the API
+
+The API expects **raw base64**, not the `data:image/jpeg;base64,` prefix that `toDataURL` produces.
+
+```typescript
+function captureFrameRaw(quality = 0.8): string | null {
+  const dataUri = captureFrame(quality)
+  if (!dataUri) return null
+
+  // Strip "data:image/jpeg;base64," prefix → raw base64
+  return dataUri.replace(/^data:image\/\w+;base64,/, '')
+}
+```
+
+This raw base64 string is what goes into the `model.query` event's `base64_img` field.
+
+### Step 5: Send frames for analysis (model.query)
+
+This uses **request/response — NOT SSE**. Each frame is sent as a `model.query` event and the response comes back directly in the POST response.
+
+The `instruction` and `focus` in the `model.query` event **must match** the values used at lens registration. Pass them through — do not hardcode different values.
+
+```typescript
+function createModelQueryEvent(
+  rawBase64Images: string[],  // Already stripped of data URI prefix
+  instruction: string,         // Same as lens config
+  focus: string,               // Same as lens config
+  modelVersion = 'Newton::c2_4_7b_251215a172f6d7',
+  templateName = 'image_qa_template_task',
+  maxNewTokens = 512,
+) {
+  return {
+    type: 'model.query' as const,
+    event_data: {
+      model_version: modelVersion,
+      template_name: templateName,
+      instruction,
+      focus,
+      max_new_tokens: maxNewTokens,
+      data: rawBase64Images.map(img => ({
+        type: 'base64_img',
+        base64_img: img,
+      })),
+    },
+  }
+}
+
+// Send a frame and get the response (uses the same instruction + focus from Step 1)
+async function analyzeFrame(instruction: string, focus: string): Promise<string> {
+  const frame = captureFrameRaw()  // Raw base64 (no data URI prefix)
+  if (!frame) throw new Error('Failed to capture frame')
+
+  const event = createModelQueryEvent([frame], instruction, focus)
+
+  const response = await apiPost<{
+    type: string
+    event_data?: { response?: string | string[]; message?: string }
+  }>(
+    '/lens/sessions/events/process', apiKey,
+    { session_id: sessionId, event },
+    60000  // 60s timeout for model inference
+  )
+
+  // Extract text from response
+  if (response.type === 'model.query.response' && response.event_data) {
+    const text = response.event_data.response
+    if (typeof text === 'string') return text
+    if (Array.isArray(text)) return text.join('\n')
+    return JSON.stringify(response.event_data)
+  }
+
+  return JSON.stringify(response)
+}
+```
+
+### Step 6: Continuous capture loop
+
+Send the first frame **immediately** after initialization — do not wait for the interval. The processor expects data promptly after `session.modify`.
+
+```typescript
+let isSending = false
+
+async function captureAndSend() {
+  if (isSending) {
+    console.log('Previous request still in progress, skipping frame')
+    return
+  }
+
+  isSending = true
+  try {
+    const result = await analyzeFrame(instruction, focus)
+    console.log('Result:', result)
+  } catch (error) {
+    console.error('Frame analysis failed:', error)
+  } finally {
+    isSending = false
+  }
+}
+
+// Send first frame immediately
+captureAndSend()
+
+// Then continue at 1 frame per second
+const intervalId = setInterval(captureAndSend, 1000)
+```
+
+### Step 7: Cleanup
+
+```typescript
+// Stop capture loop
+clearInterval(intervalId)
+
+// Stop camera
+stream.getTracks().forEach(track => track.stop())
+
+// Destroy session
+await apiPost('/lens/sessions/destroy', apiKey, { session_id: sessionId })
+```
+
+### Web Lifecycle Summary
+
+```
+1. List existing lenses  ->  GET  /lens/metadata
+2. Delete stale lens     ->  POST /lens/delete  { lens_id }  (if same name exists)
+3. Register fresh lens   ->  POST /lens/register  { lens_config: config }
+4. Create session        ->  POST /lens/sessions/create  { lens_id }
+5. Wait for ready        ->  POST /lens/sessions/events/process  (poll session.status)
+6. Initialize processor  ->  POST /lens/sessions/events/process  { session_id, event: session.modify }
+7. Start webcam          ->  navigator.mediaDevices.getUserMedia()
+8. Send first frame NOW  ->  POST /lens/sessions/events/process  { session_id, event: model.query }
+9. Capture loop (1fps)   ->  POST /lens/sessions/events/process  { session_id, event: model.query }
+10. Stop camera          ->  stream.getTracks().forEach(t => t.stop())
+11. Destroy session      ->  POST /lens/sessions/destroy  { session_id }
+```
+
+---
+
+## Use Cases
+
+- **Quick scene analysis**: Single frame description
+- **Presence detection**: Check if someone is at their desk
+- **Safety monitoring**: Verify safety equipment usage
+- **Object identification**: Identify specific items in view
+- **Continuous monitoring**: Stream frames with periodic analysis
+
+## Troubleshooting
+
+- **"Input stream is unhealthy!"**: Stale lens from previous run. Always delete existing lens before registering a new one (see Step 1).
+- **Camera not found**: Try different camera indices (Python) or check browser permissions (Web)
+- **API errors**: Verify API key is set correctly
+- **Session fails**: Ensure `session.modify` (Step 3) is called before sending queries
+- **Timeout on inference**: Model queries can take 10-30s; use 60s timeout
+- **Frame too large**: Use JPEG encoding with quality 0.8 to reduce payload size
+- **Requests overlap**: Gate with `isSending` flag to skip frames while previous request is in-flight