npm - @nextsparkjs/core - Versions diffs - 0.1.0-beta.3 → 0.1.0-beta.4 - Mend

@nextsparkjs/core 0.1.0-beta.3 → 0.1.0-beta.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (1780) hide show

package/dist/templates/contents/plugins/langchain/docs/04-advanced/01-observability.md DELETED Viewed

@@ -1,645 +0,0 @@
-# Observability & Tracing
-This guide covers the observability system for LangChain agents, including tracing, metrics, and the debugging dashboard.
-## Overview
-The observability system provides **end-to-end visibility** into AI agent execution:
-```
-User Request → Trace
-    ├── Router Span (LLM call)
-    ├── Task Handler Span (tool call)
-    ├── Customer Handler Span (tool call)
-    └── Combiner Span (LLM call)
-```
-**Key Features**:
-- **Traces**: Track complete agent invocations
-- **Spans**: Track individual operations within traces
-- **Metrics**: Aggregate performance data
-- **Dashboard**: Visual debugging interface
-- **PII Protection**: Mask sensitive data automatically
----
-## Architecture
-### Tracing Flow
-```
-┌─────────────────────────────────────────────────────────────┐
-│                    AGENT INVOCATION                          │
-│                                                              │
-│  ┌──────────┐    ┌──────────┐    ┌──────────┐              │
-│  │   LLM    │    │   Tool   │    │   LLM    │              │
-│  │  Call    │───►│   Call   │───►│   Call   │              │
-│  └──────────┘    └──────────┘    └──────────┘              │
-│       │               │               │                     │
-│       ▼               ▼               ▼                     │
-│  ┌──────────────────────────────────────────────┐          │
-│  │              TRACE RECORD                     │          │
-│  │  ├─ Span: router (llm)                       │          │
-│  │  ├─ Span: task_handler (tool)                │          │
-│  │  └─ Span: combiner (llm)                     │          │
-│  └──────────────────────────────────────────────┘          │
-│                         │                                   │
-│                         ▼                                   │
-│                ┌─────────────┐                             │
-│                │  DATABASE   │                             │
-│                │  (RLS)      │                             │
-│                └─────────────┘                             │
-└─────────────────────────────────────────────────────────────┘
-```
-### Database Schema
-Two tables with full Row-Level Security:
-**`langchain_traces`** - Main trace records
-| Column | Type | Description |
-|--------|------|-------------|
-| `traceId` | TEXT | Unique identifier |
-| `userId` | TEXT | User who triggered |
-| `teamId` | TEXT | Team context |
-| `sessionId` | TEXT | Session identifier |
-| `agentName` | TEXT | Agent name |
-| `input` | TEXT | User input (truncated) |
-| `output` | TEXT | Agent output (truncated) |
-| `status` | TEXT | running/success/error |
-| `error` | TEXT | Error message |
-| `durationMs` | INTEGER | Execution time |
-| `inputTokens` | INTEGER | Input token count |
-| `outputTokens` | INTEGER | Output token count |
-| `totalCost` | DECIMAL | Cost in USD |
-| `llmCalls` | INTEGER | Number of LLM calls |
-| `toolCalls` | INTEGER | Number of tool calls |
-| `metadata` | JSONB | Additional data |
-| `tags` | TEXT[] | Filtering tags |
-**`langchain_spans`** - Individual operations
-| Column | Type | Description |
-|--------|------|-------------|
-| `spanId` | TEXT | Unique identifier |
-| `traceId` | TEXT | Parent trace |
-| `parentSpanId` | TEXT | Parent span (for nesting) |
-| `name` | TEXT | Span name |
-| `type` | TEXT | llm/tool/chain |
-| `provider` | TEXT | LLM provider |
-| `model` | TEXT | Model name |
-| `toolName` | TEXT | Tool name (if tool) |
-| `input` | JSONB | Span input |
-| `output` | JSONB | Span output |
-| `status` | TEXT | running/success/error |
-| `durationMs` | INTEGER | Span duration |
-| `depth` | INTEGER | Nesting depth |
----
-## Configuration
-Configure in `langchain.config.ts`:
-```typescript
-export const langchainConfig = {
-  observability: {
-    enabled: true,
-    retention: {
-      traces: 30,  // Days to keep traces
-    },
-    sampling: {
-      rate: 1.0,            // 1.0 = 100% of requests
-      alwaysTraceErrors: true,  // Always trace errors
-    },
-    pii: {
-      maskInputs: false,     // Mask PII in inputs
-      maskOutputs: true,     // Mask PII in outputs
-      truncateAt: 10000,     // Max content length
-    },
-  },
-}
-```
-### Configuration Options
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `enabled` | boolean | `true` | Enable/disable tracing |
-| `retention.traces` | number | `30` | Days to keep trace data |
-| `sampling.rate` | number | `1.0` | Sample rate (0.0-1.0) |
-| `sampling.alwaysTraceErrors` | boolean | `true` | Always trace errors |
-| `pii.maskInputs` | boolean | `false` | Mask PII in inputs |
-| `pii.maskOutputs` | boolean | `true` | Mask PII in outputs |
-| `pii.truncateAt` | number | `10000` | Max content length |
-### Sampling
-Control how many requests are traced:
-```typescript
-// Development: trace everything
-sampling: { rate: 1.0, alwaysTraceErrors: true }
-// Production: trace 10% + all errors
-sampling: { rate: 0.1, alwaysTraceErrors: true }
-// High traffic: trace 1% + errors
-sampling: { rate: 0.01, alwaysTraceErrors: true }
-```
----
-## Tracer Service
-### Basic Usage
-```typescript
-import { tracer } from '@/contents/plugins/langchain/lib/tracer'
-// Start a trace
-const traceContext = await tracer.startTrace(
-  { userId, teamId },
-  'orchestrator',
-  userMessage,
-  { sessionId, metadata: { source: 'chat' } }
-)
-try {
-  // Your agent logic here
-  const result = await processMessage(userMessage)
-  // End trace with success
-  await tracer.endTrace(
-    { userId, teamId },
-    traceContext.traceId,
-    {
-      output: result.content,
-      tokens: { input: 100, output: 50, total: 150 },
-      cost: 0.002,
-      llmCalls: 2,
-      toolCalls: 1,
-    }
-  )
-} catch (error) {
-  // End trace with error
-  await tracer.endTrace(
-    { userId, teamId },
-    traceContext.traceId,
-    { error }
-  )
-}
-```
-### Spans
-Track individual operations within a trace:
-```typescript
-// Start a span
-const spanContext = await tracer.startSpan(
-  { userId, teamId },
-  traceId,
-  {
-    name: 'router',
-    type: 'llm',
-    provider: 'anthropic',
-    model: 'claude-3-haiku',
-    input: { message: userInput },
-  }
-)
-try {
-  const result = await routerNode(state)
-  // End span with success
-  await tracer.endSpan(
-    { userId, teamId },
-    traceId,
-    spanContext.spanId,
-    {
-      output: { intents: result.intents },
-      tokens: { input: 50, output: 20 },
-    }
-  )
-} catch (error) {
-  await tracer.endSpan(
-    { userId, teamId },
-    traceId,
-    spanContext.spanId,
-    { error }
-  )
-}
-```
-### Span Types
-| Type | Description | Example |
-|------|-------------|---------|
-| `llm` | LLM call | Router, Combiner |
-| `tool` | Tool invocation | TaskHandler, CustomerHandler |
-| `chain` | Chain/workflow | Graph execution |
-| `retriever` | RAG retrieval | Vector search |
----
-## API Endpoints
-All endpoints require **superadmin** authentication.
-### List Traces
-```
-GET /api/v1/theme/default/observability/traces
-```
-**Query Parameters**:
-- `limit` - Results per page (default: 50)
-- `offset` - Pagination offset
-- `status` - Filter by status (success/error)
-- `agentName` - Filter by agent
-- `teamId` - Filter by team
-- `from` - Start date (ISO)
-- `to` - End date (ISO)
-**Response**:
-```json
-{
-  "success": true,
-  "data": {
-    "traces": [
-      {
-        "traceId": "abc-123",
-        "agentName": "orchestrator",
-        "status": "success",
-        "durationMs": 2340,
-        "inputTokens": 150,
-        "outputTokens": 80,
-        "totalCost": 0.002,
-        "llmCalls": 2,
-        "toolCalls": 1,
-        "startedAt": "2024-12-23T10:00:00Z"
-      }
-    ],
-    "total": 156,
-    "hasMore": true
-  }
-}
-```
-### Get Trace Detail
-```
-GET /api/v1/theme/default/observability/traces/[traceId]
-```
-**Response**:
-```json
-{
-  "success": true,
-  "data": {
-    "trace": {
-      "traceId": "abc-123",
-      "input": "Show my tasks",
-      "output": "Found 3 tasks...",
-      "status": "success",
-      "durationMs": 2340,
-      "spans": [
-        {
-          "spanId": "span-1",
-          "name": "router",
-          "type": "llm",
-          "provider": "anthropic",
-          "model": "claude-3-haiku",
-          "durationMs": 1200,
-          "status": "success"
-        },
-        {
-          "spanId": "span-2",
-          "name": "task_handler",
-          "type": "tool",
-          "durationMs": 340,
-          "status": "success"
-        }
-      ]
-    }
-  }
-}
-```
-### Get Metrics
-```
-GET /api/v1/theme/default/observability/metrics
-```
-**Query Parameters**:
-- `period` - Time period: `1h`, `24h`, `7d`, `30d`
-**Response**:
-```json
-{
-  "success": true,
-  "data": {
-    "summary": {
-      "totalTraces": 1500,
-      "successRate": 98.5,
-      "avgLatencyMs": 2340,
-      "totalCost": 15.23,
-      "totalLlmCalls": 2800,
-      "totalToolCalls": 1200
-    },
-    "byAgent": [
-      {
-        "agentName": "orchestrator",
-        "count": 1500,
-        "successRate": 98.5,
-        "avgLatencyMs": 2340
-      }
-    ],
-    "byStatus": {
-      "success": 1478,
-      "error": 22
-    }
-  }
-}
-```
----
-## Dashboard
-The observability dashboard is available at:
-```
-/sector7/ai-observability
-```
-> **Note**: This is a superadmin-only page in the Sector7 protected zone.
-### Features
-1. **Traces Table**
-   - Paginated list of all traces
-   - Filter by status, agent, date range
-   - Click to view details
-2. **Trace Detail**
-   - Full trace timeline
-   - All spans with durations
-   - Input/output inspection
-   - Error details
-3. **Metrics Overview**
-   - Total traces
-   - Success rate
-   - Average latency
-   - Cost tracking
-### Components
-| Component | Purpose |
-|-----------|---------|
-| `ObservabilityDashboard` | Main dashboard container |
-| `TracesTable` | Paginated trace list |
-| `TraceDetail` | Single trace view |
-| `SpansList` | Span timeline |
-| `FiltersPanel` | Filter controls |
-| `TraceStatusBadge` | Status indicator |
----
-## React Hooks
-### useTraces
-Fetch paginated trace list:
-```typescript
-import { useTraces } from '@/contents/plugins/langchain/hooks/observability'
-function TracesList() {
-  const { data, isLoading, error } = useTraces({
-    status: 'error',
-    agentName: 'orchestrator',
-    limit: 20,
-  })
-  if (isLoading) return <Loading />
-  if (error) return <Error />
-  return (
-    <table>
-      {data.traces.map(trace => (
-        <TraceRow key={trace.traceId} trace={trace} />
-      ))}
-    </table>
-  )
-}
-```
-### useTraceDetail
-Fetch single trace with spans:
-```typescript
-import { useTraceDetail } from '@/contents/plugins/langchain/hooks/observability'
-function TraceView({ traceId }) {
-  const { data, isLoading } = useTraceDetail(traceId)
-  if (isLoading) return <Loading />
-  return (
-    <div>
-      <h2>{data.trace.agentName}</h2>
-      <pre>{data.trace.input}</pre>
-      <SpanTimeline spans={data.trace.spans} />
-    </div>
-  )
-}
-```
-### useMetrics
-Fetch aggregate metrics:
-```typescript
-import { useMetrics } from '@/contents/plugins/langchain/hooks/observability'
-function MetricsDashboard() {
-  const { data } = useMetrics({ period: '24h' })
-  return (
-    <div>
-      <Stat label="Total Traces" value={data.summary.totalTraces} />
-      <Stat label="Success Rate" value={`${data.summary.successRate}%`} />
-      <Stat label="Avg Latency" value={`${data.summary.avgLatencyMs}ms`} />
-    </div>
-  )
-}
-```
----
-## PII Protection
-### Automatic Masking
-When enabled, the tracer masks common PII patterns:
-| Pattern | Example | Masked |
-|---------|---------|--------|
-| Email | john@example.com | [EMAIL] |
-| Phone | +1 (555) 123-4567 | [PHONE] |
-| Credit Card | 1234-5678-9012-3456 | [CARD] |
-| SSN | 123-45-6789 | [SSN] |
-### Configuration
-```typescript
-pii: {
-  maskInputs: false,   // Don't mask inputs (for debugging)
-  maskOutputs: true,   // Mask PII in outputs
-  truncateAt: 10000,   // Truncate long content
-}
-```
-### Content Truncation
-Long content is automatically truncated:
-```
-Original: "Very long message..." (15000 chars)
-Stored:   "Very long message..." (10000 chars) + "[truncated]"
-```
----
-## Integration with Graph Orchestrator
-The graph orchestrator automatically tracks traces and spans:
-```typescript
-// In orchestrator.ts
-const traceContext = await tracer.startTrace(
-  { userId: context.userId, teamId: context.teamId },
-  'graph-orchestrator',
-  message,
-  { sessionId }
-)
-const result = await invokeOrchestrator(
-  message,
-  sessionId,
-  context,
-  history,
-  { traceId: traceContext?.traceId }
-)
-// Router node creates spans automatically
-// Handler nodes create spans automatically
-// Combiner node creates spans automatically
-await tracer.endTrace(
-  { userId, teamId },
-  traceContext.traceId,
-  {
-    output: result.finalResponse,
-    llmCalls: 2,
-    toolCalls: result.completedHandlers.length,
-  }
-)
-```
----
-## Best Practices
-### 1. Use Meaningful Span Names
-```typescript
-// Good
-{ name: 'router', type: 'llm' }
-{ name: 'task_handler', type: 'tool' }
-// Bad
-{ name: 'step1', type: 'llm' }
-{ name: 'process', type: 'tool' }
-```
-### 2. Include Provider/Model Info
-```typescript
-await tracer.startSpan(context, traceId, {
-  name: 'router',
-  type: 'llm',
-  provider: 'anthropic',  // Include this
-  model: 'claude-3-haiku', // Include this
-})
-```
-### 3. Always End Traces
-```typescript
-try {
-  // Logic
-  await tracer.endTrace(ctx, traceId, { output })
-} catch (error) {
-  await tracer.endTrace(ctx, traceId, { error })  // Always end
-  throw error
-}
-```
-### 4. Use Tags for Filtering
-```typescript
-await tracer.startTrace(context, agent, input, {
-  tags: ['production', 'chat', 'high-priority'],
-})
-```
-### 5. Sample in Production
-```typescript
-// Production config
-sampling: {
-  rate: 0.1,  // Only trace 10%
-  alwaysTraceErrors: true,  // But always trace errors
-}
-```
----
-## Troubleshooting
-### Traces Not Appearing
-1. Check `enabled: true` in config
-2. Verify sampling rate
-3. Check superadmin permissions
-4. Look for console errors
-### Missing Spans
-1. Ensure `startSpan` is called before `endSpan`
-2. Check traceId is passed correctly
-3. Verify RLS policies allow insert
-### High Storage Usage
-1. Reduce `retention.traces`
-2. Lower `pii.truncateAt`
-3. Reduce `sampling.rate`
----
-## Related Documentation
-- [Graph Orchestrator](../03-orchestration/01-graph-orchestrator.md) - How tracing integrates with the graph
-- [Token Tracking](./02-token-tracking.md) - Cost and usage tracking
-- [Configuration](../01-getting-started/03-configuration.md) - Full configuration reference