@meller/tokentalos 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Asaf Meller
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,121 @@
+# TokenTalos 🛡️
+
+**The "ORM for LLMs"** — A library-first LLM Gateway and Proxy that empowers developers to build modular, tracked, and cost-optimized prompts with built-in safety filtering and reasoning verification.
+
+Think of TokenTalos as an **Object-Relational Mapper (ORM)**, but for your Large Language Model interactions. Instead of sending raw, expensive, and potentially insecure strings to providers, you define **parameterized prompt parts** that TokenTalos binds, compresses, and secures before execution.
+
+## 🚀 Key Features
+
+### 1. Active Guard (Security & Safety)
+*   **Prompt Injection Scanning (LLM01):** Heuristic detection of jailbreaks, "Ignore instructions", and system overrides.
+*   **Secret Detection (LLM06):** Automatic scanning for API keys, AWS secrets, and high-entropy strings.
+*   **PII Redaction:** Automatic masking or rejection of sensitive data (Emails, Keys) before logging or execution.
+*   **Input Neutralization:** Automatic XML-wrapping of untrusted data with injected security instructions for the LLM.
+
+### 2. Efficiency & Cost Optimization
+*   **Semantic Caching:** Skip redundant LLM calls for identical prompt constructions.
+*   **Lossless Compression:** Automatic minification of whitespace and JSON blocks within prompts.
+*   **Cost Analysis:** Detailed token attribution and real-time provider cost comparisons.
+*   **Optimization Recommendations:** Heuristic-based suggestions for model switching and prompt pruning to reduce token weight.
+*   **Tokenizer Discrepancy Detection:** Advanced analysis of local vs. provider token counting.
+
+### 3. Intelligence & Observability
+*   **Streaming OPV (Optimized Process Verification):** Real-time analysis of "thinking tokens" to verify if reasoning is on-track or looping.
+*   **Variable Attribution:** Granular tracking of token weight per prompt part (e.g., system vs context vs query).
+*   **Engine Insights:** Heuristic-based recommendations for semantic summarization and prompt pruning.
+
+## 🚀 Quick Start (SDK)
+
+TokenTalos is primarily a developer tool. You can use it as a standalone library (direct DB access) or as a client to a remote Gateway.
+
+### Standalone Mode (Library-First)
+Ideal for local development or Node.js backends where you want Zero-Install tracking.
+
+```javascript
+import TokenTalos from 'tokentalos';
+
+const tt = new TokenTalos({
+  mode: 'standalone',
+  projectId: 'my-project-id',
+  config: {
+    // Database and Persistence
+    databaseType: 'sqlite',
+    sqlitePath: './tokentalos.db',
+
+    // Regional and Provider Settings
+    location: 'us-central1', // GCP/Vertex Region
+    
+    // Feature and Policy Configuration
+    securityFeatures: ['injection', 'secrets'],
+    formattingFeatures: ['pii', 'neutralize'],
+    intelligenceFeatures: ['cache', 'explain'],
+    piiAction: 'mask' // Automatic PII masking
+  }
+});
+
+await tt.init();
+
+const result = await tt.execute({
+  provider: 'gemini',
+  model: 'gemini-3-flash-preview',
+  parts: {
+    system: 'You are a technical writer.',
+    user_query: 'Explain TokenTalos in 20 words.'
+  }
+});
+
+console.log(result.content);
+```
+
+> **Note:** TokenTalos includes native support for Google Gemini. To use other providers (OpenAI, Anthropic, etc.), ensure you have their respective API keys configured and dependencies installed.
+
+### Proxy Mode (Gateway)
+Ideal for production environments or non-Node.js apps (PHP, Python, Go) connecting to a central TokenTalos server.
+
+```javascript
+const tt = new TokenTalos({
+  mode: 'proxy',
+  apiUrl: 'https://your-gateway.com/api/v1',
+  apiKey: 'your-secret-key'
+});
+```
+
+### 🌍 Cross-Language Support
+TokenTalos is designed as a language-agnostic Gateway. You can use standard HTTP clients in any language (PHP, Python, Go, etc.) to communicate with the TokenTalos Proxy.
+
+Check out the [examples/](./examples) directory for a **PHP cURL** example.
+
+## 🛠️ Installation
+
+```bash
+# As a project dependency
+npm install tokentalos
+
+# For CLI and Dashboard access
+npx tokentalos setup
+```
+
+## 🏗️ Architecture
+
+## 💻 CLI Usage
+
+| Command | Description |
+| :--- | :--- |
+| `tokentalos setup` | Run the interactive configuration wizard. |
+| `tokentalos start` | Start the full service (Collector + Dashboard). |
+| `tokentalos start collector` | Start only the API ingestion service. |
+| `tokentalos start dashboard` | Start only the visual interface. |
+| `tokentalos stop` | Stop all services. |
+| `tokentalos stats` | Show aggregate token and cost statistics in the terminal. |
+| `tokentalos list` | Display a table of recent prompt logs. |
+| `tokentalos export` | Export usage logs to JSONL or LangSmith formats. |
+
+## 🧩 Special Variables
+
+TokenTalos recognizes specific variable names to enable enhanced features:
+*   `safety_guardrails`: Used as ground-truth context for OPV verification.
+*   `thinking` / `reasoning`: Targeted for chain-of-thought analysis.
+*   `system` / `context` / `history`: Recognized for specialized tracking and bloating analysis.
+
+---
+*For a full list of variable behaviors, see [VARIABLES.md](./VARIABLES.md).*

package/api/api/v1/analytics.js

@@ -0,0 +1,153 @@
+import express from 'express';
+import { getDb } from '../../../lib/engine/db.js';
+import { authMiddleware } from '../../middleware/auth.js';
+import { getCostCalculator } from '../../../lib/engine/pricing.js';
+
+const router = express.Router();
+
+// GET /api/v1/analytics/stats
+router.get('/stats', authMiddleware, async (req, res) => {
+  const db = getDb();
+  const projectId = req.query.projectId;
+  const orgId = req.orgId;
+  
+  try {
+    let whereClause = ' WHERE org_id = ?';
+    let params = [orgId];
+
+    if (projectId) {
+      whereClause += ' AND project_id = ?';
+      params.push(projectId);
+    }
+
+    const totals = await db.get(`
+      SELECT 
+        SUM(total_tokens) as total_tokens,
+        SUM(total_cost) as total_cost,
+        SUM(CASE WHEN type = 'cache_hit' THEN saved_tokens ELSE 0 END) as cache_saved_tokens,
+        SUM(CASE WHEN type = 'execution' THEN saved_tokens ELSE 0 END) as compression_saved_tokens,
+        SUM(saved_cost) as total_saved_cost,
+        COUNT(id) as total_requests,
+        COUNT(CASE WHEN type = 'cache_hit' THEN 1 END) as total_cache_hits
+      FROM usage_data
+      ${whereClause}
+    `, params);
+
+    // Calculate Potential Savings (Optimization Opportunity)
+    const recentExecutions = await db.all(`
+      SELECT provider, model, input_tokens, output_tokens, total_cost 
+      FROM usage_data 
+      ${whereClause} AND type = 'execution'
+      ORDER BY timestamp DESC LIMIT 100
+    `, params);
+
+    const calculator = getCostCalculator();
+    let totalPotentialSavings = 0;
+    
+    for (const record of recentExecutions) {
+      const bestAlt = calculator.getBestAlternative(record.provider, record.model, record.input_tokens, record.output_tokens);
+      if (bestAlt && bestAlt.cost < record.total_cost) {
+        totalPotentialSavings += (record.total_cost - bestAlt.cost);
+      }
+    }
+
+    const byProvider = await db.all(`
+      SELECT provider, SUM(total_tokens) as total_tokens, SUM(total_cost) as total_cost
+      FROM usage_data 
+      ${whereClause}
+      GROUP BY provider
+    `, params);
+
+    const byModel = await db.all(`
+      SELECT model, SUM(total_tokens) as total_tokens, SUM(total_cost) as total_cost
+      FROM usage_data 
+      ${whereClause}
+      GROUP BY model
+    `, params);
+
+    const piiHits = await db.get(`
+      SELECT COUNT(*) as count FROM pii_hits
+      JOIN usage_data ON pii_hits.usage_id = usage_data.id
+      ${whereClause}
+    `, params);
+
+    const securityAlerts = await db.get(`
+      SELECT COUNT(*) as count FROM security_alerts
+      JOIN usage_data ON security_alerts.usage_id = usage_data.id
+      ${whereClause}
+    `, params);
+
+    res.json({
+      total_tokens: totals.total_tokens || 0,
+      total_cost: totals.total_cost || 0,
+      total_saved_tokens: (totals.cache_saved_tokens || 0) + (totals.compression_saved_tokens || 0),
+      cache_saved_tokens: totals.cache_saved_tokens || 0,
+      compression_saved_tokens: totals.compression_saved_tokens || 0,
+      total_cache_hits: totals.total_cache_hits || 0,
+      total_saved_cost: totals.total_saved_cost || 0,
+      total_requests: totals.total_requests || 0,
+      pii_hits: piiHits.count || 0,
+      security_alerts: securityAlerts.count || 0,
+      potential_savings: totalPotentialSavings,
+      by_provider: byProvider,
+      by_model: byModel
+    });
+  } catch (err) {
+    console.error('Stats error:', err);
+    res.status(500).json({ error: 'Failed to fetch statistics' });
+  }
+});
+
+// GET /api/v1/analytics/heatmap
+router.get('/heatmap', authMiddleware, async (req, res) => {
+  const db = getDb();
+  const days = req.query.days || 30;
+  const projectId = req.query.projectId;
+  const orgId = req.orgId;
+
+  try {
+    const timestampFilter = db.type === 'sqlite' 
+      ? `usage_data.timestamp >= datetime('now', '-${days} days')` 
+      : `usage_data.timestamp >= CURRENT_TIMESTAMP - INTERVAL '${days} days'`;
+
+    let filterClause = ` WHERE ${timestampFilter} AND usage_data.org_id = ?`;
+    let params = [orgId];
+
+    if (projectId) {
+      filterClause += ` AND usage_data.project_id = ?`;
+      params.push(projectId);
+    }
+
+    const heatmap = await db.all(`
+      SELECT 
+        name as variable_name,
+        SUM(token_count) as total_tokens,
+        COUNT(usage_id) as request_count
+      FROM prompt_variables
+      JOIN usage_data ON prompt_variables.usage_id = usage_data.id
+      ${filterClause}
+      AND name NOT IN ('system', 'context')
+      GROUP BY name
+      ORDER BY total_tokens DESC
+    `, params);
+
+    res.json({ heatmap });
+  } catch (err) {
+    console.error('Heatmap error:', err);
+    res.status(500).json({ error: 'Failed to fetch heatmap data' });
+  }
+});
+
+// GET /api/v1/analytics/projects
+router.get('/projects', authMiddleware, async (req, res) => {
+  const db = getDb();
+  const orgId = req.orgId;
+  try {
+    const projects = await db.all('SELECT DISTINCT project_id FROM usage_data WHERE org_id = ?', [orgId]);
+    res.json(projects.map(p => p.project_id));
+  } catch (err) {
+    res.status(500).json({ error: 'Failed to fetch projects' });
+  }
+});
+
+export default router;

package/api/api/v1/opv.js

@@ -0,0 +1,36 @@
+import express from 'express';
+import { TokenTalosEngine } from '../../../lib/engine/index.js';
+
+const router = express.Router();
+let config = {};
+
+export function setConfig(c) {
+  config = c;
+}
+
+// POST /api/v1/opv/heartbeat - Real-time reasoning analysis
+router.post('/heartbeat', async (req, res) => {
+  const { thinking_sample, task_description, previous_status } = req.body;
+
+  if (!thinking_sample || !task_description) {
+    return res.status(400).json({ error: 'thinking_sample and task_description are required' });
+  }
+
+  try {
+    const engine = new TokenTalosEngine(config);
+    await engine.init();
+    
+    const result = await engine.verifyReasoning({
+      thinking_sample,
+      task_description,
+      previous_status
+    });
+
+    res.json(result);
+  } catch (err) {
+    console.error('OPV heartbeat error:', err);
+    res.status(500).json({ error: 'Failed to verify reasoning', details: err.message });
+  }
+});
+
+export default router;

package/api/api/v1/usage.js

@@ -0,0 +1,318 @@
+import express from 'express';
+import { getDb } from '../../../lib/engine/db.js';
+import { v4 as uuidv4 } from 'uuid';
+import { TokenTalosPrompt } from '../../../lib/engine/parameterizer.js';
+import { getCostCalculator } from '../../../lib/engine/pricing.js';
+import { processPromptParts } from '../../../lib/engine/processor.js';
+import { detectPII } from '../../../lib/engine/pii_detector.js';
+import { getLLMGateway } from '../../../lib/engine/llm_clients.js';
+import { TokenTalosEngine } from '../../../lib/engine/index.js';
+import { authMiddleware } from '../../middleware/auth.js';
+
+const router = express.Router();
+let config = {};
+
+export function setConfig(c) {
+  config = c;
+}
+
+// POST /api/v1/usage/ingest - Ingest usage data from external sources
+router.post('/ingest', authMiddleware, async (req, res) => {
+  const db = getDb();
+  const data = req.body;
+  const usageId = uuidv4();
+  const projectId = data.projectId || 'default';
+  const orgId = req.orgId;
+
+  const provider = data.provider || config.llmProvider || 'gemini';
+  const model = data.model || config.defaultModel || 'gemini-3-flash-preview';
+
+  const totalTokens = (data.input_tokens || 0) + (data.output_tokens || 0);
+  const calculator = getCostCalculator();
+  const [inputCost, outputCost] = calculator.calculateCost(
+    provider, 
+    model, 
+    data.input_tokens || 0, 
+    data.output_tokens || 0
+  );
+  
+  const totalCost = inputCost + outputCost;
+  const limitExceeded = totalTokens > (config.maxTokens || 32000);
+
+  try {
+    await db.run(`
+      INSERT INTO usage_data (
+        id, org_id, project_id, type, provider, model, full_prompt, response_content, input_tokens, output_tokens, total_tokens, 
+        input_cost, output_cost, total_cost, endpoint, latency_ms, token_limit_exceeded, timestamp
+      ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+    `, [
+      usageId, orgId, projectId, 'ingested', provider, model, data.full_prompt || null, data.response_content || null, 
+      data.input_tokens || 0, data.output_tokens || 0,
+      totalTokens, inputCost, outputCost, totalCost, data.endpoint, data.latency_ms,
+      limitExceeded ? 1 : 0, data.timestamp || new Date().toISOString()
+    ]);
+
+    if (data.variables && Array.isArray(data.variables)) {
+      for (const v of data.variables) {
+        await db.run(`
+          INSERT INTO prompt_variables (usage_id, name, content, original_content, token_count, char_count, position)
+          VALUES (?, ?, ?, ?, ?, ?, ?)
+        `, [
+          usageId, 
+          v.name, 
+          v.content || '', 
+          v.original_content || v.content || '',
+          v.token_count || 0, 
+          v.char_count || 0, 
+          v.position || 0
+        ]);
+
+        if (config.formattingFeatures && config.formattingFeatures.includes('pii')) {
+          const findings = detectPII(v.content);
+          if (findings.length > 0) {
+            for (const finding of findings) {
+              await db.run(`
+                INSERT INTO pii_hits (usage_id, variable_name, pii_type, action_taken)
+                VALUES (?, ?, ?, ?)
+              `, [usageId, v.name, finding.type, 'ingested_warning']);
+            }
+          }
+        }
+      }
+    }
+
+    if (data.actions_taken && Array.isArray(data.actions_taken)) {
+      for (const action of data.actions_taken) {
+        await db.run(`
+          INSERT INTO variable_actions (usage_id, variable_name, action_type, action_method, details)
+          VALUES (?, ?, ?, ?, ?)
+        `, [
+          usageId, 
+          action.target, 
+          action.type, 
+          action.method || null, 
+          JSON.stringify(action)
+        ]);
+      }
+    }
+
+    res.status(201).json({ id: usageId, message: 'Usage data ingested successfully' });
+  } catch (err) {
+    console.error('Ingestion error:', err);
+    res.status(500).json({ error: 'Failed to ingest usage data' });
+  }
+});
+
+// GET /api/v1/usage/recent - Get recent usage records
+router.get('/recent', authMiddleware, async (req, res) => {
+  const db = getDb();
+  const limit = req.query.limit || 50;
+  const projectId = req.query.projectId;
+  const orgId = req.orgId;
+
+  try {
+    let sql = 'SELECT * FROM usage_data WHERE org_id = ?';
+    let params = [orgId];
+    
+    if (projectId) {
+      sql += ' AND project_id = ?';
+      params.push(projectId);
+    }
+    
+    sql += ' ORDER BY timestamp DESC LIMIT ?';
+    params.push(parseInt(limit));
+
+    const usageRecords = await db.all(sql, params);
+
+    for (const record of usageRecords) {
+      record.variables = await db.all(`
+        SELECT name, content, original_content, token_count, char_count, position 
+        FROM prompt_variables 
+        WHERE usage_id = ?
+      `, [record.id]);
+      
+      record.explain_plan = await db.get(`
+        SELECT * FROM explain_plans WHERE usage_id = ?
+      `, [record.id]);
+
+      if (record.explain_plan) {
+        if (record.explain_plan.variable_analysis) record.explain_plan.variable_analysis = JSON.parse(record.explain_plan.variable_analysis);
+        if (record.explain_plan.detected_issues) record.explain_plan.detected_issues = JSON.parse(record.explain_plan.detected_issues);
+        if (record.explain_plan.optimization_suggestions) record.explain_plan.optimization_suggestions = JSON.parse(record.explain_plan.optimization_suggestions);
+        
+        // On-the-fly MCE calculation if missing from DB (for existing records)
+        if (!record.explain_plan.mce_best_alternative_model) {
+          const calculator = getCostCalculator();
+          const bestAlt = calculator.getBestAlternative(record.provider, record.model, record.input_tokens, record.output_tokens);
+          if (bestAlt) {
+            const savingsPct = record.total_cost > 0 ? ((record.total_cost - bestAlt.cost) / record.total_cost) * 100 : 0;
+            if (savingsPct > 10) {
+              record.explain_plan.mce_best_alternative_model = bestAlt.model;
+              record.explain_plan.mce_best_alternative_provider = bestAlt.provider;
+              record.explain_plan.mce_best_alternative_cost = bestAlt.cost;
+              record.explain_plan.mce_savings_pct = savingsPct;
+            }
+          }
+        }
+      }
+    }
+
+    res.json(usageRecords);
+  } catch (err) {
+    console.error('Recent usage error:', err);
+    res.status(500).json({ error: 'Failed to fetch recent usage' });
+  }
+});
+
+// POST /api/v1/usage/execute - Gateway execution
+router.post('/execute', authMiddleware, async (req, res) => {
+  try {
+    const engine = new TokenTalosEngine(config);
+    await engine.init();
+    
+    const result = await engine.execute({
+      ...req.body,
+      orgId: req.orgId,
+      projectId: req.body.projectId || req.query.projectId,
+      bypassCache: req.query.bypassCache === 'true'
+    });
+
+    res.json(result);
+  } catch (err) {
+    console.error('Execution error:', err);
+    res.status(500).json({ error: 'Failed to execute prompt', details: err.message });
+  }
+});
+
+// POST /api/v1/usage/prompt/construct - Active orchestration
+router.post('/prompt/construct', authMiddleware, async (req, res) => {
+  const { provider, model, parts, endpoint, projectId } = req.body;
+  const orgId = req.orgId;
+  
+  const { processedParts, metadata } = await processPromptParts(parts, config);
+
+  const finalProvider = provider || config.llmProvider || 'gemini';
+  const finalModel = model || config.defaultModel || 'gemini-3-flash-preview';
+
+  const prompt = new TokenTalosPrompt(finalProvider, finalModel);
+  
+  for (const key in processedParts) {
+    if (key === 'system') prompt.addSystem(processedParts[key], parts[key]);
+    else if (key === 'context') prompt.addContext(processedParts[key], parts[key]);
+    else if (key === 'history') prompt.addHistory(processedParts[key], parts[key]);
+    else if (key === 'user_query') prompt.addUserQuery(processedParts[key], parts[key]);
+    else prompt.add(key, processedParts[key], parts[key]);
+  }
+
+  const messages = prompt.toMessages();
+  const trackingData = prompt.getTrackingData();
+  
+  const maxTokens = config.maxTokens || 32000;
+  const thresholdAction = config.thresholdAction || 'warning';
+
+  if (trackingData.total_tokens > maxTokens && thresholdAction === 'reject') {
+    return res.status(400).json({
+      error: 'Token limit exceeded',
+      total_tokens: trackingData.total_tokens,
+      max_tokens: maxTokens,
+      message: 'Construction rejected by policy. Truncate parts or increase limit in setup.'
+    });
+  }
+
+  const db = getDb();
+  const calculator = getCostCalculator();
+  const [inputCost] = calculator.calculateCost(finalProvider, finalModel, trackingData.total_tokens, 0);
+
+  const limitExceeded = trackingData.total_tokens > maxTokens;
+  const finalProjectId = projectId || 'default';
+
+  try {
+    await db.run(`
+      INSERT INTO usage_data (id, org_id, project_id, type, provider, model, input_tokens, total_tokens, input_cost, total_cost, endpoint, token_limit_exceeded, timestamp)
+      VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+    `, [
+      trackingData.id, orgId, finalProjectId, 'construction', finalProvider, finalModel, trackingData.total_tokens, trackingData.total_tokens,
+      inputCost, inputCost, endpoint, limitExceeded ? 1 : 0, trackingData.timestamp
+    ]);
+
+    for (const v of trackingData.variables) {
+      await db.run(`
+        INSERT INTO prompt_variables (usage_id, name, content, original_content, token_count, char_count, position)
+        VALUES (?, ?, ?, ?, ?, ?, ?)
+      `, [trackingData.id, v.name, v.content, v.original_content, v.token_count, v.char_count, v.position]);
+    }
+
+    for (const action of metadata.actions_taken) {
+      // ... same generic action log ...
+      await db.run(`
+        INSERT INTO variable_actions (usage_id, variable_name, action_type, action_method, details)
+        VALUES (?, ?, ?, ?, ?)
+      `, [
+        trackingData.id, 
+        action.target, 
+        action.type, 
+        action.method || null, 
+        JSON.stringify(action)
+      ]);
+
+      // 2. Legacy PII hit log (for backward compat/dashboards)
+      if (action.type === 'pii') {
+        for (const finding of action.findings) {
+          await db.run(`
+            INSERT INTO pii_hits (usage_id, variable_name, pii_type, action_taken)
+            VALUES (?, ?, ?, ?)
+          `, [trackingData.id, action.target, finding.type, action.method]);
+        }
+      }
+    }
+
+    // 3. Heuristic Analysis
+    const analysis = runHeuristicAnalysis({ 
+      total_tokens: trackingData.total_tokens, 
+      total_cost: inputCost,
+      provider: finalProvider,
+      model: finalModel
+    }, trackingData.variables);
+
+    if (analysis) {
+      const planId = uuidv4();
+      await db.run(`
+        INSERT INTO explain_plans (
+          id, usage_id, variable_analysis, detected_issues, optimization_suggestions, 
+          estimated_savings_pct, estimated_savings_usd,
+          mce_best_alternative_model, mce_best_alternative_provider, mce_best_alternative_cost, mce_savings_pct
+        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+      `, [
+        planId, 
+        trackingData.id, 
+        JSON.stringify(analysis.variable_analysis), 
+        JSON.stringify(analysis.detected_issues), 
+        JSON.stringify(analysis.optimization_suggestions),
+        analysis.estimated_savings_pct, 
+        analysis.estimated_savings_usd,
+        analysis.mce_best_alternative_model || null,
+        analysis.mce_best_alternative_provider || null,
+        analysis.mce_best_alternative_cost || 0,
+        analysis.mce_savings_pct || 0
+      ]);
+    }
+
+    res.json({
+      id: trackingData.id,
+      messages: messages,
+      full_prompt_string: prompt.toString(),
+      total_tokens: trackingData.total_tokens,
+      estimated_input_cost: inputCost,
+      token_limit_exceeded: limitExceeded,
+      max_tokens: maxTokens,
+      threshold_action: thresholdAction,
+      checks_run: metadata.checks_run,
+      actions_taken: metadata.actions_taken
+    });
+  } catch (err) {
+    console.error('Construction error:', err);
+    res.status(500).json({ error: 'Failed to construct prompt' });
+  }
+});
+
+export default router;