tokentrace 0.7.0 → 0.8.1

package/CHANGELOG.md

@@ -2,6 +2,42 @@
 
 All notable changes to TokenTrace are documented here.
 
+## [0.8.1] - 2026-05-13
+
+### Fixed
+
+- Codex CLI imports now read exact `token_count` totals, including cached input and reasoning output tokens, so cleared Codex sessions are no longer undercounted.
+- Claude Code and Codex session artifacts can exceed the generic file-size cap without being skipped during discovery.
+- OpenAI-style, Claude-style, and generic usage parsers now normalize cached, cache-write, and reasoning token fields without double-counting them.
+- Parser-version-aware rescans now reprocess stale imports, and source-file replacement is atomic so a failed replacement cannot delete prior trusted sessions.
+- Unknown-cost cause summaries now assign each interaction to one primary cause instead of overlapping buckets.
+- Pricing, scan, settings, repair, and pricing-refresh APIs now reject malformed JSON with clean 400 responses.
+- Manual pricing saves now reject blank model/provider names and invalid numeric prices instead of silently storing unknown prices.
+- CLI commands now reject unknown flags across scan, pricing refresh, status, doctor, digest, and insights commands.
+- Local release checks now run ProjScan through `projscan@latest`, matching the GitHub security workflow guardrail.
+
+## [0.8.0] - 2026-05-12
+
+### Added
+
+- Evidence detail pages and `tokentrace evidence --json` for tracing metric totals back to sessions, source files, parser status, and pricing context.
+- Unknown Cost Repair workbench and `tokentrace repair --json` for grouped local repair state, alias hints, parser review links, and pricing follow-up.
+- Parser Trust Report and Scan History Diff panels in Diagnostics for latest scan parser coverage and scan-to-scan import changes.
+
+### Changed
+
+- Overview metric cards now link major totals to evidence trails and route unknown cost work to Unknown Cost Repair.
+- Overview now places Token Trend and Cost Trend directly after Usage Pulse and the metric cards, with Monthly Guardrails and Recommended Next Actions below the charts.
+- Dense evidence, repair, parser trust, and scan diff tables preserve horizontal scrolling and stable source-path truncation.
+- Evidence and repair copy now states local-first behavior, support-file ignores, and parser-review requirements for unsupported files.
+- README screenshots now use public-safe synthetic Evidence + Repair views, and obsolete screenshot assets were removed from the package payload.
+
+### Fixed
+
+- Overview custom period date fields now use an intentionally inset calendar icon while preserving native `type="date"` submission fields and the single-line desktop toolbar.
+- The app shell now constrains page width on small screens so wide toolbars scroll internally instead of widening the whole page.
+- `tokentrace statusline setup claude` and piped Claude status-line input no longer touch the TokenTrace app database or start the dashboard.
+
 ## [0.7.0] - 2026-05-12
 
 ### Added

package/README.md

@@ -8,7 +8,7 @@ Local-first analytics for AI CLI usage. TokenTrace scans local CLI logs, normali
 
 TokenTrace is designed for local development machines first, with macOS-oriented defaults. It does not require a cloud account and does not send telemetry or logs anywhere.
 
-![TokenTrace 0.7.0 overview dashboard](docs/assets/overview-0.7.0.png)
+![TokenTrace overview dashboard](docs/assets/overview-0.8.0.png)
 
 ## Start In Seconds
 
@@ -37,6 +37,10 @@ tokentrace serve --port 3210 --no-open
 tokentrace scan         # Scan local AI CLI usage logs
 tokentrace doctor --json
                         # Inspect scan health and repair recommendations
+tokentrace evidence --json
+                        # Print metric evidence trails as JSON
+tokentrace repair --json
+                        # Print unknown-cost repair groups as JSON
 tokentrace digest --json
                         # Print current-month local usage digest
 tokentrace insights --json
@@ -184,6 +188,8 @@ tokentrace statusline claude
 
 Claude Code sends session JSON to the command on stdin. TokenTrace reads the transcript path, model, context usage, and session cost, then prints one compact local line:
 
+Do not set the Claude Code `statusLine.command` to plain `tokentrace`. Plain `tokentrace` starts the dashboard, while `tokentrace statusline claude` prints exactly one status-line response.
+
 ![TokenTrace Claude Code status line](docs/assets/claude-statusline.svg)
 
 You can also inspect the same local status outside Claude Code:
@@ -197,21 +203,15 @@ Codex CLI status-line integration is intentionally deferred until its status-lin
 
 ## Screenshots
 
-Usage Intelligence views from `0.7.0`:
-
-![TokenTrace 0.7.0 overview dashboard](docs/assets/overview-0.7.0.png)
-
-![TokenTrace 0.7.0 Usage Intelligence review queue](docs/assets/usage-intelligence-0.7.0.png)
-
-![TokenTrace 0.7.0 session comparison flags](docs/assets/sessions-0.7.0.png)
+Evidence + Repair views:
 
-![TokenTrace 0.7.0 project signals](docs/assets/projects-0.7.0.png)
+![TokenTrace overview dashboard](docs/assets/overview-0.8.0.png)
 
-![TokenTrace 0.7.0 local guardrail settings](docs/assets/settings-guardrails-0.7.0.png)
+![TokenTrace processed tokens evidence trail](docs/assets/evidence-0.8.0.png)
 
-Stable Daily Tool views from `0.6.0`:
+![TokenTrace unknown cost repair queue](docs/assets/repair-0.8.0.png)
 
-![TokenTrace 0.6.0 Scan Doctor](docs/assets/doctor-0.6.0.png)
+![TokenTrace Scan Doctor parser trust report](docs/assets/doctor-parser-trust-0.8.0.png)
 
 CLI startup and help:
 
@@ -225,24 +225,6 @@ Optional wrapper diagnostics:
 
 ![TokenTrace wrapper command](docs/assets/cli-wrapper.gif)
 
-Session exploration:
-
-![TokenTrace session explorer](docs/assets/session-explorer.png)
-
-Scan Doctor and file discovery:
-
-![TokenTrace ingestion diagnostics](docs/assets/diagnostics.png)
-
-![TokenTrace file discovery](docs/assets/discovery.png)
-
-Editable model pricing:
-
-![TokenTrace pricing configuration](docs/assets/pricing.png)
-
-Mobile overview:
-
-![TokenTrace mobile overview](docs/assets/mobile-overview.png)
-
 ## Privacy Model
 
 - All processing runs locally on your machine.

package/app/api/prices/refresh/route.ts

@@ -1,10 +1,15 @@
 import { NextResponse } from "next/server";
+import { readJsonObject } from "@/src/lib/api-json";
 import { refreshPricing } from "@/src/lib/pricing-refresh";
 
 export const dynamic = "force-dynamic";
 
 export async function POST(request: Request) {
-  const body = await request.json().catch(() => ({}));
+  const parsed = await readJsonObject(request);
+  if (!parsed.ok) {
+    return NextResponse.json({ error: parsed.error }, { status: 400 });
+  }
+  const body = parsed.body;
   const result = await refreshPricing({
     source: body?.source === "bundled" ? "bundled" : "remote",
     force: Boolean(body?.force)

package/app/api/prices/route.ts

@@ -1,12 +1,21 @@
 import { NextResponse } from "next/server";
+import { readJsonObject } from "@/src/lib/api-json";
 import { getPricingRows, upsertPricing } from "@/src/lib/pricing";
 
 export const dynamic = "force-dynamic";
 
-function nullableNumber(value: unknown) {
-  if (value === "" || value == null) return null;
+function requiredText(value: unknown) {
+  return typeof value === "string" ? value.trim() : "";
+}
+
+function nullablePrice(value: unknown, field: string) {
+  if (value == null) return { ok: true as const, value: null };
+  if (typeof value === "string" && value.trim() === "") return { ok: true as const, value: null };
   const number = Number(value);
-  return Number.isFinite(number) ? number : null;
+  if (!Number.isFinite(number) || number < 0) {
+    return { ok: false as const, error: `${field} must be a non-negative number or empty` };
+  }
+  return { ok: true as const, value: number };
 }
 
 export async function GET() {
@@ -14,20 +23,34 @@ export async function GET() {
 }
 
 export async function POST(request: Request) {
-  const body = await request.json();
-  if (!body.providerId || !body.model) {
+  const parsed = await readJsonObject(request);
+  if (!parsed.ok) {
+    return NextResponse.json({ error: parsed.error }, { status: 400 });
+  }
+  const body = parsed.body;
+  const providerId = requiredText(body.providerId);
+  const model = requiredText(body.model);
+  if (!providerId || !model) {
     return NextResponse.json({ error: "providerId and model are required" }, { status: 400 });
   }
+  const inputTokenPrice = nullablePrice(body.inputTokenPrice, "inputTokenPrice");
+  const outputTokenPrice = nullablePrice(body.outputTokenPrice, "outputTokenPrice");
+  const cachedInputTokenPrice = nullablePrice(body.cachedInputTokenPrice, "cachedInputTokenPrice");
+  const cacheWriteTokenPrice = nullablePrice(body.cacheWriteTokenPrice, "cacheWriteTokenPrice");
+  if (!inputTokenPrice.ok) return NextResponse.json({ error: inputTokenPrice.error }, { status: 400 });
+  if (!outputTokenPrice.ok) return NextResponse.json({ error: outputTokenPrice.error }, { status: 400 });
+  if (!cachedInputTokenPrice.ok) return NextResponse.json({ error: cachedInputTokenPrice.error }, { status: 400 });
+  if (!cacheWriteTokenPrice.ok) return NextResponse.json({ error: cacheWriteTokenPrice.error }, { status: 400 });
 
   const id = upsertPricing({
-    providerId: String(body.providerId),
-    providerName: body.providerName ? String(body.providerName) : undefined,
-    model: String(body.model),
-    inputTokenPrice: nullableNumber(body.inputTokenPrice),
-    outputTokenPrice: nullableNumber(body.outputTokenPrice),
-    cachedInputTokenPrice: nullableNumber(body.cachedInputTokenPrice),
-    cacheWriteTokenPrice: nullableNumber(body.cacheWriteTokenPrice),
-    currency: body.currency ? String(body.currency) : "USD"
+    providerId,
+    providerName: requiredText(body.providerName) || undefined,
+    model,
+    inputTokenPrice: inputTokenPrice.value,
+    outputTokenPrice: outputTokenPrice.value,
+    cachedInputTokenPrice: cachedInputTokenPrice.value,
+    cacheWriteTokenPrice: cacheWriteTokenPrice.value,
+    currency: requiredText(body.currency) || "USD"
   });
 
   return NextResponse.json({ id, costsRecalculated: true });

package/app/api/repair-items/route.ts

@@ -0,0 +1,86 @@
+import { NextResponse } from "next/server";
+import {
+  buildUnknownCostRepairWorkbench,
+  getUnknownCostReview,
+  saveUnknownCostReview,
+  type UnknownCostRepairStatus
+} from "@/src/lib/unknown-cost-repair";
+import { readJsonObject } from "@/src/lib/api-json";
+
+export const dynamic = "force-dynamic";
+
+const reviewStates = new Set<UnknownCostRepairStatus>([
+  "unresolved",
+  "ignored",
+  "resolved",
+  "needs-parser-review"
+]);
+
+function text(value: unknown, maxLength: number) {
+  return typeof value === "string" ? value.trim().slice(0, maxLength) : "";
+}
+
+function reviewState(value: unknown): UnknownCostRepairStatus | null {
+  return typeof value === "string" && reviewStates.has(value as UnknownCostRepairStatus)
+    ? (value as UnknownCostRepairStatus)
+    : null;
+}
+
+function workbenchGroupForKey(key: string) {
+  return buildUnknownCostRepairWorkbench().groups.find((group) => group.key === key) ?? null;
+}
+
+export async function GET() {
+  return NextResponse.json(buildUnknownCostRepairWorkbench());
+}
+
+export async function PUT(request: Request) {
+  const parsed = await readJsonObject(request);
+  if (!parsed.ok) {
+    return NextResponse.json({ error: parsed.error }, { status: 400 });
+  }
+  const body = parsed.body;
+  const key = text(body.key, 1000);
+  const status = reviewState(body.status ?? body.state);
+
+  if (!key) {
+    return NextResponse.json({ error: "key is required" }, { status: 400 });
+  }
+
+  if (!status) {
+    return NextResponse.json({ error: "status must be unresolved, ignored, resolved, or needs-parser-review" }, { status: 400 });
+  }
+
+  const group = workbenchGroupForKey(key);
+  const existing = group ? null : getUnknownCostReview(key);
+  if (!group && (!existing || existing.updatedAt == null)) {
+    return NextResponse.json({ error: "repair key was not found in current workbench evidence" }, { status: 404 });
+  }
+
+  const metadata = {
+    sourceFile: "",
+    model: "",
+    cause: ""
+  };
+  if (group) {
+    metadata.sourceFile = group.sourceFile;
+    metadata.model = group.model;
+    metadata.cause = group.cause;
+  } else if (existing) {
+    metadata.sourceFile = existing.sourceFile;
+    metadata.model = existing.model;
+    metadata.cause = existing.cause;
+  }
+  const review = saveUnknownCostReview({
+    key,
+    status,
+    notes: text(body.notes ?? body.note, 500),
+    sourceFile: metadata.sourceFile,
+    model: metadata.model,
+    cause: metadata.cause
+  });
+
+  return NextResponse.json({ review });
+}
+
+export const PATCH = PUT;

package/app/api/scan/route.ts

@@ -1,10 +1,15 @@
 import { NextResponse } from "next/server";
 import { runScan } from "@/src/ingestion/scan";
+import { readJsonObject } from "@/src/lib/api-json";
 
 export const dynamic = "force-dynamic";
 
 export async function POST(request: Request) {
-  const body = await request.json().catch(() => ({}));
+  const parsed = await readJsonObject(request);
+  if (!parsed.ok) {
+    return NextResponse.json({ error: parsed.error }, { status: 400 });
+  }
+  const body = parsed.body;
   const result = await runScan({
     folders: Array.isArray(body.folders)
       ? body.folders.filter((folder: unknown): folder is string => typeof folder === "string")

package/app/api/settings/route.ts

@@ -1,6 +1,7 @@
 import { NextResponse } from "next/server";
 import { getDatabasePath } from "@/src/db/client";
 import { getAppSettings, normalizeUsageGuardrails, saveAppSettings } from "@/src/db/settings";
+import { readJsonObject } from "@/src/lib/api-json";
 
 export const dynamic = "force-dynamic";
 
@@ -12,7 +13,11 @@ export async function GET() {
 }
 
 export async function PUT(request: Request) {
-  const body = await request.json();
+  const parsed = await readJsonObject(request);
+  if (!parsed.ok) {
+    return NextResponse.json({ error: parsed.error }, { status: 400 });
+  }
+  const body = parsed.body;
   const customFolders = Array.isArray(body.customFolders)
     ? body.customFolders.filter((folder: unknown): folder is string => typeof folder === "string")
     : [];

package/app/diagnostics/page.tsx

@@ -2,6 +2,7 @@ import Link from "next/link";
 import { AlertTriangle, ArrowRight, CheckCircle2, CircleDashed } from "lucide-react";
 import { Badge } from "@/components/ui/badge";
 import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card";
+import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table";
 import { DataValue, FieldLabel, MonoText, PageHeader } from "@/components/ui/typography";
 import { ScanHealthSummary } from "@/components/scan-health-summary";
 import { getAnalyticsData, getScanTrustData, type DebugScanRun } from "@/src/lib/analytics";
@@ -136,6 +137,7 @@ function TrustChecklist({
 function DoctorReportPanel({ report }: { report: DoctorReport }) {
   const statusRows = [
     ["Imported", report.fileStatus.imported],
+    ["With errors", report.fileStatus.importedWithErrors],
     ["Duplicates", report.fileStatus.duplicates],
     ["Ignored", report.fileStatus.ignored],
     ["Unsupported", report.fileStatus.unsupported],
@@ -192,7 +194,7 @@ function DoctorReportPanel({ report }: { report: DoctorReport }) {
         <div className="grid gap-6 xl:grid-cols-[1fr_1fr]">
           <div className="space-y-3">
             <div className="mb-3 text-sm font-semibold">File handling</div>
-            <div className="grid border-y sm:grid-cols-5 sm:divide-x xl:grid-cols-2">
+            <div className="grid border-y sm:grid-cols-6 sm:divide-x xl:grid-cols-2">
               {statusRows.map(([label, value]) => (
                 <div key={label} className="p-2">
                   <FieldLabel>{label}</FieldLabel>
@@ -261,6 +263,139 @@ function DoctorReportPanel({ report }: { report: DoctorReport }) {
   );
 }
 
+function ParserTrustPanel({ report }: { report: DoctorReport["parserTrust"] }) {
+  return (
+    <Card>
+      <CardHeader>
+        <CardTitle>Parser Trust Report</CardTitle>
+        <CardDescription>
+          Latest scan files grouped by parser, source family, version, status, and import yield. Ignored files are known support files, not usage transcripts. Unsupported files need parser review before they become usage.
+        </CardDescription>
+      </CardHeader>
+      <CardContent className="table-scroll">
+        {report.parsers.length ? (
+          <Table className="min-w-[72rem]">
+            <TableHeader>
+              <TableRow>
+                <TableHead>Parser</TableHead>
+                <TableHead>Version</TableHead>
+                <TableHead>Source</TableHead>
+                <TableHead className="text-right">Imported</TableHead>
+                <TableHead className="text-right">With errors</TableHead>
+                <TableHead className="text-right">Ignored</TableHead>
+                <TableHead className="text-right">Unsupported</TableHead>
+                <TableHead className="text-right">Failed</TableHead>
+                <TableHead className="text-right">Duplicate</TableHead>
+                <TableHead className="text-right">Records</TableHead>
+                <TableHead className="min-w-56">Latest reason</TableHead>
+              </TableRow>
+            </TableHeader>
+            <TableBody>
+              {report.parsers.map((row) => (
+                <TableRow key={`${row.parser}:${row.version}:${row.sourceFamily}`}>
+                  <TableCell className="font-medium">{row.parser}</TableCell>
+                  <TableCell>
+                    <Badge variant="secondary">{row.version}</Badge>
+                  </TableCell>
+                  <TableCell>{row.sourceFamily}</TableCell>
+                  <TableCell className="text-right">{row.imported.toLocaleString()}</TableCell>
+                  <TableCell className="text-right">{row.importedWithErrors.toLocaleString()}</TableCell>
+                  <TableCell className="text-right">{row.ignored.toLocaleString()}</TableCell>
+                  <TableCell className="text-right">{row.unsupported.toLocaleString()}</TableCell>
+                  <TableCell className="text-right">{row.failed.toLocaleString()}</TableCell>
+                  <TableCell className="text-right">{row.duplicate.toLocaleString()}</TableCell>
+                  <TableCell className="text-right">{row.recordsImported.toLocaleString()}</TableCell>
+                  <TableCell className="max-w-md text-xs text-muted-foreground">
+                    {row.latestReason || "No parser reason recorded."}
+                  </TableCell>
+                </TableRow>
+              ))}
+            </TableBody>
+          </Table>
+        ) : (
+          <div className="px-4 py-6 text-sm text-muted-foreground">
+            No parser trust data yet. Run `tokentrace scan` to populate the latest scan report.
+          </div>
+        )}
+      </CardContent>
+    </Card>
+  );
+}
+
+function formatDelta(value: number) {
+  if (value > 0) return `+${value.toLocaleString()}`;
+  return value.toLocaleString();
+}
+
+function ScanDiffPanel({ report }: { report: DoctorReport["scanDiff"] }) {
+  const rows: Array<[string, keyof DoctorReport["scanDiff"]["current"]]> = [
+    ["Files scanned", "filesScanned"],
+    ["Records imported", "recordsImported"],
+    ["Imported", "imported"],
+    ["With errors", "importedWithErrors"],
+    ["Duplicates", "duplicates"],
+    ["Ignored", "ignored"],
+    ["Unsupported", "unsupported"],
+    ["Failed", "failed"]
+  ];
+
+  return (
+    <Card>
+      <CardHeader>
+        <CardTitle>Scan History Diff</CardTitle>
+        <CardDescription>
+          Latest scan compared with the previous scan using deterministic scan ordering. Ignored files are known support files, not usage transcripts.
+        </CardDescription>
+      </CardHeader>
+      <CardContent className="table-scroll space-y-4">
+        <div className="grid border-y md:grid-cols-2 md:divide-x">
+          <div className="min-w-0 p-3">
+            <FieldLabel>Latest scan</FieldLabel>
+            <div className="mt-1 text-sm font-semibold">{formatDate(report.latestCompletedAt ?? report.latestStartedAt)}</div>
+            <MonoText className="mt-1 block truncate text-xs text-muted-foreground">
+              {report.latestScanId ?? "No scan"}
+            </MonoText>
+          </div>
+          <div className="min-w-0 p-3">
+            <FieldLabel>Previous scan</FieldLabel>
+            <div className="mt-1 text-sm font-semibold">{formatDate(report.previousCompletedAt ?? report.previousStartedAt)}</div>
+            <MonoText className="mt-1 block truncate text-xs text-muted-foreground">
+              {report.previousScanId ?? "No previous scan"}
+            </MonoText>
+          </div>
+        </div>
+
+        <Table>
+          <TableHeader>
+            <TableRow>
+              <TableHead>Count</TableHead>
+              <TableHead className="text-right">Current</TableHead>
+              <TableHead className="text-right">Previous</TableHead>
+              <TableHead className="text-right">Delta</TableHead>
+            </TableRow>
+          </TableHeader>
+          <TableBody>
+            {rows.map(([label, key]) => (
+              <TableRow key={key}>
+                <TableCell className="font-medium">{label}</TableCell>
+                <TableCell className="text-right">{report.current[key].toLocaleString()}</TableCell>
+                <TableCell className="text-right">{report.previous[key].toLocaleString()}</TableCell>
+                <TableCell className="text-right">{formatDelta(report.delta[key])}</TableCell>
+              </TableRow>
+            ))}
+          </TableBody>
+        </Table>
+
+        {report.explanation ? (
+          <div className="rounded-md border bg-muted/30 p-3 text-sm leading-relaxed text-muted-foreground">
+            {report.explanation}
+          </div>
+        ) : null}
+      </CardContent>
+    </Card>
+  );
+}
+
 function scanRunVariant(scanRun: DebugScanRun) {
   if (scanRun.errors.length > 0) return "destructive";
   if (scanRun.warnings.length > 0) return "warning";
@@ -366,6 +501,10 @@ export default async function DiagnosticsPage() {
 
       <DoctorReportPanel report={doctorReport} />
 
+      <ParserTrustPanel report={doctorReport.parserTrust} />
+
+      <ScanDiffPanel report={doctorReport.scanDiff} />
+
       <ScanHistoryPanel scanRuns={data.scanRuns} />
 
       <ScanHealthSummary health={data.health} />