agentic-dataset-builder 0.1.1 → 0.2.1

package/README.md

@@ -1,125 +1,154 @@
 # Agentic Dataset Builder
 
-Build one merged training dataset from local Pi and Codex session history.
+Pure TypeScript CLI for turning local Pi, Codex, and Claude Code history into one validated `dataset.parquet` file.
 
-## Runtime
+## Goal
 
-The core implementation is Python. User-facing entrypoints do not require bash.
+Use this repo when you want an AI coding assistant to do one job end-to-end:
 
-Requirements:
+1. discover local session history
+2. normalize it into the local Qwen35-compatible schema
+3. label records by training use
+4. write one final parquet dataset
 
-- Python 3.10+
-- Node 18+ if you want to run via `npx`
+The CLI is native Node.js + TypeScript. It does not require Python.
 
-## Recommended usage
+## Fastest path
 
-Published npm-style entrypoint:
+If the package is published on npm:
 
 ```bash
-npx agentic-dataset-builder --output-root ./out
+npx --registry=https://registry.npmjs.org/ agentic-dataset-builder@0.2.1 --output-root ./out
 ```
 
-Local repo usage without bash:
+If working from this repo locally:
 
 ```bash
-node cli.mjs --output-root ./out
+npm install
+npm run build
+node dist/cli.js --output-root ./out
 ```
 
-Direct Python entrypoint:
+## What the command does
 
-```bash
-python run.py --output-root ./out
-```
-
-If you want to pre-create the Python environment yourself:
-
-```bash
-pip install -r requirements.txt
-python run.py --output-root ./out
-```
+The CLI will:
 
-## What users run
+- detect local session roots for `pi`, `codex`, and `claude`
+- read supported history files
+- validate normalized records with `Zod`
+- keep only the labels you requested
+- write one final parquet file
+- write a manifest and a run log
 
-From this directory, the simplest no-bash local command is:
+## Default source behavior
 
-```bash
-node cli.mjs --output-root ./out
-```
+- `pi`
+  - full agent traces
+  - can produce `cot_eligible` or `agent_only`
+- `codex`
+  - full agent traces
+  - usually produces `agent_only`
+- `claude`
+  - prompt history only for now
+  - produces `prompt_only`
 
-That one command will:
+Claude is intentionally low-fidelity right now. It is not treated as a full assistant/tool trace source.
 
-- scan `~/.pi/agent/sessions`
-- scan `~/.codex/sessions`
-- convert session history into the local Qwen3.5 schema
-- label records as `cot_eligible`, `agent_only`, or `discard`
-- keep `cot_eligible` and `agent_only`
-- merge them into one final parquet file
-- remove intermediate directories automatically after success
+## Default output
 
-## Final output
-
-Each run creates one directory under `./out/`:
+Each run creates one directory:
 
 ```text
-out/agentic-dataset-<timestamp>/
+<output-root>/agentic-dataset-<timestamp>/
   dataset.parquet
   manifest.json
   run.log
 ```
 
-Default deliverable is just one user-facing dataset file:
+Files:
 
 - `dataset.parquet`
+  - final merged dataset
+- `manifest.json`
+  - source roots, source counts, labels kept, output path
+- `run.log`
+  - step-by-step execution log for debugging
 
-Supporting files:
-
-- `manifest.json`: what was scanned, what was kept, summary stats
-- `run.log`: full step-by-step execution log
+## Recommended commands
 
-## Common options
+Pi + Codex:
 
 ```bash
-# only Pi
-node cli.mjs --output-root ./out --include-sources pi
+node dist/cli.js --output-root ./out --include-sources pi,codex --include-labels cot_eligible,agent_only
+```
 
-# only Codex
-node cli.mjs --output-root ./out --include-sources codex
+Codex + Claude prompt-only:
 
-# keep intermediates for debugging
-node cli.mjs --output-root ./out --keep-intermediates
+```bash
+node dist/cli.js --output-root ./out --include-sources codex,claude --include-labels agent_only,prompt_only
+```
+
+Pi only:
 
-# also emit final merged jsonl/jsonl.gz
-node cli.mjs --output-root ./out --final-format both
+```bash
+node dist/cli.js --output-root ./out --include-sources pi --include-labels cot_eligible,agent_only
 ```
 
-## What is kept by default
+## Important flags
 
-- `cot_eligible`: agentic traces with visible reasoning
-- `agent_only`: agentic traces without visible reasoning
+- `--output-root <dir>`
+  - required output root
+- `--include-sources <csv>`
+  - any of: `pi,codex,claude`
+- `--include-labels <csv>`
+  - any of: `cot_eligible,agent_only,prompt_only,discard`
+- `--pi-root <dir>`
+  - override detected Pi session path
+- `--codex-root <dir>`
+  - override detected Codex session path
+- `--claude-root <dir>`
+  - override detected Claude project-history path
+- `--help`
+  - print CLI help
 
-`discard` records are excluded from the final dataset by default.
+## Auto-detected paths
 
-## Package layout
+The CLI tries OS-specific defaults automatically.
 
-```text
-agentic_dataset/
-  build_agentic_dataset.py
-  export_pi_session.py
-  export_pi_session_to_qwen35.py
-  export_codex_session_to_qwen35.py
-  label_qwen35_agentic.py
-  export_qwen35_training.py
-  qwen35_training_record.py
-run.sh
-run.py
-cli.mjs
-README.md
+Typical paths:
+
+- Pi: `~/.pi/agent/sessions`
+- Codex: `~/.codex/sessions`
+- Claude: `~/.claude/projects`
+
+On Windows it also checks `APPDATA` and `LOCALAPPDATA` variants.
+
+## Verification checklist
+
+After a run, verify these three things:
+
+1. `dataset.parquet` exists
+2. `manifest.json` exists
+3. `run.log` does not end with an uncaught error
+
+Typical quick check:
+
+```bash
+ls ./out/agentic-dataset-*/
+```
+
+## Development notes
+
+Useful development commands:
+
+```bash
+npm run check
+npm run test
+npm run build
 ```
 
-## Notes
+This repo currently includes:
 
-- default session roots are auto-detected for Linux, macOS, and Windows
-- override session paths with `--pi-root` and `--codex-root` if needed
-- Pi currently provides much better visible reasoning coverage than Codex.
-- Codex traces are still useful for agent-behavior distillation even when reasoning is encrypted-only.
-- Redaction is not included yet. Add it before distributing the tool broadly if users may have sensitive local data.
+- Zod validation for source events and final records
+- Vitest coverage for core schema and labeling paths
+- native parquet writing in TypeScript

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,166 @@
+#!/usr/bin/env node
+import fs from 'node:fs';
+import path from 'node:path';
+import { candidateClaudeRoots, candidateCodexRoots, candidatePiRoots, firstExisting } from './platform/paths.js';
+import { collectPiRecords } from './sources/pi.js';
+import { collectCodexRecords } from './sources/codex.js';
+import { collectClaudePromptOnlyRecords } from './sources/claude.js';
+import { labelRecord } from './labeling.js';
+import { Qwen35RecordSchema } from './schemas/qwen35.js';
+import { writeParquet } from './parquet.js';
+function parseArgs(argv) {
+    if (argv.includes('--help') || argv.includes('-h')) {
+        console.log(`agentic-dataset-builder@0.2.1
+
+Usage:
+  npx agentic-dataset-builder@0.2.1 --output-root ./out
+
+Options:
+  --output-root <dir>       Output directory root
+  --include-sources <list>  Comma-separated: pi,codex,claude
+  --include-labels <list>   Comma-separated: cot_eligible,agent_only,prompt_only,discard
+  --pi-root <dir>           Override Pi session root
+  --codex-root <dir>        Override Codex session root
+  --claude-root <dir>       Override Claude project history root
+  --help                    Show this help message
+`);
+        process.exit(0);
+    }
+    const args = {
+        outputRoot: './out',
+        includeSources: ['pi', 'codex'],
+        includeLabels: new Set(['cot_eligible', 'agent_only']),
+    };
+    for (let i = 0; i < argv.length; i += 1) {
+        const arg = argv[i];
+        const next = argv[i + 1];
+        if (arg === '--output-root' && next) {
+            args.outputRoot = next;
+            i += 1;
+        }
+        else if (arg === '--include-sources' && next) {
+            args.includeSources = next.split(',').map((v) => v.trim()).filter(Boolean);
+            i += 1;
+        }
+        else if (arg === '--include-labels' && next) {
+            args.includeLabels = new Set(next.split(',').map((v) => v.trim()).filter(Boolean));
+            i += 1;
+        }
+        else if (arg === '--pi-root' && next) {
+            args.piRoot = next;
+            i += 1;
+        }
+        else if (arg === '--codex-root' && next) {
+            args.codexRoot = next;
+            i += 1;
+        }
+        else if (arg === '--claude-root' && next) {
+            args.claudeRoot = next;
+            i += 1;
+        }
+    }
+    return args;
+}
+function stampDir(base) {
+    const iso = new Date().toISOString().replace(/[:.]/g, '-');
+    return path.resolve(base, `agentic-dataset-${iso}`);
+}
+function createLogger(runDir) {
+    const logPath = path.join(runDir, 'run.log');
+    fs.writeFileSync(logPath, '', 'utf8');
+    return {
+        logPath,
+        log(step, message) {
+            const line = `[${step}] ${message}`;
+            console.log(line);
+            fs.appendFileSync(logPath, `${line}\n`, 'utf8');
+        },
+    };
+}
+async function main() {
+    const args = parseArgs(process.argv.slice(2));
+    const runDir = stampDir(args.outputRoot);
+    fs.mkdirSync(runDir, { recursive: true });
+    const logger = createLogger(runDir);
+    const allRecords = [];
+    const sourceStats = {};
+    for (const source of args.includeSources) {
+        if (source === 'pi') {
+            const root = path.resolve(args.piRoot ?? firstExisting(candidatePiRoots()));
+            logger.log('pi', `reading ${root}`);
+            const records = await collectPiRecords(root);
+            sourceStats.pi = { records: records.length };
+            for (const record of records)
+                pushLabeled(record, 'pi', args.includeLabels, allRecords);
+            logger.log('pi', `kept ${allRecords.filter((r) => r.source_system === 'pi').length} labeled records`);
+        }
+        if (source === 'codex') {
+            const root = path.resolve(args.codexRoot ?? firstExisting(candidateCodexRoots()));
+            logger.log('codex', `reading ${root}`);
+            const records = await collectCodexRecords(root);
+            sourceStats.codex = { records: records.length };
+            for (const record of records)
+                pushLabeled(record, 'codex', args.includeLabels, allRecords);
+            logger.log('codex', `kept ${allRecords.filter((r) => r.source_system === 'codex').length} labeled records`);
+        }
+        if (source === 'claude') {
+            const root = path.resolve(args.claudeRoot ?? firstExisting(candidateClaudeRoots()));
+            logger.log('claude', `reading ${root}`);
+            const records = await collectClaudePromptOnlyRecords(root);
+            sourceStats.claude = { records: records.length };
+            for (const record of records)
+                pushLabeled(record, 'claude', args.includeLabels, allRecords);
+            logger.log('claude', `kept ${allRecords.filter((r) => r.source_system === 'claude').length} labeled records`);
+        }
+    }
+    const datasetPath = path.join(runDir, 'dataset.parquet');
+    logger.log('write', `writing ${allRecords.length} records to ${datasetPath}`);
+    await writeParquet(datasetPath, allRecords);
+    const manifest = {
+        runDir,
+        datasetParquetPath: datasetPath,
+        recordCount: allRecords.length,
+        includeSources: args.includeSources,
+        includeLabels: [...args.includeLabels],
+        sourceStats,
+        runLog: logger.logPath,
+    };
+    fs.writeFileSync(path.join(runDir, 'manifest.json'), `${JSON.stringify(manifest, null, 2)}\n`, 'utf8');
+    console.log(JSON.stringify(manifest));
+}
+function pushLabeled(record, sourceSystem, includeLabels, target) {
+    const info = labelRecord(record);
+    if (!includeLabels.has(info.label))
+        return;
+    const enriched = Qwen35RecordSchema.parse({
+        ...record,
+        label: info.label,
+        source_system: sourceSystem,
+        source_bucket: record.meta.lossy_source ? 'lossy' : 'strict',
+        source_file: record.meta.source,
+        agentic_label: {
+            label: info.label,
+            tool_call_count: info.toolCallCount,
+            tool_message_count: info.toolMessageCount,
+            dialogue_rounds_est: info.dialogueRounds,
+            reasoning_chars: info.reasoningChars,
+            has_reasoning: info.hasReasoning,
+            lossy_source: record.meta.lossy_source,
+            lossy_reasons: info.lossyReasons,
+        },
+        meta: {
+            ...record.meta,
+            dataset_label: info.label,
+            dataset_source_system: sourceSystem,
+            dataset_source_bucket: record.meta.lossy_source ? 'lossy' : 'strict',
+            dataset_source_file: record.meta.source,
+            dataset_has_reasoning: info.hasReasoning,
+            dataset_reasoning_chars: info.reasoningChars,
+        },
+    });
+    target.push(enriched);
+}
+main().catch((error) => {
+    console.error(error instanceof Error ? error.stack ?? error.message : String(error));
+    process.exit(1);
+});

package/dist/labeling.d.ts

@@ -0,0 +1,12 @@
+import type { Qwen35Record } from './schemas/qwen35.js';
+export type Label = 'cot_eligible' | 'agent_only' | 'prompt_only' | 'discard';
+export interface LabelInfo {
+    label: Label;
+    toolCallCount: number;
+    toolMessageCount: number;
+    dialogueRounds: number;
+    reasoningChars: number;
+    hasReasoning: boolean;
+    lossyReasons: string[];
+}
+export declare function labelRecord(record: Qwen35Record): LabelInfo;

package/dist/labeling.js

@@ -0,0 +1,22 @@
+export function labelRecord(record) {
+    const toolCallCount = record.messages
+        .filter((message) => message.role === 'assistant')
+        .reduce((sum, message) => sum + (message.tool_calls?.length ?? 0), 0);
+    const toolMessageCount = record.messages.filter((message) => message.role === 'tool').length;
+    const dialogueRounds = record.messages.filter((message) => message.role === 'user').length;
+    const reasoningChars = record.messages
+        .filter((message) => message.role === 'assistant')
+        .reduce((sum, message) => sum + (typeof message.reasoning_content === 'string' ? message.reasoning_content.length : 0), 0);
+    const hasReasoning = reasoningChars > 0;
+    const lossyReasons = record.meta.lossy_reasons;
+    const promptOnly = lossyReasons.includes('prompt_history_only');
+    const agentic = toolCallCount >= 1 && toolMessageCount >= 1 && dialogueRounds >= 1;
+    let label = 'discard';
+    if (promptOnly)
+        label = 'prompt_only';
+    else if (agentic && hasReasoning)
+        label = 'cot_eligible';
+    else if (agentic)
+        label = 'agent_only';
+    return { label, toolCallCount, toolMessageCount, dialogueRounds, reasoningChars, hasReasoning, lossyReasons };
+}

package/dist/parquet.d.ts

@@ -0,0 +1,4 @@
+import type { Qwen35Record } from './schemas/qwen35.js';
+export declare function parquetSchema(): any;
+export declare function recordToParquetRow(record: Qwen35Record): Record<string, unknown>;
+export declare function writeParquet(filePath: string, records: Qwen35Record[]): Promise<void>;

package/dist/parquet.js

@@ -0,0 +1,115 @@
+import parquet from 'parquetjs-lite';
+export function parquetSchema() {
+    return new parquet.ParquetSchema({
+        id: { type: 'UTF8' },
+        request_id: { type: 'UTF8', optional: true },
+        endpoint: { type: 'UTF8' },
+        status: { type: 'INT64' },
+        ts: { type: 'UTF8' },
+        key: { type: 'UTF8', optional: true },
+        source: { type: 'UTF8', optional: true },
+        requested_model: { type: 'UTF8', optional: true },
+        actual_model: { type: 'UTF8', optional: true },
+        stream: { type: 'BOOLEAN', optional: true },
+        thinking_level: { type: 'UTF8', optional: true },
+        reasoning_summary_mode_json: { type: 'UTF8', optional: true },
+        thinking_type: { type: 'UTF8', optional: true },
+        thinking_budget_tokens: { type: 'INT64', optional: true },
+        max_output_tokens: { type: 'INT64', optional: true },
+        tool_spec_count: { type: 'INT64', optional: true },
+        tool_choice_json: { type: 'UTF8', optional: true },
+        request_contains_non_text_content: { type: 'BOOLEAN' },
+        request_image_block_count: { type: 'INT64' },
+        request_video_block_count: { type: 'INT64' },
+        request_tool_call_block_count: { type: 'INT64' },
+        request_tool_result_block_count: { type: 'INT64' },
+        request_thinking_block_count: { type: 'INT64' },
+        response_contains_non_text_content: { type: 'BOOLEAN' },
+        response_image_block_count: { type: 'INT64' },
+        response_video_block_count: { type: 'INT64' },
+        response_tool_call_block_count: { type: 'INT64' },
+        response_tool_result_block_count: { type: 'INT64' },
+        response_thinking_block_count: { type: 'INT64' },
+        request_truncated: { type: 'BOOLEAN' },
+        response_truncated: { type: 'BOOLEAN' },
+        lossy_source: { type: 'BOOLEAN' },
+        lossy_reasons_json: { type: 'UTF8' },
+        user_message_count: { type: 'INT64' },
+        assistant_message_count: { type: 'INT64' },
+        tool_message_count: { type: 'INT64' },
+        dialogue_rounds_est: { type: 'INT64' },
+        tool_call_count: { type: 'INT64' },
+        has_reasoning: { type: 'BOOLEAN' },
+        reasoning_chars: { type: 'INT64' },
+        content_chars_total: { type: 'INT64' },
+        messages_json: { type: 'UTF8' },
+        tools_json: { type: 'UTF8' },
+        meta_json: { type: 'UTF8' },
+    });
+}
+export function recordToParquetRow(record) {
+    const messages = record.messages;
+    const userMessageCount = messages.filter((m) => m.role === 'user').length;
+    const assistantMessages = messages.filter((m) => m.role === 'assistant');
+    const toolMessageCount = messages.filter((m) => m.role === 'tool').length;
+    const toolCallCount = assistantMessages.reduce((sum, message) => sum + (message.tool_calls?.length ?? 0), 0);
+    const reasoningChars = assistantMessages.reduce((sum, message) => sum + (typeof message.reasoning_content === 'string' ? message.reasoning_content.length : 0), 0);
+    const contentCharsTotal = messages.reduce((sum, message) => sum + JSON.stringify(message.content).length, 0);
+    return {
+        id: record.id,
+        request_id: record.request_id,
+        endpoint: record.meta.endpoint,
+        status: record.meta.status,
+        ts: record.meta.ts,
+        key: record.meta.key,
+        source: record.meta.source,
+        requested_model: record.meta.requested_model ?? undefined,
+        actual_model: record.meta.actual_model ?? undefined,
+        stream: record.meta.stream,
+        thinking_level: record.meta.thinking_level ?? undefined,
+        reasoning_summary_mode_json: JSON.stringify(record.meta.reasoning_summary_mode ?? null),
+        thinking_type: record.meta.thinking_type ?? undefined,
+        thinking_budget_tokens: record.meta.thinking_budget_tokens ?? undefined,
+        max_output_tokens: record.meta.max_output_tokens ?? undefined,
+        tool_spec_count: record.meta.tool_spec_count,
+        tool_choice_json: JSON.stringify(record.meta.tool_choice ?? null),
+        request_contains_non_text_content: record.meta.request_contains_non_text_content,
+        request_image_block_count: record.meta.request_image_block_count,
+        request_video_block_count: record.meta.request_video_block_count,
+        request_tool_call_block_count: record.meta.request_tool_call_block_count,
+        request_tool_result_block_count: record.meta.request_tool_result_block_count,
+        request_thinking_block_count: record.meta.request_thinking_block_count,
+        response_contains_non_text_content: record.meta.response_contains_non_text_content,
+        response_image_block_count: record.meta.response_image_block_count,
+        response_video_block_count: record.meta.response_video_block_count,
+        response_tool_call_block_count: record.meta.response_tool_call_block_count,
+        response_tool_result_block_count: record.meta.response_tool_result_block_count,
+        response_thinking_block_count: record.meta.response_thinking_block_count,
+        request_truncated: record.meta.request_truncated,
+        response_truncated: record.meta.response_truncated,
+        lossy_source: record.meta.lossy_source,
+        lossy_reasons_json: JSON.stringify(record.meta.lossy_reasons),
+        user_message_count: userMessageCount,
+        assistant_message_count: assistantMessages.length,
+        tool_message_count: toolMessageCount,
+        dialogue_rounds_est: userMessageCount,
+        tool_call_count: toolCallCount,
+        has_reasoning: reasoningChars > 0,
+        reasoning_chars: reasoningChars,
+        content_chars_total: contentCharsTotal,
+        messages_json: JSON.stringify(record.messages),
+        tools_json: JSON.stringify(record.tools),
+        meta_json: JSON.stringify(record.meta),
+    };
+}
+export async function writeParquet(filePath, records) {
+    const writer = await parquet.ParquetWriter.openFile(parquetSchema(), filePath);
+    try {
+        for (const record of records) {
+            await writer.appendRow(recordToParquetRow(record));
+        }
+    }
+    finally {
+        await writer.close();
+    }
+}

package/dist/platform/paths.d.ts

@@ -0,0 +1,4 @@
+export declare function candidatePiRoots(): string[];
+export declare function candidateCodexRoots(): string[];
+export declare function candidateClaudeRoots(): string[];
+export declare function firstExisting(candidates: string[]): string;

package/dist/platform/paths.js

@@ -0,0 +1,62 @@
+import os from 'node:os';
+import path from 'node:path';
+import fs from 'node:fs';
+export function candidatePiRoots() {
+    const home = os.homedir();
+    const appdata = process.env.APPDATA;
+    const localappdata = process.env.LOCALAPPDATA;
+    return dedupe([
+        process.env.PI_SESSION_ROOT,
+        path.join(home, '.pi', 'agent', 'sessions'),
+        appdata ? path.join(appdata, 'pi', 'agent', 'sessions') : undefined,
+        appdata ? path.join(appdata, '.pi', 'agent', 'sessions') : undefined,
+        localappdata ? path.join(localappdata, 'pi', 'agent', 'sessions') : undefined,
+        localappdata ? path.join(localappdata, '.pi', 'agent', 'sessions') : undefined,
+    ]);
+}
+export function candidateCodexRoots() {
+    const home = os.homedir();
+    const appdata = process.env.APPDATA;
+    const localappdata = process.env.LOCALAPPDATA;
+    return dedupe([
+        process.env.CODEX_SESSION_ROOT,
+        path.join(home, '.codex', 'sessions'),
+        appdata ? path.join(appdata, 'Codex', 'sessions') : undefined,
+        appdata ? path.join(appdata, '.codex', 'sessions') : undefined,
+        localappdata ? path.join(localappdata, 'Codex', 'sessions') : undefined,
+        localappdata ? path.join(localappdata, '.codex', 'sessions') : undefined,
+    ]);
+}
+export function candidateClaudeRoots() {
+    const home = os.homedir();
+    const appdata = process.env.APPDATA;
+    const localappdata = process.env.LOCALAPPDATA;
+    return dedupe([
+        process.env.CLAUDE_SESSION_ROOT,
+        path.join(home, '.claude', 'projects'),
+        appdata ? path.join(appdata, 'Claude', 'projects') : undefined,
+        appdata ? path.join(appdata, '.claude', 'projects') : undefined,
+        localappdata ? path.join(localappdata, 'Claude', 'projects') : undefined,
+        localappdata ? path.join(localappdata, '.claude', 'projects') : undefined,
+    ]);
+}
+export function firstExisting(candidates) {
+    for (const candidate of candidates) {
+        if (fs.existsSync(candidate))
+            return path.resolve(candidate);
+    }
+    return path.resolve(candidates[0]);
+}
+function dedupe(values) {
+    const seen = new Set();
+    const result = [];
+    for (const value of values) {
+        if (!value)
+            continue;
+        if (seen.has(value))
+            continue;
+        seen.add(value);
+        result.push(value);
+    }
+    return result;
+}