opentraces-schema 0.1.0__tar.gz

opentraces_schema-0.1.0/.gitignore

@@ -0,0 +1,24 @@
+.venv/
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+*.egg-link
+web/site/node_modules/
+web/site/.next/
+web/viewer/node_modules/
+web/coming-soon/.vercel/
+.gstack/
+.opentraces/
+
+.desloppify/
+
+# Private research (not shipped in OSS release)
+kb/
+.opentraces/staging/
+.opentraces/config.json
+.agents/skills/opentraces
+.claude
+.opentraces

opentraces_schema-0.1.0/CHANGELOG.md

@@ -0,0 +1,34 @@
+# Changelog
+
+All notable changes to the opentraces-schema package will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html) with
+schema-specific semantics described in VERSION-POLICY.md.
+
+## [Unreleased]
+
+## [0.1.0] - 2026-03-27
+
+### Added
+- Initial schema release with 15 Pydantic v2 models
+- `TraceRecord` top-level model: one JSONL line per complete agent session
+- `Step` model oriented around TAO (Thought-Action-Observation) loops, not conversational turns
+- `Outcome` model with RL-ready signals: `success`, `signal_source`, `signal_confidence` (derived/inferred/annotated)
+- `Attribution` block (experimental) bridging trajectory data and code attribution per Agent Trace spec
+- Sub-agent hierarchy via `Step.parent_step`, `Step.agent_role`, `Step.subagent_trajectory_ref`
+- `Step.call_type` (main/subagent/warmup) for filtering cache-priming calls
+- System prompt deduplication via hash-keyed `system_prompts` dict on `TraceRecord`
+- `SecurityMetadata` with 3-tier classification (1=open, 2=guarded, 3=strict)
+- Content hashing (SHA-256) on `TraceRecord` for cross-upload deduplication
+- `AttributionRange.content_hash` using murmur3 for cross-refactor tracking
+- `Observation.output_summary` for lightweight filtering without loading full tool results
+- `TokenUsage` with `prefix_reuse_tokens`, `cache_read_tokens`, `cache_write_tokens`
+- `Metrics` model with session-level aggregates and `estimated_cost_usd`
+- `Environment` and `VCS` models for runtime context and reproducibility
+- `Task` model with `source`, `repository`, `base_commit`
+- `Agent` model using `provider/model-name` convention from models.dev
+- `Snippet` model for extracted code blocks linked to source steps
+
+### Design References
+- See [RATIONALE-0.1.0.md](RATIONALE-0.1.0.md) for the design basis of each decision in this version

opentraces_schema-0.1.0/FIELD-MAPPINGS.md

@@ -0,0 +1,159 @@
+# Field Mappings: opentraces -> Downstream Formats
+
+Reference tables for converting opentraces TraceRecord to downstream schemas.
+These mappings are implemented in `src/opentraces/exporters/` but documented here
+for ML researchers who want to write their own converters.
+
+## opentraces -> ATIF v1.6
+
+ATIF (Agent Trajectory Interchange Format) is designed for SFT and RL training pipelines.
+The export is lossy: opentraces fields with no ATIF equivalent are dropped.
+
+### Root Level
+
+| opentraces | ATIF v1.6 | Notes |
+|-----------|-----------|-------|
+| `schema_version` | `schema_version: "ATIF-v1.6"` | Hardcoded |
+| `session_id` | `session_id` | Direct |
+| `agent.name` | `agent.name` | Direct |
+| `agent.version` | `agent.version` | Direct |
+| `agent.model` | `agent.model_name` | Rename |
+| `tool_definitions` | `agent.tool_definitions` | Direct |
+| `trace_id` | - | Dropped (ATIF uses session_id only) |
+| `content_hash` | - | Dropped |
+| `timestamp_start` | - | Dropped (per-step timestamps preserved) |
+| `timestamp_end` | - | Dropped |
+| `environment` | - | Dropped |
+| `outcome` | - | Dropped |
+| `dependencies` | - | Dropped |
+| `metrics` | - | Dropped (per-step metrics preserved) |
+| `security` | - | Dropped |
+| `attribution` | - | Dropped |
+| `system_prompts` | - | Dropped (ATIF stores inline per step) |
+| `metadata` | - | Dropped |
+
+### Step Level
+
+| opentraces Step | ATIF Step | Notes |
+|----------------|-----------|-------|
+| `step_index` | `step_id` | Renumbered sequentially from 1 at export time |
+| `role` | `source` | Direct (system/user/agent are the same) |
+| `content` | `message` | Direct, omitted if None |
+| `reasoning_content` | `reasoning_content` | Direct |
+| `model` | `model_name` | Rename |
+| `timestamp` | `timestamp` | Direct |
+| `system_prompt_hash` | - | Dropped |
+| `agent_role` | - | Dropped |
+| `parent_step` | - | Dropped |
+| `call_type` | - | Dropped |
+| `subagent_trajectory_ref` | - | Dropped (ATIF supports this but we don't populate it) |
+| `tools_available` | - | Dropped |
+| `snippets` | - | Dropped |
+
+### Tool Calls
+
+| opentraces ToolCall | ATIF ToolCallSchema | Notes |
+|--------------------|---------------------|-------|
+| `tool_call_id` | `tool_call_id` | Direct |
+| `tool_name` | `function_name` | Rename |
+| `input` (dict) | `arguments` (dict) | Direct (ATIF accepts dict) |
+| `duration_ms` | - | Dropped |
+
+### Observations
+
+opentraces stores observations as a flat list on each Step. ATIF wraps them
+in a singular `observation` object with a `results` array.
+
+| opentraces Observation | ATIF ObservationResult | Notes |
+|-----------------------|-----------------------|-------|
+| `source_call_id` | `source_call_id` | Direct |
+| `content` | `content` | Direct |
+| `output_summary` | - | Dropped |
+| `error` | `content` | Mapped as `[error: {value}]` string |
+
+**Structure transformation:**
+```
+opentraces: step.observations = [Obs1, Obs2]
+ATIF:       step.observation = {"results": [Result1, Result2]}
+```
+
+When a step has zero observations, the `observation` key is omitted entirely.
+
+### Token Usage
+
+| opentraces TokenUsage | ATIF MetricsSchema | Notes |
+|-----------------------|-------------------|-------|
+| `input_tokens` | `prompt_tokens` | Rename |
+| `output_tokens` | `completion_tokens` | Rename |
+| `cache_read_tokens` | `cached_tokens` | Rename |
+| `cache_write_tokens` | - | Dropped (no ATIF equivalent) |
+| `prefix_reuse_tokens` | - | Dropped (opentraces-only metric) |
+
+ATIF also supports `cost_usd`, `prompt_token_ids`, `completion_token_ids`, and
+`logprobs`, but these are not available from CLI-level agent traces and are
+omitted from the export.
+
+---
+
+## opentraces -> ADP (Agent Data Protocol)
+
+ADP is designed as a training interlingua for SFT across multiple agent harnesses.
+The export flattens opentraces' hierarchical steps into ADP's alternating
+action/observation list.
+
+*Exporter implementation planned. Mapping table below is a reference for
+researchers writing their own converters.*
+
+### Core Mapping
+
+| opentraces | ADP | Notes |
+|-----------|-----|-------|
+| `session_id` | `Trajectory.id` | Direct |
+| Step(role=agent, tool_calls=[tc]) | `APIAction(function=tc.tool_name, kwargs=tc.input)` | Each tool call becomes a separate APIAction |
+| Step(role=agent, content=code) | `CodeAction(language=..., content=...)` | Only if step contains executable code |
+| Step(role=agent, content=text) | `MessageAction(content=text)` | Agent messages without tool calls |
+| Observation(content=text) | `TextObservation(source="environment", content=text)` | Tool results |
+| Step(role=user) | `TextObservation(source="user", content=text)` | User messages |
+| `reasoning_content` | `APIAction.description` or `CodeAction.description` | Reasoning attached to the action |
+| `metadata` | `Trajectory.details` | Flexible dict |
+
+### Fields Dropped by ADP Export
+
+All of: `attribution`, `security`, `environment`, `outcome`, `dependencies`,
+`metrics`, `system_prompts`, `tool_definitions`, `content_hash`, `token_usage`,
+`snippets`, `hierarchy` (parent_step, agent_role, call_type).
+
+ADP's key strength is simplicity: 3 action types + 2 observation types cover
+coding, browsing, tool use, and SWE. The trade-off is losing all the metadata
+that makes opentraces traces useful for analytics, RL reward modeling,
+attribution, and security auditing.
+
+---
+
+## opentraces -> OTel GenAI (future)
+
+OTel GenAI Semantic Conventions represent traces as span trees, which is a
+fundamentally different structure from our step arrays. Each opentraces Step
+would become a span, with tool calls as child spans.
+
+*Exporter implementation planned for v0.2.*
+
+---
+
+## Notes for Converter Authors
+
+1. **step_id renumbering**: ATIF uses 1-indexed step_id. opentraces step_index
+   may be 0 or 1-indexed depending on the parser. Always renumber at export time.
+
+2. **observation wrapping**: ATIF uses singular `observation` with `results[]`.
+   opentraces uses plural `observations[]`. Don't just rename the field.
+
+3. **token_usage partial mapping**: opentraces tracks 5 token sub-fields, ATIF
+   tracks 3. The two cache fields unique to opentraces (cache_write, prefix_reuse)
+   are our key differentiator for cost analysis.
+
+4. **content=None steps**: Steps that are pure tool calls (no text content) should
+   omit the `message`/`content` field, not set it to empty string.
+
+5. **dangling tool calls**: Observations with `error="no_result"` indicate tool
+   calls that never received a response. Map these to a descriptive error string.

opentraces_schema-0.1.0/PKG-INFO

@@ -0,0 +1,68 @@
+Metadata-Version: 2.4
+Name: opentraces-schema
+Version: 0.1.0
+Summary: Schema models for the opentraces.ai agent trace JSONL format
+Project-URL: Homepage, https://opentraces.ai
+Project-URL: Repository, https://github.com/JayFarei/opentraces
+License-Expression: MIT
+Requires-Python: >=3.10
+Requires-Dist: pydantic>=2.0
+Description-Content-Type: text/markdown
+
+# opentraces-schema
+
+Pydantic v2 models for the opentraces.ai JSONL trace format.
+
+## Install
+
+```bash
+pip install -e packages/opentraces-schema
+```
+
+## Usage
+
+```python
+from opentraces_schema import TraceRecord, SCHEMA_VERSION
+
+record = TraceRecord(
+    trace_id="abc-123",
+    session_id="sess-456",
+    agent={"name": "claude-code", "version": "1.0.32"},
+)
+line = record.to_jsonl_line()
+```
+
+## Version
+
+The schema version (`0.1.0`) lives in `src/opentraces_schema/version.py` as the
+single source of truth. See [VERSION-POLICY.md](VERSION-POLICY.md) for semver
+semantics and the bump checklist.
+
+## Schema Rationale
+
+Every version of the schema ships with a rationale document explaining why each
+model and field exists, grounded in public standards (ATIF, Agent Trace, ADP, OTel)
+and empirical observations from real agent traces.
+
+The current rationale is [RATIONALE-0.1.0.md](RATIONALE-0.1.0.md). Each new version
+will have its own rationale file linked from the [CHANGELOG](CHANGELOG.md).
+
+## Contributing
+
+Schema feedback, questions, and proposals are welcome via
+[GitHub Issues](https://github.com/opentraces/opentraces/issues). When suggesting
+a schema change, please include:
+
+- **What** field or model you would add, change, or remove
+- **Why** it matters for your use case (training, analytics, attribution, etc.)
+- **How** it relates to existing standards (ATIF, Agent Trace, ADP, OTel) if applicable
+
+Breaking changes (field renames, removals, type changes) require a major version bump.
+New optional fields and models are minor bumps. See [VERSION-POLICY.md](VERSION-POLICY.md)
+for details.
+
+## Documentation
+
+- [CHANGELOG.md](CHANGELOG.md) - What changed in each version
+- [VERSION-POLICY.md](VERSION-POLICY.md) - What version numbers mean for a schema package
+- [RATIONALE-0.1.0.md](RATIONALE-0.1.0.md) - Design rationale for v0.1.0

opentraces_schema-0.1.0/RATIONALE-0.1.0.md

@@ -0,0 +1,264 @@
+# Schema Rationale: opentraces-schema v0.1.0
+
+Why this schema is what it is. Each section connects a design decision to the
+standards, empirical observations, and constraints that motivated it.
+
+Standards referenced:
+- [Agent Trace spec](https://github.com/nichochar/agent-trace) (Cursor RFC, CC BY 4.0)
+- [ATIF v1.6](https://github.com/harbor-ai/agent-trajectory-interchange-format) (Agent Trajectory Interchange Format)
+- [ADP](https://arxiv.org/abs/2410.10762) (Agent Data Protocol)
+- [OTel GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
+
+
+## TraceRecord (Top-Level)
+
+### Why an independent schema, not ATIF-native
+
+Three existing standards each serve a different purpose:
+- **ATIF** optimizes for training pipelines (token IDs, logprobs)
+- **Agent Trace** captures attribution only (who wrote which lines)
+- **OTel** captures observability only (latency, error rates)
+
+No single standard covers trajectory + attribution + security + environment.
+This schema bridges all three as a superset, with `opentraces export --format atif`
+planned for standards-compatible output.
+
+### Why content_hash (SHA-256)
+
+With sharded JSONL upload (one file per push, never append to existing shards),
+dedup must happen at record level. Existing community trace datasets on HuggingFace
+use content hashing for deduplication at upload time.
+
+SHA-256 chosen over cheaper hashes because traces are large enough that hash
+computation time is negligible vs I/O. The `compute_content_hash()` method excludes
+`content_hash` and `trace_id` so re-parsing identical source data produces the same
+hash regardless of the random UUID assigned.
+
+### Why session_id + trace_id (two IDs)
+
+- `session_id`: the agent's native session identifier, stable across re-parsing
+- `trace_id`: random UUID for database-level uniqueness
+
+Agent sessions group many API calls under a single session ID, but a given session
+may be re-ingested multiple times (e.g. re-exported after a schema upgrade). Separate
+IDs allow re-ingestion without collision while preserving join keys to the original
+agent session.
+
+
+## Steps: TAO-Loop Oriented
+
+### Why `steps` not `turns`
+
+Each Step represents one LLM API call (request + response), not a conversational turn.
+Multi-agent coding sessions routinely involve 50-100+ API calls spanning multiple
+parallel sub-agents within a single user-visible "conversation." Conversational turns
+would collapse this hierarchy into a flat sequence, losing the architectural signal
+needed for caching analysis and training data segmentation.
+
+Both ATIF and ADP use step-based models. The TAO (Thought-Action-Observation) loop
+has converged as the canonical trajectory primitive across agent frameworks
+(OpenHands, SWE-Agent, AgentLab) and community datasets.
+
+### Why `role: "agent"` not `"assistant"`
+
+ATIF and the broader agent community convention use `system | user | agent`. "Agent"
+is semantically accurate for autonomous coding agents that reason, act, and observe
+in loops, as opposed to chat assistants that respond to single prompts.
+
+### Why parent_step + agent_role + subagent_trajectory_ref
+
+Multi-agent coding systems exhibit a hierarchical phase structure:
+
+1. **Warm-up**: cache priming calls with no reasoning
+2. **Main agent**: full system prompt, full tool set
+3. **Explore**: parallel sub-agents with fresh context, reduced tool sets, role-specific prompts
+4. **Plan**: receives only summarized explore findings, not raw context
+5. **Execute**: main agent follows plan as checklist
+
+Sub-agents receive fresh context (not the parent's) and a subset of tools. This
+hierarchical context isolation creates stable prefixes within each sub-agent loop,
+which is why multi-agent architectures achieve high prefix cache reuse rates.
+
+Three fields capture this hierarchy:
+- `parent_step`: tree edge (which main-agent step spawned this sub-agent)
+- `agent_role`: phase label (main, explore, plan) for filtering without reading system prompts
+- `subagent_trajectory_ref`: links to a separate TraceRecord when sub-agent trajectories are stored independently
+
+ATIF provides the `subagent_trajectory_ref` pattern for multi-agent delegation.
+
+### Why call_type: main | subagent | warmup
+
+Multi-agent systems include warm-up calls that exist purely to seed the KV cache.
+They contain no reasoning and produce empty or minimal output, but are architecturally
+significant: they explain why later calls achieve high prefix reuse.
+
+For training data, warm-up calls should be filterable (they add noise to SFT datasets).
+For caching analysis, they are essential. `call_type` enables both use cases.
+
+### Why system_prompt_hash + top-level system_prompts dict
+
+In multi-agent sessions, system prompts can be 20K+ tokens and repeat identically
+across every call within a sub-agent phase. Storing inline would multiply storage
+dramatically for long sessions.
+
+Hash-keyed deduplication: store each unique system prompt once in a top-level dict,
+reference by hash in each step. This separates the queryable step metadata from
+the bulk content.
+
+### Why reasoning_content as explicit field
+
+Extended thinking and chain-of-thought content is returned in a separate field by
+LLM APIs that support it. Benchmark evaluations suggest that including hidden
+reasoning in training data improves downstream task performance.
+
+A dedicated `reasoning_content` field preserves the API-level separation for training
+pipelines that may want to include or exclude chain-of-thought independently of the
+main content.
+
+### Why output_summary on Observation
+
+In multi-agent architectures, downstream sub-agents (e.g. plan phase) receive only
+summarized findings from upstream sub-agents (e.g. explore phase), not raw results.
+This is an information bottleneck pattern.
+
+`output_summary` serves the same purpose for trace consumers: scan summaries to assess
+relevance without loading full tool outputs (which can be megabytes for file reads or
+grep results over large codebases).
+
+
+## ToolCall and Observation: Separated
+
+### Why tool calls and observations are separate lists
+
+Training pipelines (SFT, RLHF) depend on clean tool_call / tool_result separation
+for learning tool selection and result interpretation as distinct capabilities.
+ATIF and ADP both maintain this separation. Observations link back via `source_call_id`
+for 1:1 matching. Dangling tool calls (agent requested a tool but no result was
+recorded) are marked with `error: "no_result"` rather than dropped.
+
+
+## TokenUsage: Cache-Aware
+
+### Why prefix_reuse_tokens, cache_read_tokens, cache_write_tokens
+
+Prefix reuse is the dominant cost driver for multi-agent architectures. Empirical
+measurements show that well-structured hierarchical agents achieve 90%+ prefix reuse
+(representing 80%+ cost savings), while agents with dynamic mid-prompt mutations
+achieve under 50%, and template-based agents under 5%.
+
+| Architecture pattern | Typical prefix reuse | Implication |
+|---------------------|---------------------|-------------|
+| Hierarchical multi-agent (stable prefixes) | 90%+ | Well-structured, cache-friendly |
+| Dynamic memory mutation | ~40-50% | Memory updates break prefix alignment |
+| Template-based prompting | <5% | Variable insertion near prompt start destroys caching |
+
+Per-step cache breakdown enables phase-level cost analysis and cross-architecture
+comparisons.
+
+
+## Outcome: RL-Ready Signals
+
+### Why success + signal_source + signal_confidence
+
+The training community needs trajectory-level reward signals for RLHF, DPO, and RLVR.
+Most existing community trace datasets lack outcome fields entirely, making them
+unsuitable for reward modeling without manual annotation.
+
+Three confidence tiers communicate trustworthiness:
+- **derived**: deterministic extraction (e.g. `committed` from git state)
+- **inferred**: heuristic-based (e.g. success from test output patterns)
+- **annotated**: human or CI label
+
+This lets training pipelines filter by confidence: use only `derived` signals for
+high-confidence reward, include `inferred` for larger but noisier datasets.
+
+### Why committed + commit_sha
+
+The cheapest deterministic quality signal available: did the agent's changes get
+committed to git? Derivable from git state with zero annotation cost, zero LLM
+enrichment. Aligns with the project principle: zero required annotation, all
+enrichment in v0.1.0 is deterministic.
+
+
+## Attribution: Embedded Agent Trace Block
+
+### Why embed attribution rather than link externally
+
+The Agent Trace spec stores attribution in a separate file from trajectories.
+This schema bridges both: trajectory (process) AND attribution (output) in a single
+record. Embedding keeps traces self-contained for dataset consumption, where each
+JSONL line should be independently useful without external file lookups.
+
+Marked `experimental: true` because attribution confidence varies by session
+complexity: single-file edits produce high-confidence attribution, multi-file
+refactors with interleaved tool calls produce lower confidence.
+
+### Why murmur3 for AttributionRange.content_hash
+
+The Agent Trace spec uses `algorithm:value` format (e.g. `murmur3:9f2e8a1b`) for
+position-independent content tracking. murmur3 is fast and non-cryptographic,
+sufficient for detecting code movement across refactors and file renames.
+
+SHA-256 is used at the trace level (collision resistance needed for dedup integrity),
+murmur3 at the line-range level (speed needed, no security requirement).
+
+
+## SecurityMetadata: 3-Tier
+
+### Why a 3-tier security model
+
+- **Tier 1** (open): full content, suitable for public repos and open-source projects
+- **Tier 2** (guarded): redacted secrets, anonymized paths/usernames
+- **Tier 3** (strict): structural metadata only, no content
+
+Existing trace-sharing tools typically offer a single redaction mode: everything is
+processed the same way. A tier system enables per-project configuration with
+per-session override, so a user working on both open-source and proprietary code
+can publish traces from both with appropriate protection levels.
+
+
+## Environment and VCS
+
+### Why capture OS, shell, language_ecosystem
+
+Reproducibility: the same task on macOS vs Linux may produce different agent behavior
+(different shell commands, different tool availability, different file paths).
+Filtering: researchers can select traces by ecosystem (Python, TypeScript, Rust)
+to build domain-specific training datasets without parsing file extensions from
+tool call arguments.
+
+
+## Deliberate Exclusions
+
+### No token IDs or logprobs
+
+ATIF v1.6 includes `prompt_token_ids`, `completion_token_ids`, and `logprobs` per
+step, enabling RL without retokenization drift. These fields are not available from
+agent CLI tools (Claude Code, Cursor, Codex CLI, Gemini CLI) because they intercept
+at the application layer, not the inference layer.
+
+Planned for inclusion when agent APIs expose them. Until then, ATIF export
+(`opentraces export --format atif`) will bridge this gap for training pipelines
+that need token-level data from other sources.
+
+### No OTel span IDs
+
+OTel provides distributed tracing metadata (trace_id, span_id, parent_span_id)
+designed for production observability across microservices. Our `trace_id` is
+session-scoped, not request-scoped. OTel interop is planned via export, not
+native embedding, because the primary consumers of this schema are training
+pipelines and researchers, not production monitoring dashboards.
+
+### No AGENTS.md content
+
+AGENTS.md is a project-level instruction file for agents, not trace data. While
+it provides context for understanding agent behavior, embedding it in every trace
+record would be wasteful. It may be referenced via the `metadata` dict in future
+versions if there is demand.
+
+### No LLM enrichment fields
+
+Fields like `task_description`, `domain_tags`, and `task_type` are not in v0.1.0
+because they require LLM inference to generate. The project principle is zero
+required annotation: all enrichment in v0.1.0 is deterministic. LLM-enriched
+metadata may be added as optional fields in a future minor version.

opentraces_schema-0.1.0/README.md

@@ -0,0 +1,57 @@
+# opentraces-schema
+
+Pydantic v2 models for the opentraces.ai JSONL trace format.
+
+## Install
+
+```bash
+pip install -e packages/opentraces-schema
+```
+
+## Usage
+
+```python
+from opentraces_schema import TraceRecord, SCHEMA_VERSION
+
+record = TraceRecord(
+    trace_id="abc-123",
+    session_id="sess-456",
+    agent={"name": "claude-code", "version": "1.0.32"},
+)
+line = record.to_jsonl_line()
+```
+
+## Version
+
+The schema version (`0.1.0`) lives in `src/opentraces_schema/version.py` as the
+single source of truth. See [VERSION-POLICY.md](VERSION-POLICY.md) for semver
+semantics and the bump checklist.
+
+## Schema Rationale
+
+Every version of the schema ships with a rationale document explaining why each
+model and field exists, grounded in public standards (ATIF, Agent Trace, ADP, OTel)
+and empirical observations from real agent traces.
+
+The current rationale is [RATIONALE-0.1.0.md](RATIONALE-0.1.0.md). Each new version
+will have its own rationale file linked from the [CHANGELOG](CHANGELOG.md).
+
+## Contributing
+
+Schema feedback, questions, and proposals are welcome via
+[GitHub Issues](https://github.com/opentraces/opentraces/issues). When suggesting
+a schema change, please include:
+
+- **What** field or model you would add, change, or remove
+- **Why** it matters for your use case (training, analytics, attribution, etc.)
+- **How** it relates to existing standards (ATIF, Agent Trace, ADP, OTel) if applicable
+
+Breaking changes (field renames, removals, type changes) require a major version bump.
+New optional fields and models are minor bumps. See [VERSION-POLICY.md](VERSION-POLICY.md)
+for details.
+
+## Documentation
+
+- [CHANGELOG.md](CHANGELOG.md) - What changed in each version
+- [VERSION-POLICY.md](VERSION-POLICY.md) - What version numbers mean for a schema package
+- [RATIONALE-0.1.0.md](RATIONALE-0.1.0.md) - Design rationale for v0.1.0

opentraces_schema-0.1.0/VERSION-POLICY.md

@@ -0,0 +1,39 @@
+# Version Policy
+
+opentraces-schema follows Semantic Versioning (semver) with schema-specific semantics.
+
+## What the version numbers mean
+
+- **MAJOR** (X.0.0): Breaking changes to existing fields. Renaming, removing, or
+  changing the type of an existing field. Consumers must update parsers.
+- **MINOR** (0.X.0): New optional fields, new models, new enum values added to
+  existing Literal types. Existing parsers continue to work without changes.
+- **PATCH** (0.0.X): Docstring fixes, validation constraint adjustments that do
+  not change the serialized format, bug fixes in computed fields.
+
+## Pre-1.0 stability
+
+During 0.x development, MINOR bumps may include breaking changes. The schema is
+not yet stable. Pin to exact versions (`opentraces-schema==0.1.0`) rather than
+ranges until 1.0.
+
+## Where the version lives
+
+The single source of truth is `src/opentraces_schema/version.py`. The `SCHEMA_VERSION`
+constant is used by:
+
+- `pyproject.toml` via hatch dynamic versioning
+- `TraceRecord.schema_version` default
+- `Attribution.version` default
+
+Note: `TraceRecord.schema_version` and `Attribution.version` are currently coupled
+to the same `SCHEMA_VERSION`. If the attribution spec needs to version independently
+in the future, this coupling can be broken by introducing a separate constant.
+
+## Bump checklist
+
+1. Update `SCHEMA_VERSION` in `src/opentraces_schema/version.py`
+2. Add entry to `CHANGELOG.md` under `[Unreleased]`, then move to new version header
+3. Create `RATIONALE-{VERSION}.md` documenting design decisions for the new version
+4. Link the rationale file from the CHANGELOG entry
+5. Tag the commit: `git tag schema-v{VERSION}`

opentraces_schema-0.1.0/pyproject.toml

@@ -0,0 +1,25 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "opentraces-schema"
+dynamic = ["version"]
+description = "Schema models for the opentraces.ai agent trace JSONL format"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+dependencies = [
+    "pydantic>=2.0",
+]
+
+[project.urls]
+Homepage = "https://opentraces.ai"
+Repository = "https://github.com/JayFarei/opentraces"
+
+[tool.hatch.version]
+path = "src/opentraces_schema/version.py"
+pattern = "SCHEMA_VERSION = \"(?P<version>[^\"]+)\""
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/opentraces_schema"]

opentraces_schema-0.1.0/src/opentraces_schema/__init__.py

@@ -0,0 +1,43 @@
+"""opentraces-schema: Pydantic models for the opentraces.ai JSONL trace format."""
+
+from .models import (
+    Agent,
+    Attribution,
+    AttributionConversation,
+    AttributionFile,
+    AttributionRange,
+    Environment,
+    Metrics,
+    Observation,
+    Outcome,
+    SecurityMetadata,
+    Snippet,
+    Step,
+    Task,
+    TokenUsage,
+    ToolCall,
+    TraceRecord,
+    VCS,
+)
+from .version import SCHEMA_VERSION
+
+__all__ = [
+    "Agent",
+    "Attribution",
+    "AttributionConversation",
+    "AttributionFile",
+    "AttributionRange",
+    "Environment",
+    "Metrics",
+    "Observation",
+    "Outcome",
+    "SCHEMA_VERSION",
+    "SecurityMetadata",
+    "Snippet",
+    "Step",
+    "Task",
+    "TokenUsage",
+    "ToolCall",
+    "TraceRecord",
+    "VCS",
+]

opentraces_schema-0.1.0/src/opentraces_schema/models.py

@@ -0,0 +1,242 @@
+"""Pydantic v2 models for the opentraces.ai JSONL trace schema.
+
+This module defines the complete schema for enriched agent session traces.
+Each TraceRecord represents one complete agent session or task unit.
+
+The schema is informed by ATIF v1.6, ADP, Agent Trace spec, and field patterns
+found in existing HF datasets (nlile, Nebius, CoderForge).
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from typing import Any, Literal
+
+from pydantic import BaseModel, Field
+
+from .version import SCHEMA_VERSION
+
+
+class Task(BaseModel):
+    """Structured task metadata for filtering and grouping."""
+
+    description: str | None = None
+    source: str | None = Field(None, description="user_prompt, cli_arg, skill, etc.")
+    repository: str | None = Field(None, description="owner/repo format")
+    base_commit: str | None = None
+
+
+class Agent(BaseModel):
+    """Agent identity following provider/model convention."""
+
+    name: str = Field(description="Agent identifier: claude-code, cursor, codex, etc.")
+    version: str | None = None
+    model: str | None = Field(None, description="provider/model-name, e.g. anthropic/claude-sonnet-4-20250514")
+
+
+class VCS(BaseModel):
+    """Version control metadata. type='none' when not in a git repo."""
+
+    type: Literal["git", "none"] = "none"
+    base_commit: str | None = None
+    branch: str | None = None
+    diff: str | None = Field(None, description="Unified diff string or null")
+
+
+class Environment(BaseModel):
+    """Runtime environment metadata for filtering and reproducibility."""
+
+    os: str | None = None
+    shell: str | None = None
+    vcs: VCS = Field(default_factory=VCS)
+    language_ecosystem: list[str] = Field(default_factory=list)
+
+
+class ToolCall(BaseModel):
+    """A single tool invocation within a step."""
+
+    tool_call_id: str
+    tool_name: str
+    input: dict[str, Any] = Field(default_factory=dict)
+    duration_ms: int | None = None
+
+
+class Observation(BaseModel):
+    """Result of a tool call, linked back via source_call_id."""
+
+    source_call_id: str
+    content: str | None = None
+    output_summary: str | None = Field(None, description="Lightweight preview of tool result")
+    error: str | None = Field(None, description="Error info, e.g. 'no_result' for dangling tool calls")
+
+
+class Snippet(BaseModel):
+    """Code block extracted from tool results or agent responses."""
+
+    file_path: str
+    start_line: int | None = None
+    end_line: int | None = None
+    language: str | None = None
+    text: str | None = None
+    source_step: int | None = Field(None, description="Step index that produced this snippet")
+
+
+class TokenUsage(BaseModel):
+    """Per-step token usage breakdown for cost and efficiency analysis."""
+
+    input_tokens: int = 0
+    output_tokens: int = 0
+    cache_read_tokens: int = 0
+    cache_write_tokens: int = 0
+    prefix_reuse_tokens: int = 0
+
+
+class Step(BaseModel):
+    """A single LLM API call (request + response) in the TAO loop.
+
+    Each step represents one thought-action-observation cycle, not a
+    conversational turn. This aligns with ATIF's step-based model.
+    """
+
+    step_index: int
+    role: Literal["system", "user", "agent"]
+    content: str | None = None
+    reasoning_content: str | None = Field(None, description="Chain-of-thought / extended thinking")
+    model: str | None = None
+    system_prompt_hash: str | None = Field(None, description="Key into top-level system_prompts map")
+    agent_role: str | None = Field(None, description="main, explore, plan, etc.")
+    parent_step: int | None = Field(None, description="Step index of parent for sub-agent hierarchy")
+    call_type: Literal["main", "subagent", "warmup"] | None = None
+    subagent_trajectory_ref: str | None = Field(None, description="Session ID of sub-agent trajectory")
+    tools_available: list[str] = Field(default_factory=list)
+    tool_calls: list[ToolCall] = Field(default_factory=list)
+    observations: list[Observation] = Field(default_factory=list)
+    snippets: list[Snippet] = Field(default_factory=list)
+    token_usage: TokenUsage = Field(default_factory=TokenUsage)
+    timestamp: str | None = None
+
+
+class Outcome(BaseModel):
+    """Session outcome signals for RL/reward modeling.
+
+    signal_confidence indicates how trustworthy the signal is:
+    - derived: deterministic extraction (e.g. committed from git)
+    - inferred: heuristic-based (e.g. success from test output patterns)
+    - annotated: human or CI annotation
+    """
+
+    success: bool | None = None
+    signal_source: str = "deterministic"
+    signal_confidence: Literal["derived", "inferred", "annotated"] = "derived"
+    description: str | None = None
+    patch: str | None = Field(None, description="Unified diff produced by the session")
+    committed: bool = False
+    commit_sha: str | None = None
+
+
+class AttributionRange(BaseModel):
+    """A range of lines attributed to an agent conversation."""
+
+    start_line: int
+    end_line: int
+    content_hash: str | None = Field(None, description="murmur3 hash for cross-refactor tracking")
+    confidence: Literal["high", "medium", "low"] | None = None
+
+
+class AttributionConversation(BaseModel):
+    """Links attributed code ranges to the conversation that produced them."""
+
+    contributor: dict[str, str] = Field(
+        default_factory=dict,
+        description="e.g. {type: 'ai', model_id: 'anthropic/claude-sonnet-4-20250514'}",
+    )
+    url: str | None = Field(None, description="opentraces://trace_id/step_N")
+    ranges: list[AttributionRange] = Field(default_factory=list)
+
+
+class AttributionFile(BaseModel):
+    """Attribution data for a single file."""
+
+    path: str
+    conversations: list[AttributionConversation] = Field(default_factory=list)
+
+
+class Attribution(BaseModel):
+    """Embedded Agent Trace-compatible attribution block.
+
+    Bridges trajectory (process) and attribution (output). Records which
+    files and line ranges were produced by the agent session.
+
+    Marked experimental in v0.1 - confidence varies by session complexity.
+    """
+
+    version: str = SCHEMA_VERSION
+    experimental: bool = True
+    files: list[AttributionFile] = Field(default_factory=list)
+
+
+class Metrics(BaseModel):
+    """Aggregated session-level metrics for analytics and cost modeling."""
+
+    total_steps: int = 0
+    total_input_tokens: int = 0
+    total_output_tokens: int = 0
+    total_duration_s: float | None = None
+    cache_hit_rate: float | None = Field(None, ge=0.0, le=1.0)
+    estimated_cost_usd: float | None = None
+
+
+class SecurityMetadata(BaseModel):
+    """Records what security processing was applied and what was flagged/redacted."""
+
+    scanned: bool = False
+    flags_reviewed: int = 0
+    redactions_applied: int = 0
+    classifier_version: str | None = None
+
+
+class TraceRecord(BaseModel):
+    """Top-level model for one complete agent session trace.
+
+    Each line in the JSONL file is one TraceRecord. The schema bridges
+    trajectory data (ATIF/ADP) with code attribution (Agent Trace spec),
+    creating the complete record of process + output.
+    """
+
+    schema_version: str = SCHEMA_VERSION
+    trace_id: str
+    session_id: str
+    content_hash: str | None = None
+    timestamp_start: str | None = None
+    timestamp_end: str | None = None
+    task: Task = Field(default_factory=Task)
+    agent: Agent
+    environment: Environment = Field(default_factory=Environment)
+    system_prompts: dict[str, str] = Field(
+        default_factory=dict,
+        description="Deduplicated system prompts keyed by hash",
+    )
+    tool_definitions: list[dict[str, Any]] = Field(default_factory=list)
+    steps: list[Step] = Field(default_factory=list)
+    outcome: Outcome = Field(default_factory=Outcome)
+    dependencies: list[str] = Field(default_factory=list)
+    metrics: Metrics = Field(default_factory=Metrics)
+    security: SecurityMetadata = Field(default_factory=SecurityMetadata)
+    attribution: Attribution | None = None
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+    def compute_content_hash(self) -> str:
+        """Compute SHA-256 hash of the trace content for deduplication.
+
+        Excludes content_hash and trace_id so re-parsing identical content
+        produces the same hash regardless of the random UUID assigned.
+        """
+        data = self.model_dump(exclude={"content_hash", "trace_id"})
+        serialized = json.dumps(data, sort_keys=True, default=str)
+        return hashlib.sha256(serialized.encode()).hexdigest()
+
+    def to_jsonl_line(self) -> str:
+        """Serialize to a single JSONL line with computed content_hash."""
+        self.content_hash = self.compute_content_hash()
+        return self.model_dump_json()

opentraces_schema-0.1.0/src/opentraces_schema/version.py

@@ -0,0 +1,3 @@
+"""Schema version for opentraces JSONL format."""
+
+SCHEMA_VERSION = "0.1.0"