agent-apprenticeship 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +269 -3
- package/bin/agent-apprenticeship.js +1 -2
- package/package.json +4 -4
- package/pyproject.toml +10 -4
- package/schemas/environment_replay.schema.json +41 -0
- package/schemas/eval_package.schema.json +37 -0
- package/schemas/experience_compilation.schema.json +53 -0
- package/schemas/runtime_training.schema.json +33 -0
- package/schemas/source_evidence_map.schema.json +25 -0
- package/schemas/trace.schema.json +904 -0
- package/schemas/trace_step.schema.json +622 -0
- package/schemas/training_rows.schema.json +29 -0
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/__init__.py +1 -1
- package/src/agent_apprenticeship/agent_env_exporter.py +53 -0
- package/src/agent_apprenticeship/agent_env_verify.py +32 -0
- package/src/agent_apprenticeship/atif.py +152 -0
- package/src/agent_apprenticeship/atif_models/LICENSE +201 -0
- package/src/agent_apprenticeship/atif_models/__init__.py +38 -0
- package/src/agent_apprenticeship/atif_models/agent.py +35 -0
- package/src/agent_apprenticeship/atif_models/content.py +159 -0
- package/src/agent_apprenticeship/atif_models/final_metrics.py +43 -0
- package/src/agent_apprenticeship/atif_models/metrics.py +47 -0
- package/src/agent_apprenticeship/atif_models/observation.py +19 -0
- package/src/agent_apprenticeship/atif_models/observation_result.py +45 -0
- package/src/agent_apprenticeship/atif_models/step.py +142 -0
- package/src/agent_apprenticeship/atif_models/subagent_trajectory_ref.py +88 -0
- package/src/agent_apprenticeship/atif_models/tool_call.py +34 -0
- package/src/agent_apprenticeship/atif_models/trajectory.py +196 -0
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/batch_runner.py +1 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/bundle_exporter.py +8 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/certification.py +1 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/cli.py +26 -7
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/codex_runner.py +17 -17
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/config.py +2 -2
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/ecosystem_r2.py +3 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/evaluator.py +8 -27
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/grader.py +6 -5
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/integration_status.py +1 -1
- package/src/agent_apprenticeship/io.py +104 -0
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/learning.py +7 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/loop.py +5 -5
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/loop_review.py +1 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/mentor_checkpoints.py +2 -2
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/package_exporter.py +5 -3
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/progress.py +1 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/public_run.py +1 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/public_sanitizer.py +2 -2
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/release_exporter.py +7 -4
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/rubric_generation.py +1 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/schemas.py +17 -4
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/session_events.py +4 -4
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/task_intake.py +1 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/tdo.py +5 -1
- package/src/agent_apprenticeship/trace_normalizer.py +103 -0
- package/src/agent_apprenticeship/trace_prompt.py +48 -0
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/training_signals.py +1 -1
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/validation.py +3 -3
- package/src/{agent_apprenticeship_trace → agent_apprenticeship}/verifier.py +6 -5
- package/docs/bioactive_evidence_mapping_schema.md +0 -113
- package/docs/dataset_schema.md +0 -160
- package/docs/local_usage.md +0 -191
- package/docs/scaling_to_1000_tasks.md +0 -3
- package/docs/trace_schema.md +0 -192
- package/docs/v0.2_release_notes.md +0 -61
- package/src/agent_apprenticeship_trace/io.py +0 -20
- package/src/agent_apprenticeship_trace/trace_normalizer.py +0 -1026
- package/src/agent_apprenticeship_trace/trace_prompt.py +0 -78
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/actual_outputs_normalizer.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/apprentice_adapters.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/artifact_capture.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/artifact_previews.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/artifact_resolver.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/command_discovery.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/contract_diagnostics.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/contract_verifier.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/env.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/lesson_extractor.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/llm_output_normalizer.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/openai_structured.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/recipes.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/revision.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/role_runners.py +0 -0
- /package/src/{agent_apprenticeship_trace → agent_apprenticeship}/version.py +0 -0
|
@@ -6,7 +6,7 @@ from datetime import datetime, timezone
|
|
|
6
6
|
from pathlib import Path
|
|
7
7
|
from typing import Any, Callable, Literal
|
|
8
8
|
|
|
9
|
-
from .io import append_jsonl, read_json, read_jsonl, write_json
|
|
9
|
+
from .io import append_jsonl, read_workflow_json as read_json, read_workflow_jsonl as read_jsonl, write_json
|
|
10
10
|
|
|
11
11
|
RUN_TERMINAL_STATUSES = {"completed", "partial", "failed"}
|
|
12
12
|
|
|
@@ -10,7 +10,7 @@ from pathlib import Path
|
|
|
10
10
|
from .bundle_exporter import create_contribution_bundle
|
|
11
11
|
from .config import Settings, apprentice_agent_display_name, apprentice_agent_readiness_status, get_settings, settings_override
|
|
12
12
|
from .env import redact_secrets
|
|
13
|
-
from .io import read_json, write_json
|
|
13
|
+
from .io import read_workflow_json as read_json, write_json
|
|
14
14
|
from .loop import run_task
|
|
15
15
|
from .loop_review import record_followup_review_update, record_human_loop_feedback
|
|
16
16
|
from .mentor_checkpoints import write_mentor_checkpoints
|
|
@@ -3,9 +3,9 @@ import hashlib, json, re, shutil
|
|
|
3
3
|
from pathlib import Path
|
|
4
4
|
from typing import Any
|
|
5
5
|
from .env import redact_secrets, contains_secret
|
|
6
|
-
from .io import read_jsonl, append_jsonl, write_json
|
|
6
|
+
from .io import read_workflow_jsonl as read_jsonl, append_jsonl, write_json
|
|
7
7
|
|
|
8
|
-
PROMPT_TEMPLATE_ID='
|
|
8
|
+
PROMPT_TEMPLATE_ID='agent_apprenticeship_worker_v0'
|
|
9
9
|
PROMPT_TEMPLATE_VERSION='0.1'
|
|
10
10
|
PROMPT_PUBLICATION_STATUS='template_available_in_codebase'
|
|
11
11
|
RAW_LOG_NAMES=set()
|
|
@@ -1,10 +1,11 @@
|
|
|
1
1
|
from __future__ import annotations
|
|
2
2
|
import shutil
|
|
3
3
|
from pathlib import Path
|
|
4
|
-
from .io import read_json, write_json, append_jsonl, read_jsonl
|
|
4
|
+
from .io import read_workflow_json as read_json, write_json, append_jsonl, read_workflow_jsonl as read_jsonl
|
|
5
5
|
from .public_sanitizer import create_public_release
|
|
6
6
|
from .package_exporter import IGNORED_RELEASE_DIR_NAMES, is_ignored_release_path, public_task_record
|
|
7
7
|
from .trace_normalizer import normalize_trace_for_export
|
|
8
|
+
from .atif import to_workflow
|
|
8
9
|
|
|
9
10
|
RELEASE_FILES=['full_task_records.jsonl','tasks.jsonl','task_intake_specs.jsonl','rubrics.jsonl','rubric_items.jsonl','raw_agent_traces.jsonl','agent_traces.jsonl','trace_normalization_reports.jsonl','actual_outputs_normalization_reports.jsonl','actual_outputs.jsonl','grader_results.jsonl','verifier_results.jsonl','evaluator_feedback.jsonl','revision_plans.jsonl','hillclimb_results.jsonl','lessons.jsonl','training_signals.jsonl','process_supervision.jsonl','reward_modeling.jsonl','verifier_training.jsonl','revision_preference_pairs.jsonl','role_results_index.jsonl','artifacts_index.json','packages_index.jsonl','forsy_like_collections.jsonl']
|
|
10
11
|
|
|
@@ -141,7 +142,7 @@ def _propagate_release_status_fields(release_root):
|
|
|
141
142
|
_write_jsonl(path, rows)
|
|
142
143
|
|
|
143
144
|
# Do not inject task_status into agent_traces.jsonl here.
|
|
144
|
-
# Trace rows are governed by the
|
|
145
|
+
# Trace rows are governed by the WorkflowTrace schema; attempt_status belongs
|
|
145
146
|
# there, but task_status is task/package-level metadata and is exported via
|
|
146
147
|
# tasks.jsonl and packages_index.jsonl.
|
|
147
148
|
|
|
@@ -190,14 +191,14 @@ def create_release(run_root: Path, release_root: Path) -> Path:
|
|
|
190
191
|
if norm is not None:
|
|
191
192
|
normalized_trace_count += 1; normalized_trace_step_count += len(norm.get('steps') or [])
|
|
192
193
|
if tr is not None:
|
|
193
|
-
tr_row=normalize_trace_for_export(dict(tr), report)
|
|
194
|
+
tr_row=to_workflow(normalize_trace_for_export(dict(tr), report))
|
|
194
195
|
attempt_statuses.append(tr_row.get('attempt_status'))
|
|
195
196
|
tr_row['iteration_index']=0 if a == 'baseline' else 1
|
|
196
197
|
tr_row['previous_attempt_id']=None if a == 'baseline' else manifest.get('baseline_attempt_id')
|
|
197
198
|
tr_row['revision_group_id']=tid
|
|
198
199
|
tr_row['completion_reason']=manifest.get('loop_stop_reason')
|
|
199
200
|
append_jsonl(release_root/'agent_traces.jsonl', tr_row)
|
|
200
|
-
append_jsonl(release_root/'forsy_like_collections.jsonl', {'collection_id': tr.get('collection_id') or tid, 'trace_id': tr.get('trace_id'), 'attempt_kind': a, 'iteration_index': tr_row['iteration_index'], 'previous_attempt_id': tr_row['previous_attempt_id'], '
|
|
201
|
+
append_jsonl(release_root/'forsy_like_collections.jsonl', {'collection_id': tr.get('collection_id') or tid, 'trace_id': tr.get('trace_id'), 'attempt_kind': a, 'iteration_index': tr_row['iteration_index'], 'previous_attempt_id': tr_row['previous_attempt_id'], 'trajectory': normalize_trace_for_export(tr, report), 'trace_ref': f'packages/{tid}/attempts/{a}/agent_trace.json'})
|
|
201
202
|
else:
|
|
202
203
|
missing_traces += 1; blockers.append(f'missing attempts/{a}/agent_trace.json')
|
|
203
204
|
attempt_trace_refs[a]={'raw': f'packages/{tid}/attempts/{a}/agent_trace.raw.json' if (pkg/f'attempts/{a}/agent_trace.raw.json').exists() else None, 'normalized': f'packages/{tid}/attempts/{a}/agent_trace.normalized.json' if (pkg/f'attempts/{a}/agent_trace.normalized.json').exists() else None, 'canonical': f'packages/{tid}/attempts/{a}/agent_trace.json' if (pkg/f'attempts/{a}/agent_trace.json').exists() else None, 'normalization_report': f'packages/{tid}/attempts/{a}/trace_normalization_report.json' if (pkg/f'attempts/{a}/trace_normalization_report.json').exists() else None}
|
|
@@ -256,4 +257,6 @@ def create_release(run_root: Path, release_root: Path) -> Path:
|
|
|
256
257
|
except Exception:
|
|
257
258
|
pass
|
|
258
259
|
_propagate_release_status_fields(release_root)
|
|
260
|
+
from .agent_env_exporter import export_agent_env
|
|
261
|
+
export_agent_env(release_root)
|
|
259
262
|
return release_root
|
|
@@ -4,7 +4,7 @@ from pathlib import Path
|
|
|
4
4
|
from pydantic import BaseModel
|
|
5
5
|
from .schemas import TaskIntakeSpec, RubricItem, RubricSpec, RubricQualityReport
|
|
6
6
|
from .config import apprentice_agent_display_name, get_settings
|
|
7
|
-
from .io import read_json
|
|
7
|
+
from .io import read_workflow_json as read_json
|
|
8
8
|
from .openai_structured import get_model_provider_status, run_structured_role
|
|
9
9
|
from .public_sanitizer import sanitize_public_obj, sha256_text
|
|
10
10
|
|
|
@@ -102,7 +102,7 @@ class ActualOutputs(StrictModel):
|
|
|
102
102
|
error_message: str | None = None
|
|
103
103
|
metadata_json: DictAny = Field(default_factory=dict)
|
|
104
104
|
|
|
105
|
-
class
|
|
105
|
+
class WorkflowTraceStep(StrictModel):
|
|
106
106
|
step: int
|
|
107
107
|
turn: int
|
|
108
108
|
actor: str
|
|
@@ -393,8 +393,18 @@ class AgentTraceStep(StrictModel):
|
|
|
393
393
|
raise ValueError("operation is required for non-user steps")
|
|
394
394
|
return self
|
|
395
395
|
|
|
396
|
-
class
|
|
397
|
-
|
|
396
|
+
class WorkflowTrace(StrictModel):
|
|
397
|
+
"""Internal task/evaluation projection. Persist using ATIF I/O only."""
|
|
398
|
+
|
|
399
|
+
@model_validator(mode="before")
|
|
400
|
+
@classmethod
|
|
401
|
+
def from_atif(cls, value):
|
|
402
|
+
from .atif import to_workflow
|
|
403
|
+
if isinstance(value, dict) and "agent" in value and "steps" in value:
|
|
404
|
+
return to_workflow(value)
|
|
405
|
+
return value
|
|
406
|
+
|
|
407
|
+
schema_version: str = "ATIF-v1.8"
|
|
398
408
|
trace_id: str
|
|
399
409
|
collection_id: str | None = None
|
|
400
410
|
run_id: str | None = None
|
|
@@ -425,7 +435,7 @@ class AgentTrace(StrictModel):
|
|
|
425
435
|
final_output_summary: str | None = None
|
|
426
436
|
learning: str | None = None
|
|
427
437
|
termination_reason: Literal["task_complete","verifier_passed","verifier_failed","max_iterations_reached","agent_blocked","timeout","error_unrecoverable","partial_then_stopped","provider_usage_limit","other"]
|
|
428
|
-
steps: list[
|
|
438
|
+
steps: list[WorkflowTraceStep]
|
|
429
439
|
actual_outputs: ActualOutputs | None = None
|
|
430
440
|
artifacts: list[ArtifactRef] = Field(default_factory=list)
|
|
431
441
|
iteration_index: int | None = None
|
|
@@ -646,3 +656,6 @@ class TDOReport(StrictModel):
|
|
|
646
656
|
generation_notes: list[str] = Field(default_factory=list)
|
|
647
657
|
recommended_uses: list[str] = Field(default_factory=list)
|
|
648
658
|
metadata_json: DictAny = Field(default_factory=dict)
|
|
659
|
+
|
|
660
|
+
# Public trace models use the Harbor interchange contract.
|
|
661
|
+
from .atif_models import Trajectory as AgentTrace, Step as AgentTraceStep
|
|
@@ -7,7 +7,7 @@ from typing import Any, Literal
|
|
|
7
7
|
|
|
8
8
|
from pydantic import BaseModel, ConfigDict, Field
|
|
9
9
|
|
|
10
|
-
from .io import append_jsonl, read_jsonl
|
|
10
|
+
from .io import append_jsonl, read_workflow_jsonl as read_jsonl
|
|
11
11
|
|
|
12
12
|
|
|
13
13
|
SessionEventType = Literal[
|
|
@@ -84,9 +84,9 @@ def backfill_session_event_task_ids(run_root: Path, task_id: str | None) -> None
|
|
|
84
84
|
changed = True
|
|
85
85
|
if not changed:
|
|
86
86
|
return
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
87
|
+
path.write_text("")
|
|
88
|
+
for row in rows:
|
|
89
|
+
append_jsonl(path, row)
|
|
90
90
|
|
|
91
91
|
|
|
92
92
|
def next_followup_index(run_root: Path) -> int:
|
|
@@ -4,7 +4,7 @@ from pathlib import Path
|
|
|
4
4
|
from pydantic import BaseModel
|
|
5
5
|
from .schemas import RawTaskRecord, TaskIntakeSpec, TaskIntakeQualityReport
|
|
6
6
|
from .config import apprentice_agent_display_name, get_settings
|
|
7
|
-
from .io import read_json, write_json
|
|
7
|
+
from .io import read_workflow_json as read_json, write_json
|
|
8
8
|
from .openai_structured import extract_json_object, get_model_provider_status, run_structured_role
|
|
9
9
|
from .public_sanitizer import sanitize_public_obj, sha256_text
|
|
10
10
|
|
|
@@ -14,7 +14,7 @@ from typing import Any
|
|
|
14
14
|
|
|
15
15
|
from .config import Settings, apprentice_agent_display_name, configured_model_provider_ready
|
|
16
16
|
from .env import contains_secret, redact_secrets
|
|
17
|
-
from .io import append_jsonl, read_json, read_jsonl, write_json
|
|
17
|
+
from .io import append_jsonl, read_workflow_json as read_json, read_workflow_jsonl as read_jsonl, write_json
|
|
18
18
|
from .progress import ProgressCallback, append_progress_event
|
|
19
19
|
from .public_sanitizer import sanitize_public_obj, sanitize_public_text
|
|
20
20
|
from .schemas import TDOJudgeResult, TDOReport
|
|
@@ -3277,6 +3277,8 @@ def run_tdo_for_package(package_root: Path, settings: Settings, *, run_root: Pat
|
|
|
3277
3277
|
)
|
|
3278
3278
|
write_json(tdo_dir / "tdo_report.json", report)
|
|
3279
3279
|
write_json(tdo_dir / "tdo_manifest.json", {"schema_version": TDO_VERSION, "tdo_status": "disabled"})
|
|
3280
|
+
from .agent_env_exporter import export_agent_env
|
|
3281
|
+
export_agent_env(package_root)
|
|
3280
3282
|
return report.model_dump(mode="json")
|
|
3281
3283
|
max_rounds = _int_env("AA_EXPERIENCE_COMPILER_MAX_ROUNDS", _int_env("AA_TDO_MAX_ROUNDS", 3))
|
|
3282
3284
|
min_quality = _float_env("AA_EXPERIENCE_COMPILER_MIN_QUALITY", _float_env("AA_TDO_MIN_QUALITY_SCORE", 0.65))
|
|
@@ -3466,6 +3468,8 @@ def run_tdo_for_package(package_root: Path, settings: Settings, *, run_root: Pat
|
|
|
3466
3468
|
write_artifacts_index(package_root)
|
|
3467
3469
|
except Exception:
|
|
3468
3470
|
pass
|
|
3471
|
+
from .agent_env_exporter import export_agent_env
|
|
3472
|
+
export_agent_env(package_root)
|
|
3469
3473
|
return report.model_dump(mode="json")
|
|
3470
3474
|
|
|
3471
3475
|
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
"""Validate inbound ATIF; never guess or repair an incompatible trace schema."""
|
|
2
|
+
from __future__ import annotations
|
|
3
|
+
from pathlib import Path
|
|
4
|
+
from typing import Any
|
|
5
|
+
from pydantic import BaseModel, Field
|
|
6
|
+
from .schemas import ActualOutputs, WorkflowTrace
|
|
7
|
+
from .atif import load_trace, to_workflow, from_workflow, validate_trace
|
|
8
|
+
|
|
9
|
+
class TraceNormalizationContext(BaseModel):
|
|
10
|
+
task_id: str
|
|
11
|
+
attempt_id: str
|
|
12
|
+
attempt_kind: str
|
|
13
|
+
task: str
|
|
14
|
+
actual_outputs: ActualOutputs | None = None
|
|
15
|
+
trace_id_prefix: str = 'trace'
|
|
16
|
+
|
|
17
|
+
class TraceNormalizationReport(BaseModel):
|
|
18
|
+
task_id: str
|
|
19
|
+
attempt_id: str
|
|
20
|
+
attempt_kind: str
|
|
21
|
+
raw_trace_ref: str | None = None
|
|
22
|
+
normalized_trace_ref: str | None = None
|
|
23
|
+
canonical_trace_ref: str | None = None
|
|
24
|
+
trace_schema_valid: bool = False
|
|
25
|
+
trace_normalized: bool = False
|
|
26
|
+
trace_lossless: bool = True
|
|
27
|
+
fallback_trace: bool = False
|
|
28
|
+
raw_step_count: int = 0
|
|
29
|
+
normalized_step_count: int = 0
|
|
30
|
+
discarded_step_count: int = 0
|
|
31
|
+
raw_trace_parse_error: bool = False
|
|
32
|
+
trace_normalization_error: bool = False
|
|
33
|
+
trace_normalization_partial: bool = False
|
|
34
|
+
validation_errors: list[str] = Field(default_factory=list)
|
|
35
|
+
warnings: list[str] = Field(default_factory=list)
|
|
36
|
+
field_mappings: dict[str,str] = Field(default_factory=dict)
|
|
37
|
+
input_dict_to_string_count: int = 0
|
|
38
|
+
output_dict_to_string_count: int = 0
|
|
39
|
+
causal_type_repair_count: int = 0
|
|
40
|
+
operation_repair_count: int = 0
|
|
41
|
+
action_repair_count: int = 0
|
|
42
|
+
metadata_json: dict[str, Any] = Field(default_factory=dict)
|
|
43
|
+
|
|
44
|
+
class NormalizationResult(BaseModel):
|
|
45
|
+
normalized_trace: dict[str, Any] | None = None
|
|
46
|
+
report: TraceNormalizationReport
|
|
47
|
+
fallback_required: bool = False
|
|
48
|
+
parse_error: str | None = None
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
def infer_attempt_status(trace: dict[str, Any], report: dict[str, Any] | None = None) -> str:
|
|
52
|
+
existing=trace.get('attempt_status')
|
|
53
|
+
if existing in {'completed', 'failed', 'blocked', 'fallback', 'partial'}:
|
|
54
|
+
return existing
|
|
55
|
+
termination=trace.get('termination_reason')
|
|
56
|
+
md=trace.get('metadata_json') if isinstance(trace.get('metadata_json'), dict) else {}
|
|
57
|
+
report=report or {}
|
|
58
|
+
if termination == 'task_complete':
|
|
59
|
+
return 'completed'
|
|
60
|
+
if termination == 'agent_blocked':
|
|
61
|
+
return 'blocked'
|
|
62
|
+
actual=trace.get('actual_outputs') if isinstance(trace.get('actual_outputs'), dict) else {}
|
|
63
|
+
if termination in {'timeout','error_unrecoverable','provider_usage_limit','verifier_failed'} or actual.get('status') in {'failed','timeout','error'}:
|
|
64
|
+
return 'failed'
|
|
65
|
+
if report.get('fallback_trace') or md.get('fallback_trace') or md.get('fallback_trace_created') or md.get('fallback_reason') or report.get('fallback_reason') or (isinstance(report.get('metadata_json'), dict) and report['metadata_json'].get('fallback_reason')):
|
|
66
|
+
return 'fallback'
|
|
67
|
+
steps=trace.get('steps') or []
|
|
68
|
+
if any(s.get('step_outcome') == 'blocked' for s in steps if isinstance(s, dict)):
|
|
69
|
+
return 'blocked'
|
|
70
|
+
if any(s.get('step_outcome') == 'failed' or s.get('success') is False for s in steps if isinstance(s, dict)):
|
|
71
|
+
return 'failed' if not any(s.get('success') is True for s in steps if isinstance(s, dict)) else 'partial'
|
|
72
|
+
if steps:
|
|
73
|
+
return 'partial'
|
|
74
|
+
return 'failed'
|
|
75
|
+
|
|
76
|
+
def normalize_trace_for_export(trace: dict, report: dict | None = None) -> dict:
|
|
77
|
+
projection = to_workflow(trace) if "agent" in trace else dict(trace)
|
|
78
|
+
projection["attempt_status"] = infer_attempt_status(projection, report)
|
|
79
|
+
return from_workflow(projection)
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
def repair_agent_trace_file(raw_trace_path: Path, context: TraceNormalizationContext) -> NormalizationResult:
|
|
83
|
+
report = TraceNormalizationReport(task_id=context.task_id, attempt_id=context.attempt_id,
|
|
84
|
+
attempt_kind=context.attempt_kind, raw_trace_ref=raw_trace_path.name)
|
|
85
|
+
try:
|
|
86
|
+
atif = load_trace(raw_trace_path)
|
|
87
|
+
trace = to_workflow(atif)
|
|
88
|
+
trace.update(task_id=context.task_id, attempt_id=context.attempt_id,
|
|
89
|
+
attempt_kind=context.attempt_kind, task=context.task)
|
|
90
|
+
if context.actual_outputs is not None:
|
|
91
|
+
trace['actual_outputs'] = context.actual_outputs.model_dump(mode='json')
|
|
92
|
+
WorkflowTrace.model_validate(trace)
|
|
93
|
+
normalized = from_workflow(trace)
|
|
94
|
+
except (ValueError, TypeError, OSError) as error:
|
|
95
|
+
report.trace_normalization_error = True
|
|
96
|
+
report.trace_lossless = False
|
|
97
|
+
report.validation_errors.append(str(error))
|
|
98
|
+
return NormalizationResult(report=report, fallback_required=True, parse_error=str(error))
|
|
99
|
+
report.raw_step_count = report.normalized_step_count = len(atif['steps'])
|
|
100
|
+
report.trace_schema_valid = report.trace_normalized = True
|
|
101
|
+
report.normalized_trace_ref = 'agent_trace.normalized.json'
|
|
102
|
+
report.canonical_trace_ref = 'agent_trace.json'
|
|
103
|
+
return NormalizationResult(normalized_trace=normalized, report=report)
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
from __future__ import annotations
|
|
2
|
+
|
|
3
|
+
TRACE_SCHEMA_GUIDANCE = """Write ./agent_trace.json as a complete Harbor ATIF v1.8 trajectory:
|
|
4
|
+
{
|
|
5
|
+
"schema_version": "ATIF-v1.8",
|
|
6
|
+
"agent": {"name": "your-agent", "version": "your-agent-version"},
|
|
7
|
+
"steps": [
|
|
8
|
+
{"step_id": 1, "source": "user", "message": "Task instruction"},
|
|
9
|
+
{"step_id": 2, "source": "agent", "message": "Read task input",
|
|
10
|
+
"tool_calls": [{"tool_call_id": "call_2", "function_name": "read_file", "arguments": {"path": "input/task.md"}}],
|
|
11
|
+
"observation": {"results": [{"source_call_id": "call_2", "content": "Observed result"}]}}
|
|
12
|
+
]
|
|
13
|
+
}
|
|
14
|
+
Use sequential integer step_id values starting at 1; source is system, user, or agent.
|
|
15
|
+
Record each meaningful interaction faithfully. Tool results belong in observation.results,
|
|
16
|
+
with source_call_id matching a tool_call_id on the same step. Preserve parallel calls.
|
|
17
|
+
Record available timestamps, model_name, metrics, final_metrics and subagent_trajectories.
|
|
18
|
+
Embedded subagents need unique trajectory_id values. References use trajectory_id or
|
|
19
|
+
trajectory_path. Image/audio message parts use type and source {media_type, path};
|
|
20
|
+
audio may include duration_sec. Never embed media bytes or invent results or metrics.
|
|
21
|
+
Put project-specific task IDs, attempt IDs, artifacts, causal links, surface fields,
|
|
22
|
+
and outcome evidence in extra.agent_apprenticeship at the root or step level.
|
|
23
|
+
Do not add non-ATIF top-level fields. Do not include hidden chain-of-thought or secrets.
|
|
24
|
+
Read inputs under ./input/ and write deliverables under ./artifacts/.
|
|
25
|
+
Also write ./actual_outputs.json with task_id, attempt_id, attempt_kind, status,
|
|
26
|
+
output_summary, primary_output_ref, deliverable_refs, artifact_refs and files_created.
|
|
27
|
+
Use package-relative artifact references. Evaluation is performed separately.
|
|
28
|
+
"""
|
|
29
|
+
|
|
30
|
+
MASKING_GUIDANCE = """Sensitive info masking is standard for this run. Do not expose API keys, private tokens, credentials, or secrets in traces, logs, or artifacts. Preserve useful operational context while masking super-sensitive values."""
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
def build_worker_prompt(
|
|
34
|
+
task_instruction: str,
|
|
35
|
+
rubric_md: str = "",
|
|
36
|
+
attempt_kind: str = "baseline",
|
|
37
|
+
input_files: list[str] | None = None,
|
|
38
|
+
deliverables: list[str] | None = None,
|
|
39
|
+
sensitive_info_masking: str = "standard",
|
|
40
|
+
workspace_path: str | None = None,
|
|
41
|
+
) -> str:
|
|
42
|
+
input_files = input_files or []
|
|
43
|
+
deliverables = deliverables or []
|
|
44
|
+
input_section = "\n".join(f"- ./input/{name}" for name in input_files) or "- No explicit input files declared."
|
|
45
|
+
deliverable_section = "\n".join(f"- ./artifacts/{name}" for name in deliverables) or "- Write required task deliverables under ./artifacts/."
|
|
46
|
+
workspace_section = f"\nCurrent task workspace: {workspace_path}\n" if workspace_path else ""
|
|
47
|
+
masking_section = f"\n\n## Sensitive info masking\n{MASKING_GUIDANCE}" if sensitive_info_masking == "standard" else ""
|
|
48
|
+
return f"# Agent Apprenticeship Workflow Task\n\nAttempt: {attempt_kind}\n{workspace_section}\n{TRACE_SCHEMA_GUIDANCE}{masking_section}\n\n## Input files\n{input_section}\n\n## Required output contract\nYou must write these files in the current task workspace before finishing:\n- ./agent_trace.json\n- ./actual_outputs.json\n- ./artifacts/\n{deliverable_section}\n\nBefore finishing, verify ./agent_trace.json, ./actual_outputs.json, and ./artifacts/ exist. Do not only answer in chat/stdout. Do not say the task is complete unless those files exist.\n\n## Task\n{task_instruction}\n\n## Worker-visible evaluation rubric\n{rubric_md}\n\nFinal instruction: write the required files in this workspace, verify they exist, then stop.\n"
|
|
@@ -11,7 +11,7 @@ def _current_evaluator_feedback_text(grader: GraderResult | None, evaluator: Eva
|
|
|
11
11
|
return 'Evaluator feedback omitted because it referenced stale artifact-contract state.'
|
|
12
12
|
return text
|
|
13
13
|
|
|
14
|
-
def process_supervision_from_trace(trace:
|
|
14
|
+
def process_supervision_from_trace(trace: WorkflowTrace, grader: GraderResult | None=None, verifier: VerifierResult | None=None, evaluator: EvaluatorFeedback | None=None) -> list[ProcessSupervisionExample]:
|
|
15
15
|
final=grader.score if grader else None
|
|
16
16
|
evaluator_text=_current_evaluator_feedback_text(grader, evaluator)
|
|
17
17
|
out=[]
|
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
from __future__ import annotations
|
|
2
2
|
import json
|
|
3
3
|
from pathlib import Path
|
|
4
|
-
from .schemas import
|
|
5
|
-
from .io import read_jsonl
|
|
4
|
+
from .schemas import WorkflowTrace
|
|
5
|
+
from .io import read_workflow_jsonl as read_jsonl
|
|
6
6
|
from .env import contains_secret
|
|
7
7
|
from .public_sanitizer import has_prompt_leak
|
|
8
8
|
from .artifact_resolver import artifact_ref_resolves, is_artifact_evidence_ref, normalize_artifact_ref
|
|
@@ -64,7 +64,7 @@ def validate_release(root: Path) -> dict[str, object]:
|
|
|
64
64
|
counters['score_source_counts']=sc; counters['hillclimb_result_count']=len(read_jsonl(root/'hillclimb_results.jsonl')); counters['process_supervision_count']=len(read_jsonl(root/'process_supervision.jsonl')); counters['reward_modeling_count']=len(read_jsonl(root/'reward_modeling.jsonl')); counters['revision_preference_pair_count']=len(read_jsonl(root/'revision_preference_pairs.jsonl'))
|
|
65
65
|
traces=read_jsonl(root/'agent_traces.jsonl'); counters['trace_count']=len(traces)
|
|
66
66
|
for t in traces:
|
|
67
|
-
try:
|
|
67
|
+
try: WorkflowTrace.model_validate(t); counters['trace_valid_count']+=1
|
|
68
68
|
except Exception: counters['trace_invalid_count']+=1; counters['release_valid']=False
|
|
69
69
|
steps=t.get('steps') or []; counters['trace_step_count_total'] += len(steps)
|
|
70
70
|
counters['trace_steps_with_tool_count'] += sum(1 for s in steps if s.get('tool'))
|
|
@@ -1,9 +1,10 @@
|
|
|
1
1
|
from __future__ import annotations
|
|
2
|
+
from .atif import from_workflow
|
|
2
3
|
import json
|
|
3
4
|
from pathlib import Path
|
|
4
|
-
from .schemas import VerifierResult, GraderResult, ActualOutputs,
|
|
5
|
+
from .schemas import VerifierResult, GraderResult, ActualOutputs, WorkflowTrace
|
|
5
6
|
from .config import get_settings
|
|
6
|
-
from .io import read_json
|
|
7
|
+
from .io import read_workflow_json as read_json
|
|
7
8
|
from .openai_structured import get_model_provider_status, run_structured_role
|
|
8
9
|
from .public_sanitizer import sha256_text
|
|
9
10
|
from .artifact_previews import build_artifact_previews
|
|
@@ -30,13 +31,13 @@ def deterministic_verify(grader: GraderResult, outputs: ActualOutputs) -> Verifi
|
|
|
30
31
|
issues.append('hidden reference leakage')
|
|
31
32
|
return VerifierResult(verifier_result_id=f'verifier_{outputs.attempt_id}', task_id=outputs.task_id, attempt_id=outputs.attempt_id, attempt_kind=outputs.attempt_kind, grader_result_id=grader.grader_result_id, verification_status='not_run', artifact_contract_ok=artifact_ok, evidence_grounding_ok=bool(grader.evidence_refs), score_consistency_ok=score_ok, hidden_reference_leaked=grader.hidden_reference_leaked, issues=issues, confidence=0.6, verifier_notes='Structural guardrail checked artifact refs, score bounds, and leakage flags only; Mentor verification did not run.', semantic_evidence_grounding_ok=None, unsupported_claims=[], leakage_check_ok=not grader.hidden_reference_leaked, model=None, provider=None, metadata_json={'score_source': grader.score_source, 'structural_guardrail': True, 'role_status': 'mentor_verifier_unavailable'})
|
|
32
33
|
|
|
33
|
-
def _verifier_prompt(grader: GraderResult, outputs: ActualOutputs, trace:
|
|
34
|
+
def _verifier_prompt(grader: GraderResult, outputs: ActualOutputs, trace: WorkflowTrace | None, artifact_preview_bundle: dict | None=None) -> str:
|
|
34
35
|
return """Return only valid JSON matching VerifierResult. Do not include markdown.
|
|
35
36
|
Check whether the grader evidence is grounded in actual_outputs.json, artifact refs, trace evidence, and especially the artifact content previews. Check score consistency, unsupported claims, and hidden-reference leakage. Do not treat Apprentice Agent self-assessment or self evaluation as outcome evidence. Do not use deterministic fallback judgement; structural guardrails are package-integrity checks only and are not verifier judgments. Return exactly the VerifierResult schema fields. Do not add top-level fields not in schema. If you want to report extra checks such as reproducible, within_time_limit, uses_only_allowed_apis, passed_checks, or within_memory_limit, put them inside metadata_json.extra_model_fields.
|
|
36
37
|
GraderResult JSON:
|
|
37
|
-
""" + json.dumps(grader.model_dump(), sort_keys=True) + "\nActualOutputs JSON:\n" + json.dumps(outputs.model_dump(), sort_keys=True) + "\nArtifact content previews JSON:\n" + json.dumps(artifact_preview_bundle or {}, sort_keys=True)[:16000] + "\nTrace summary JSON:\n" + json.dumps((trace.model_dump() if trace else {}), sort_keys=True)
|
|
38
|
+
""" + json.dumps(grader.model_dump(), sort_keys=True) + "\nActualOutputs JSON:\n" + json.dumps(outputs.model_dump(), sort_keys=True) + "\nArtifact content previews JSON:\n" + json.dumps(artifact_preview_bundle or {}, sort_keys=True)[:16000] + "\nTrace summary JSON:\n" + json.dumps((from_workflow(trace.model_dump()) if trace else {}), sort_keys=True)
|
|
38
39
|
|
|
39
|
-
def verify_attempt(grader: GraderResult, outputs: ActualOutputs, trace:
|
|
40
|
+
def verify_attempt(grader: GraderResult, outputs: ActualOutputs, trace: WorkflowTrace | None=None, role_root: Path | None=None, package_root: Path | None=None) -> VerifierResult:
|
|
40
41
|
settings=get_settings(); pre=deterministic_verify(grader, outputs); role_root=role_root or Path('outputs/roles')
|
|
41
42
|
artifact_preview_bundle=build_artifact_previews(package_root, grader.evidence_refs or outputs.deliverable_refs or outputs.artifact_refs or outputs.files_created)
|
|
42
43
|
if _model_verifier_active(settings) and settings.llm_verifier_enabled:
|
|
@@ -1,113 +0,0 @@
|
|
|
1
|
-
# Bioactive Evidence Mapping Extension Schema
|
|
2
|
-
|
|
3
|
-
This optional Agent Apprenticeship schema describes evidence-mapping rows for specialized research and decision-support datasets. It is a general Experience Compilation extension and does not change the core product into a wellness-only tool.
|
|
4
|
-
|
|
5
|
-
These rows are for provenance, uncertainty tracking, citation grounding, mechanism mapping, and verifier training. They are not medical advice, diagnosis, treatment guidance, cure claims, or sensitive personal records.
|
|
6
|
-
|
|
7
|
-
## Row Families
|
|
8
|
-
|
|
9
|
-
### evidence_mapping_sft_rows
|
|
10
|
-
|
|
11
|
-
LoRA-ready instruction-tuning view for extracting grounded ingredient, compound, product, evidence, mechanism, safety/caution, and uncertainty fields from provided local source material.
|
|
12
|
-
|
|
13
|
-
Required fields:
|
|
14
|
-
|
|
15
|
-
- `row_id`
|
|
16
|
-
- `schema_version`
|
|
17
|
-
- `data_type`
|
|
18
|
-
- `source_task_id`
|
|
19
|
-
- `domain`
|
|
20
|
-
- `subdomain`
|
|
21
|
-
- `workflow_family`
|
|
22
|
-
- `instruction`
|
|
23
|
-
- `input_context`
|
|
24
|
-
- `target_output`
|
|
25
|
-
- `source_text_or_artifact_ref`
|
|
26
|
-
- `source_evidence_ref_ids`
|
|
27
|
-
- `quality_score`
|
|
28
|
-
- `confidence`
|
|
29
|
-
|
|
30
|
-
### ingredient_claim_verifier_rows
|
|
31
|
-
|
|
32
|
-
Verifier-training view for checking whether a claim is supported by visible source evidence.
|
|
33
|
-
|
|
34
|
-
Required fields:
|
|
35
|
-
|
|
36
|
-
- `row_id`
|
|
37
|
-
- `claim`
|
|
38
|
-
- `ingredient_or_compound_entity`
|
|
39
|
-
- `visible_evidence`
|
|
40
|
-
- `expected_contract`
|
|
41
|
-
- `verifier_decision`
|
|
42
|
-
- `failure_reason`
|
|
43
|
-
- `confidence`
|
|
44
|
-
- `source_evidence_ref_ids`
|
|
45
|
-
|
|
46
|
-
### product_ingredient_extraction_rows
|
|
47
|
-
|
|
48
|
-
Extraction view for mapping product labels or specification tables into structured ingredient/product entities.
|
|
49
|
-
|
|
50
|
-
Required fields:
|
|
51
|
-
|
|
52
|
-
- `row_id`
|
|
53
|
-
- `product_entity`
|
|
54
|
-
- `ingredient_or_compound_entities`
|
|
55
|
-
- `source_text_or_artifact_ref`
|
|
56
|
-
- `extraction_contract`
|
|
57
|
-
- `citation_or_provenance_refs`
|
|
58
|
-
- `source_evidence_ref_ids`
|
|
59
|
-
- `confidence`
|
|
60
|
-
|
|
61
|
-
### mechanism_mapping_rows
|
|
62
|
-
|
|
63
|
-
Mechanism evidence view for mapping a claim or source passage into a mechanism category with uncertainty.
|
|
64
|
-
|
|
65
|
-
Required fields:
|
|
66
|
-
|
|
67
|
-
- `row_id`
|
|
68
|
-
- `ingredient_or_compound_entity`
|
|
69
|
-
- `mechanism_category`
|
|
70
|
-
- `evidence_summary`
|
|
71
|
-
- `uncertainty`
|
|
72
|
-
- `citation_or_provenance_refs`
|
|
73
|
-
- `source_evidence_ref_ids`
|
|
74
|
-
- `confidence`
|
|
75
|
-
|
|
76
|
-
### safety_caution_rows
|
|
77
|
-
|
|
78
|
-
Safety/caution evidence view for neutral flags and context constraints. Use abstract personal-context constraints only, never sensitive identity data.
|
|
79
|
-
|
|
80
|
-
Required fields:
|
|
81
|
-
|
|
82
|
-
- `row_id`
|
|
83
|
-
- `ingredient_or_compound_entity`
|
|
84
|
-
- `safety_caution_category`
|
|
85
|
-
- `personal_context_constraints`
|
|
86
|
-
- `evidence_summary`
|
|
87
|
-
- `uncertainty`
|
|
88
|
-
- `citation_or_provenance_refs`
|
|
89
|
-
- `source_evidence_ref_ids`
|
|
90
|
-
- `confidence`
|
|
91
|
-
|
|
92
|
-
### citation_grounding_rows
|
|
93
|
-
|
|
94
|
-
Grounding view for checking whether generated summaries cite visible evidence correctly.
|
|
95
|
-
|
|
96
|
-
Required fields:
|
|
97
|
-
|
|
98
|
-
- `row_id`
|
|
99
|
-
- `generated_claim_or_summary`
|
|
100
|
-
- `citation_or_provenance_refs`
|
|
101
|
-
- `visible_evidence`
|
|
102
|
-
- `verifier_decision`
|
|
103
|
-
- `failure_reason`
|
|
104
|
-
- `source_evidence_ref_ids`
|
|
105
|
-
- `confidence`
|
|
106
|
-
|
|
107
|
-
## Safety Rules
|
|
108
|
-
|
|
109
|
-
- Use only provided local source material or package evidence refs.
|
|
110
|
-
- Keep personal context as abstract constraints such as `pregnancy_context`, `medication_interaction_context`, or `allergy_context`; do not store identity or sensitive user details.
|
|
111
|
-
- Record uncertainty and provenance for every extracted claim.
|
|
112
|
-
- Do not infer treatment, cure, diagnosis, or patient-specific recommendations.
|
|
113
|
-
- Do not mark unsupported claims as accepted verifier rows.
|