@lazyingart/agintiflow 0.20.144 → 0.20.150

package/README.md

@@ -240,7 +240,7 @@ The website keeps the visual walkthrough in a carousel so this README can stay f
 | SCS mode | Optional Student-Committee-Supervisor quality gate for complicated or risky tasks. |
 | AAPS adapter | Optional `@lazyingart/aaps` integration for `.aaps` workflow init, validate, parse, compile, dry-run, and run commands. |
 | Image generation | Optional GRS AI and Venice image tools with saved manifests and canvas artifact previews. |
-| Skill library | Built-in Markdown skills for code, websites, Android/iOS, Python, Rust, Java, LaTeX, writing, reviews, GitHub, AAPS, and more. |
+| Skill library | Built-in Markdown skills for code, websites, Android/iOS, Python, Rust, Java, LaTeX, writing, reviews, source ingestion/OCR, structured JSON, autonomous artifact pipelines, GitHub, AAPS, and more. |
 | Skill Mesh | Optional strict skill recording/sharing for reviewed reusable skill packs. If unused, AgInTiFlow runs normally without background sharing. |
 | Multilingual UI | CLI and docs language support for English, Japanese, Simplified/Traditional Chinese, Korean, French, Spanish, Arabic, Vietnamese, German, and Russian. |
 

package/docs/autonomous-artifact-pipelines.md

@@ -0,0 +1,36 @@
+# Autonomous Artifact Pipelines
+
+AgInTiFlow can run a project from raw inputs to final artifacts when the work is expressed as a local, resumable pipeline. This pattern is for jobs such as source conversion, book generation, report building, dataset annotation, media preparation, or any workflow where partial outputs are valuable and failure recovery matters.
+
+## Contract
+
+Each pipeline should create these project-local files or equivalents:
+
+- Source manifest: raw files, hashes, roles, languages, extraction method, and caveats.
+- Derived inputs: Markdown, text, tables, images, or structured bundles created from the raw files.
+- Task manifest: stable chunk IDs, source locations, dependencies, prompt/schema version, and output paths.
+- Schema and validator: the exact artifact shape plus semantic checks that define a promotable output.
+- Runners: writer, reviewer, repairer, monitor, merge, compile/export, and status commands.
+- Completion report: counts, first missing ID, failed/quarantined items, latest previews, final artifact paths, and resume commands.
+
+The target repository owns its schemas, prompts, chunk policy, and rendering code. AgInTiFlow owns the behavior: inspect, create missing scripts, run observable sessions, preserve valid work, validate, repair, compile, and report evidence.
+
+## Roles
+
+The writer creates candidate artifacts. It should never be the only quality gate.
+
+The validator promotes candidates only after schema and project-specific checks pass.
+
+The reviewer inspects valid-looking artifacts for missing source units, source drift, repeated filler, malformed annotations, suspicious all-one-style output, and other known quality failures. It writes candidate fixes or failed-only repair requests.
+
+The repairer runs independently of the writer. It can wake from status files, handle failed or quarantined chunks, retry with exact validator errors, reduce chunk size, or escalate to a stronger model when the project allows it.
+
+The monitor is gentle. It waits through healthy progress and provider limits, restarts only on hard evidence of stall or crash, and records each decision.
+
+## Concurrency
+
+Parallelism is optional. When used, each worker needs deterministic shard ownership, separate logs, atomic writes, and no direct compile responsibility. Merge, promotion, compilation, publishing, and commits should be serialized unless the project already has a safe coordinator.
+
+## Completion
+
+A run is complete only when the final artifact was built from the current manifest and the status report shows full coverage or intentional quarantine. A successful tmux pane, a page count, or a single preview file is not enough.

package/docs/self-healing-pipelines.md

@@ -0,0 +1,23 @@
+# Self-Healing Pipelines
+
+AgInTiFlow treats long-running writers, reviewers, repairers, monitors, queues, ETL jobs, and batch generators as resumable pipelines rather than one-shot commands.
+
+Use the `pipeline` task profile or rely on automatic skill selection when a prompt mentions a stalled writer, monitor, reviewer, queue worker, failed chunks, retry passes, stale claims, or tmux supervision.
+
+## Operating Model
+
+The agent should first read status files, manifests, logs, and tmux panes. It should compare progress across observations before calling a job stalled unless a hard error is visible. Healthy provider waits, rate limits, and long compile steps should usually be left alone.
+
+When repair is justified, the agent should patch project-owned scripts or prompts in small reversible changes. Preferred repairs include failed-only retry modes, bounded retry passes, atomic writes, stale claim cleanup, idempotent compile commands, clear status JSON, heartbeat files, and durable logs.
+
+Concurrency is a tool, not a product stance. AgInTiFlow should choose sequential, parallel, async, or review-gated operation from the user's request and the local project evidence. It should not force a sharded design into projects that do not need it.
+
+Review and repair are separate from writing. A reviewer should detect missing source units, repeated filler, malformed structured data, source drift, and known quality failures, then produce candidate repairs or failed-only requests. A repairer should be able to run independently of the writer, wake from status files, run bounded passes, and exit without blocking healthy progress.
+
+## Boundaries
+
+AgInTiFlow should not embed project-specific schemas in its core. A book writer, data pipeline, or build system owns its own validators and artifact layout. AgInTiFlow provides the reusable behavior: diagnose, preserve valid work, patch the local workflow, verify, build checkpoint artifacts, restart only affected sessions, and report exact resume commands.
+
+## Verification
+
+After a repair, the agent should run syntax checks for changed scripts, perform a dry-run or bounded batch when safe, inspect counters and first-missing IDs, and keep unrelated tmux sessions running. If the same symptom repeats, the agent should improve the project workflow or a reusable AgInTiFlow skill instead of repeatedly sending manual nudges.

package/docs/skills-and-tools.md

@@ -12,7 +12,7 @@ AgInTiFlow separates **skills** from **tools** so the agent can stay general whi
 
 ## Built-In Skills
 
-The package ships built-in skills for code engineering, website/app building, LaTeX manuscripts, books, Microsoft Word documents, image generation, GitHub maintenance, system maintenance, tmux session control, Android, R/Stan, Python, C/C++, shell scripting, AAPS, novel writing, and supervision/student-agent training.
+The package ships built-in skills for code engineering, website/app building, LaTeX manuscripts, books, Microsoft Word documents, image generation, GitHub maintenance, system maintenance, source ingestion/OCR, structured JSON, autonomous artifact pipelines, tmux session control, Android, R/Stan, Python, C/C++, shell scripting, AAPS, novel writing, and supervision/student-agent training.
 
 List them from a project:
 
@@ -71,3 +71,12 @@ For substantial writing tasks, prefer:
 - The main agent for all non-writing work around that draft: file names, workspace edits, citations, Markdown/LaTeX/Final Draft formatting, PDF compilation, canvas publishing, and verification.
 
 The writer receives only writing context: brief, canon, style guide, prior draft, target, audience, constraints, length, and downstream format intent. It should not receive shell/file/browser policy or agent-runtime details.
+
+For schema-bound structured data, prefer:
+
+- `json_specialist` for one isolated extraction, annotation, conversion, or validation request.
+- `json_specialist_batch` for independent chunks that can be requested in parallel without shared writes.
+
+The JSON specialist receives only the task, focused instructions, minimal context, input, and JSON Schema. It tries provider-native structured output (`json_schema` or JSON object mode) when available, then falls back to prompt-and-validate parsing.
+
+For raw-input-to-final-output work, prefer the autonomous artifact pipeline pattern. The target project should own its source manifest, chunk manifest, schemas, validators, runner scripts, reviewer/repairer logic, and compiler/exporter. AgInTiFlow should create or patch those project-local pieces, run them in observable sessions, and verify checkpoint artifacts before declaring completion.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lazyingart/agintiflow",
-  "version": "0.20.144",
+  "version": "0.20.150",
   "type": "module",
   "description": "AgInTiFlow is a project-aware agent workspace for hybrid wet-dry R&D, hardware-aware intelligence, software automation, and industrial workflows.",
   "license": "Apache-2.0",

package/scripts/smoke-capabilities.js

@@ -129,6 +129,10 @@ try {
     capabilities.tools?.taskProfiles?.some((profile) => profile.id === "supervision"),
     "capabilities did not report supervision task profile"
   );
+  assert(
+    capabilities.tools?.taskProfiles?.some((profile) => profile.id === "pipeline"),
+    "capabilities did not report pipeline task profile"
+  );
   for (const profileId of ["docs", "data", "qa", "database", "devops", "security", "slides", "education", "java", "ios", "go", "rust", "dotnet", "php", "ruby"]) {
     assert(
       capabilities.tools?.taskProfiles?.some((profile) => profile.id === profileId),
@@ -138,6 +142,7 @@ try {
   const qaProfile = listTaskProfiles().find((profile) => profile.id === "qa");
   assert(qaProfile, "QA profile is missing");
   assert(defaultMaxStepsForProfile("qa") >= 40, "QA profile step budget is too low for verification and cleanup");
+  assert(defaultMaxStepsForProfile("pipeline") >= 44, "pipeline profile step budget is too low for repair/verify/resume loops");
   assert(!/misleading failing test/i.test(qaProfile.prompt), "QA profile still encourages misleading test fixtures");
   assert(/do not stage fake bugs/i.test(qaProfile.prompt), "QA profile does not discourage fake staged failures");
   assert(
@@ -168,6 +173,10 @@ try {
     capabilities.tools?.skills?.some((skill) => skill.id === "supervision-student"),
     "capabilities did not report built-in supervision skill"
   );
+  assert(
+    capabilities.tools?.skills?.some((skill) => skill.id === "self-healing-pipeline"),
+    "capabilities did not report built-in self-healing pipeline skill"
+  );
   for (const skillId of ["data-analysis", "docs-knowledge", "qa-testing", "database", "devops-deployment", "security-review", "presentation-slides", "writing-editing", "java-jvm", "ios-swift", "go", "rust", "dotnet-csharp", "php", "ruby"]) {
     assert(
       capabilities.tools?.skills?.some((skill) => skill.id === skillId),

package/scripts/smoke-coding-tools.js

@@ -25,6 +25,7 @@ import { createPlan } from "../src/model-client.js";
 import { selectModelRoute } from "../src/model-routing.js";
 import { listParallelScouts, runParallelScouts, shouldRunParallelScouts } from "../src/parallel-scouts.js";
 import { buildFailedCommandAdvice, buildPermissionAdvice } from "../src/permission-advice.js";
+import { runJsonSpecialist } from "../src/json-specialist.js";
 import { SessionStore } from "../src/session-store.js";
 import { getTaskProfile } from "../src/task-profiles.js";
 import { searchWeb } from "../src/web-search.js";
@@ -613,6 +614,30 @@ try {
     writingRun.events.some((event) => event.type === "tool.completed" && event.data?.toolName === "writing_specialist"),
     "mock agent did not route writing work through writing_specialist"
   );
+  const jsonResult = await runJsonSpecialist(
+    {
+      task: "Return a strict JSON status object.",
+      schema: {
+        type: "object",
+        properties: {
+          summary: { type: "string" },
+          complete: { type: "boolean" },
+        },
+        required: ["summary", "complete"],
+        additionalProperties: false,
+      },
+      inputText: "structured JSON smoke",
+      provider: "mock",
+    },
+    { provider: "mock", model: "mock-agent" },
+    new SessionStore(runtimeDir, "json-specialist-smoke", { projectRoot: workspace, commandCwd: workspace })
+  );
+  assert(jsonResult.ok && jsonResult.result?.complete === true, "json_specialist mock result did not satisfy schema");
+  const jsonRun = await runMock("Extract a valid JSON object with schema from this text.", "coding-json-specialist");
+  assert(
+    jsonRun.events.some((event) => event.type === "tool.completed" && event.data?.toolName === "json_specialist"),
+    "mock agent did not route structured JSON work through json_specialist"
+  );
 
   await fs.mkdir(path.join(workspace, "src"), { recursive: true });
   await fs.mkdir(path.join(workspace, "test"), { recursive: true });
@@ -1104,6 +1129,8 @@ try {
           "deepseek_pro_writing_route",
           "writing_specialist_mock",
           "writing_specialist_mock_routing",
+          "json_specialist_mock",
+          "json_specialist_mock_routing",
           "runtime_time_context",
           "large_profile_pro_route",
           "auto_system_pro_route",

package/scripts/smoke-skills.js

@@ -21,6 +21,7 @@ const ids = new Set(skills.map((skill) => skill.id));
 assert(skills.length >= 27, "expected built-in skills to load");
 for (const required of [
   "aaps",
+  "autonomous-artifact-pipeline",
   "code",
   "code-review",
   "data-analysis",
@@ -41,6 +42,9 @@ for (const required of [
   "rust",
   "php",
   "security-review",
+  "self-healing-pipeline",
+  "source-ingestion",
+  "structured-json",
   "system-maintenance",
   "supervision-student",
   "tmux-session",
@@ -57,6 +61,13 @@ assert(selectedIds("edit a Microsoft Word docx and preserve the original").inclu
 assert(selectedIds("generate a logo image with grsai nanobanana").includes("image-generation"), "image prompt did not select image-generation");
 assert(selectedIds("git status commit push with gh").includes("github-maintenance"), "git prompt did not select github-maintenance");
 assert(selectedIds("monitor a long running tmux session").includes("tmux-session"), "tmux prompt did not select tmux-session");
+assert(selectedIds("repair a stuck writer monitor pipeline and retry failed chunks").includes("self-healing-pipeline"), "stuck pipeline prompt did not select self-healing-pipeline");
+assert(
+  selectedIds("from raw pdf epub sources create markdown chunks json and compile final pdf in tmux with monitor and auto repair").includes(
+    "autonomous-artifact-pipeline"
+  ),
+  "raw-source artifact pipeline prompt did not select autonomous-artifact-pipeline"
+);
 assert(selectedIds("create an .aaps example for @lazyingart/aaps").includes("aaps"), "AAPS prompt did not select aaps");
 assert(selectedIds("debug a C++ CMake build").includes("c-cpp"), "C++ prompt did not select c-cpp");
 assert(selectedIds("set up Stan and CmdStanR reproducibly").includes("r-stan"), "Stan prompt did not select r-stan");
@@ -68,6 +79,8 @@ assert(selectedIds("debug a C# dotnet web API").includes("dotnet-csharp"), ".NET
 assert(selectedIds("fix a PHP Laravel composer project").includes("php"), "PHP prompt did not select php");
 assert(selectedIds("repair a Ruby Rails app with RSpec").includes("ruby"), "Ruby prompt did not select ruby");
 assert(selectedIds("clean a CSV dataset and make plots").includes("data-analysis"), "data prompt did not select data-analysis");
+assert(selectedIds("convert scanned PDF EPUB and image sources to markdown with OCR").includes("source-ingestion"), "source ingestion prompt did not select source-ingestion");
+assert(selectedIds("use json schema to fetch valid structured json for each chunk in parallel").includes("structured-json"), "structured JSON prompt did not select structured-json");
 assert(selectedIds("write README API docs and a tutorial").includes("docs-knowledge"), "docs prompt did not select docs-knowledge");
 assert(selectedIds("fix failing tests and add regression coverage").includes("qa-testing"), "QA prompt did not select qa-testing");
 const qaSkill = skills.find((skill) => skill.id === "qa-testing");

package/skills/autonomous-artifact-pipeline/SKILL.md

@@ -0,0 +1,69 @@
+---
+id: autonomous-artifact-pipeline
+label: Autonomous Artifact Pipeline
+description: Turn raw inputs into validated final artifacts through a resumable, observable pipeline with ingestion, chunking, writers, reviewers, repairers, monitors, and checkpoint builds.
+triggers:
+  - raw source to final output
+  - autonomous pipeline
+  - artifact pipeline
+  - final artifact
+  - prepare chunks
+  - meta tasks
+  - writer reviewer repairer
+  - tmux monitor
+  - resumable batch
+  - compile output
+  - auto repair
+tools:
+  - inspect_project
+  - read_file
+  - write_file
+  - apply_patch
+  - run_command
+  - tmux_start_session
+  - tmux_capture_pane
+  - tmux_send_keys
+  - send_to_canvas
+---
+# Autonomous Artifact Pipeline
+
+Use this skill when the user wants AgInTiFlow to start from raw materials and keep working until durable final artifacts exist, such as books, reports, datasets, PDFs, apps, slides, media bundles, or generated JSON corpora.
+
+## Pipeline Contract
+
+Before launching long work, create or identify a project-local contract:
+
+1. Source manifest: raw paths, hashes, source roles, language/type, extraction method, and caveats.
+2. Derived inputs: Markdown, text, images, tables, or structured bundles produced from the raw sources.
+3. Task manifest: stable chunk IDs, source location, dependency order, prompt/schema version, and output paths.
+4. Artifact schema: JSON Schema or other validator-owned shape for each generated unit.
+5. Runners: resumable writer, reviewer, repairer, monitor, merge, compile/export, and status commands.
+6. Completion evidence: counters, first missing item, failed IDs, current previews, final artifact paths, and resume commands.
+
+Project-specific schemas, prompts, layouts, and compilers belong in the target repository. AgInTiFlow provides the orchestration pattern and should generate or patch local scripts when they are missing.
+
+## Execution Pattern
+
+- Inspect the repository and instructions first. Preserve raw inputs and do not overwrite reviewed outputs.
+- Convert raw files into durable intermediate inputs before asking a model to generate downstream artifacts.
+- Split work into deterministic chunks that survive reruns. If chunk policy changes, map old outputs by stable source IDs instead of restarting from zero.
+- Use isolated structured-data calls for repetitive JSON units. Keep prompts focused on the chunk, schema, source references, and validation errors.
+- Run writers in tmux or another observable background process. Each worker must have disjoint claims, atomic output writes, and shard-local logs.
+- Keep review and repair asynchronous but safe. Reviewers may produce candidate fixes while writers continue; only validators or merge scripts promote candidates.
+- Compile or export checkpoint previews after successful merge batches and always at final completion.
+- Commit reusable scripts, manifests, validators, templates, and stable checkpoints when the project expects git tracking.
+
+## Autorepair Behavior
+
+A robust pipeline has an independent repair path that is not blocked by the main writer:
+
+- Heartbeats record active worker, current chunk, last success, last failure, and provider wait state.
+- Provider/rate-limit failures wait with backoff and retry at long intervals.
+- Schema/parse failures are repaired with the exact validator error and the smallest useful input.
+- Semantic/source-drift failures are retried with smaller chunks or stronger source references.
+- Repeated failures are quarantined with reasons, then handled by a bounded failed-only repair pass.
+- Monitor intervention is gentle: observe healthy progress, restart only on hard error, stale claim, repeated no-progress window, or missing child process.
+
+## Done Criteria
+
+Do not call the task complete until the final artifact was built from the current manifest and the status report shows complete or intentionally quarantined coverage. A partial PDF, stale page count, or successful worker log is not enough.

package/skills/bilingual-interlinear-book/SKILL.md

@@ -27,13 +27,14 @@ Use this skill when the task asks for a paired-language book, ruby/furigana/piny
 
 1. Inspect repository instructions, existing scripts, book plans, and ignored paths before editing.
 2. Keep original PDFs/EPUBs in source folders and do not commit large source media unless the repository explicitly tracks them.
-3. Convert source books to durable Markdown first. Keep raw and cleaned Markdown separate when OCR or EPUB extraction is noisy.
+3. Convert source books to durable Markdown first. Treat PDF, EPUB, image, JSON/wiki, and scanned sources as ingestion problems; create a source manifest with hashes, method, role, language, and caveats before generation. Keep raw and cleaned Markdown separate when OCR or EPUB extraction is noisy.
 4. Split cleaned Markdown into stable paragraph- or chapter-scoped chunks with `manifest.json` and `chunks.jsonl`. Use source paragraph IDs that survive reruns. If a paragraph is too large for reliable provider output, split it into ordered subchunks at sentence or clause boundaries while preserving the original source order and recording `split_from_chunk_id`, `split_part`, and `split_part_count`.
 5. When retuning chunk size or repairing split logic, merge any existing `split_from_chunk_id` groups back to the original paragraph text first, then split again. Do not repeatedly split already-split parts, and do not discard valid reviewed chunks unless validation proves they no longer match the manifest.
 6. Write resumable per-chunk JSON artifacts. Never overwrite a valid reviewed chunk unless a validator or prompt version requires regeneration; move stale chunks out of the compile path.
-7. Generate or repair annotations with a provider worker loop, not a monolithic prompt. Each chunk should validate independently before promotion. If JSON is malformed or validation fails, retry the chunk with the exact validator errors before marking it failed.
-8. Compile preview PDFs periodically and at the end. For paired-language books, compile both directions when renderers exist, plus color and blackwhite variants when color is supported.
-9. Run the writer and monitor in observable tmux sessions with status files, logs, retry/backoff for provider limits, and clear resume commands.
+7. Generate or repair annotations with a provider worker loop, not a monolithic prompt. Each chunk should validate independently before promotion. If JSON is malformed or validation fails, retry the chunk with the exact validator errors before marking it failed. Use the structured JSON workflow for repetitive chunk output.
+8. Keep writer, reviewer, repairer, monitor, merge, and compile roles explicit. Writers produce candidates; validators promote; reviewers check semantic quality and request fixes; repairers handle failed-only or quarantined chunks; monitors observe and resume gently.
+9. Compile preview PDFs periodically and at the end. For paired-language books, compile both directions when renderers exist, plus color and blackwhite variants when color is supported.
+10. Run long jobs in observable tmux sessions with status files, logs, retry/backoff for provider limits, and clear resume commands.
 
 ## JSON Quality Gates
 

package/skills/self-healing-pipeline/SKILL.md

@@ -0,0 +1,73 @@
+---
+id: self-healing-pipeline
+label: Self-Healing Pipeline
+description: Diagnose stalled long-running writer, reviewer, monitor, queue, ETL, build, and generation pipelines; patch project-owned scripts; verify; and resume without discarding valid work.
+triggers:
+  - self healing
+  - autorepair
+  - auto repair
+  - stuck pipeline
+  - stalled pipeline
+  - writer monitor
+  - reviewer monitor
+  - queue worker
+  - long running pipeline
+  - retry failed
+  - resume failed
+tools:
+  - inspect_project
+  - read_file
+  - search_files
+  - apply_patch
+  - run_command
+  - tmux_list_sessions
+  - tmux_capture_pane
+  - tmux_send_keys
+  - tmux_start_session
+---
+# Self-Healing Pipeline
+
+Use this skill when a project has a durable worker, writer, reviewer, monitor, queue, batch generator, ETL, build loop, or tmux job that must keep moving after malformed output, provider limits, crashes, bad chunks, stale locks, or compile failures.
+
+## Diagnose Before Repair
+
+1. Read project instructions, runner scripts, status files, manifests, logs, and current tmux panes.
+2. Classify the symptom as one of: healthy wait, provider/rate-limit wait, validation failure, deterministic data/schema failure, script crash, stale lock/claim, compile/render failure, missing dependency, or monitor failure.
+3. Compare progress counters across two observations before declaring a stall unless the logs show a hard error.
+4. Preserve valid outputs. Never delete reviewed artifacts, manifests, checkpoints, or source inputs unless validation proves they are stale and the project has a quarantine path.
+
+## Repair Pattern
+
+- Patch project-owned scripts, prompts, validators, or monitors only after the logs identify a repeatable failure.
+- Prefer small resumability upgrades: `--failed-only`, bounded retry passes, stale-claim cleanup, atomic writes, checkpoint status, idempotent compile commands, and clear resume commands.
+- Keep writer/reviewer/monitor responsibilities separate. The writer should produce and validate; the reviewer should repair quality; the monitor should observe, compile, restart, or queue the next bounded run.
+- Make monitors gentle: wait on healthy progress, restart only after explicit stop/stall/error evidence, and write a durable decision log.
+- If parallel workers exist, require disjoint output paths or claim files, atomic promotion, and merge-stage validation before compiling.
+- A reviewer is not just a promoter. It should inspect valid-looking outputs for source drift, missing units, repeated filler, malformed annotations, and known quality failures, then write candidate fixes or failed-only repair requests.
+- A companion repairer should be independent from the main writer process. It can stay dormant, wake from status files or monitor decisions, run bounded repair passes, and exit without blocking healthy writer progress.
+- If the same failure repeats after a local nudge, improve the runner, validator, prompt, or status model rather than relying on manual chat intervention.
+
+## Parallel And Async Options
+
+Parallel and async designs are optional implementation patterns, not a default preference. Use them when the user asks for concurrency, the existing project already has a parallel pipeline, or the evidence shows a sequential bottleneck that can be partitioned safely.
+
+- Sequential processing is often the safest default for small jobs, fragile prompts, scarce quota, or unclear ownership boundaries.
+- If using sharded writer/fetcher workers for independent JSON/data chunks, give each worker a deterministic shard, separate log file, and no compile responsibility.
+- An async reviewer/promoter loop can validate and promote completed candidate files while writers continue, but it must use atomic writes, locks, or merge directories so it cannot race with active writers.
+- Keep compilation, publishing, and git commits out of parallel workers unless the project already has a safe, serialized mechanism for those steps.
+- If a worker stalls on one bad chunk, it should mark the chunk failed and continue its shard. Failed-only repair passes should be bounded and observable.
+- When increasing concurrency, check provider quota/rate-limit behavior. If rate limits appear, reduce worker count or add backoff rather than letting every worker retry aggressively.
+
+## Verification And Resume
+
+After a repair:
+
+1. Run syntax checks for changed scripts.
+2. Run a dry-run or small bounded batch when safe.
+3. Verify status counters, first missing item, failed IDs, and output timestamps.
+4. Restart only the affected tmux session, not unrelated jobs.
+5. Compile or export a checkpoint artifact when the pipeline has a renderer or build command.
+6. Record the exact resume command, current status, logs inspected, remaining failed items, and any quarantined artifacts.
+7. Commit reusable script/profile/prompt fixes when the project expects git tracking.
+
+The goal is not to hide failures. The goal is to keep the pipeline observable, resumable, and able to recover from known classes of failure without overwriting good work.

package/skills/source-ingestion/SKILL.md

@@ -0,0 +1,63 @@
+---
+id: source-ingestion
+label: Source Ingestion And OCR
+description: Inspect mixed input files and convert PDFs, EPUBs, images, scans, web/JSON sources, archives, or unknown documents into useful text, Markdown, manifests, or reviewable artifacts.
+triggers:
+  - source
+  - sources
+  - input file
+  - convert to markdown
+  - extract text
+  - pdf
+  - ocr
+  - scanned
+  - epub
+  - image text
+  - recognition
+  - markdown
+tools:
+  - inspect_project
+  - read_file
+  - write_file
+  - run_command
+  - web_search
+  - send_to_canvas
+---
+# Source Ingestion And OCR
+
+Use this skill when the user gives arbitrary source files or asks to read, recognize, OCR, convert, or prepare inputs for another workflow. The task is to make the files usable without assuming they are already text.
+
+## Operating Loop
+
+1. Inventory inputs first. List paths, extensions, sizes, likely language, and whether each file is original media or derived output. Keep original files untouched.
+2. Probe before converting:
+   - `file`, `stat`, `pdfinfo`, `pdftotext`, `exiftool` when available.
+   - EPUB: inspect the archive or use an existing EPUB-to-Markdown script/tool.
+   - PDF: try text-layer extraction first; if empty or garbage, mark as image-only and choose OCR.
+   - Images/scans: inspect dimensions/orientation and run OCR only after deciding language and page segmentation.
+   - JSON/HTML/wiki manifests: read the manifest, follow local `html`, `pdf`, `iiif`, or source fields, and prefer structured extraction over OCR.
+   - Archives/directories: expand or enumerate into a durable work folder only when needed.
+3. Choose the simplest reliable route. Prefer existing project scripts and installed tools before adding dependencies. Install only when policy allows and record the command.
+4. Write durable outputs near the project workflow, usually `books/<id>/sources/markdown/`, `ocr/`, `artifacts/`, or a user-specified path. Use descriptive names; do not overwrite reviewed outputs unless asked.
+5. Produce a manifest for nontrivial ingestion. Include source path, sha256, extraction method, status (`complete`, `requires_ocr`, `failed`, `pending`), language, page/chapter counts, output path, and caveats.
+6. Validate the result externally: line/character counts, heading counts, boilerplate/debris checks, sample excerpts, and a no-text-layer check for PDFs marked `requires_ocr`.
+
+## OCR And Recognition Strategy
+
+- Treat OCR as a pipeline, not a guess: render pages, crop/deskew if needed, choose language (`chi_sim`, `chi_tra`, `jpn`, `jpn_vert`, `eng`, or combinations), test a small page range, then scale up.
+- For large scans, create resumable page-level outputs and a manifest before full OCR. Do not run a fragile all-pages command without logs and resume paths.
+- For vertical Japanese or classical Chinese scans, expect tool tuning. If OCR quality is poor, save page images and report that manual/model-assisted correction is required rather than fabricating text.
+- Preserve page references in OCR Markdown (`## Page N`) until the text is reviewed; only collapse into chapters after quality checks.
+
+## Expected Outputs
+
+When the downstream task expects Markdown, JSON, or a source bundle, create exactly that shape plus a report:
+
+- Markdown: clean body text with stable headings and paragraphs.
+- JSON: schema-valid and validated, not prose pretending to be JSON.
+- Source bundle: roles, source hashes, extraction status, and activation rules.
+- Report: what was converted, what was not, tools used, validation commands, and residual risks.
+
+## Safety Rules
+
+Do not claim a scanned PDF or image was converted if the output is empty, repeated headers, mojibake, or OCR garbage. Mark it `requires_ocr` or `failed` with evidence. Do not silently delete real content while removing boilerplate. Do not commit original large source media unless the repository already tracks that category and the user asked for it.

package/skills/structured-json/SKILL.md

@@ -0,0 +1,46 @@
+---
+id: structured-json
+label: Structured JSON Generation
+description: Use isolated schema-bound JSON generation for extraction, annotation, conversion, classification, repair, or chunked data production without mixing in agent runtime context.
+triggers:
+  - structured json
+  - json schema
+  - schema-bound
+  - json specialist
+  - json fetcher
+  - parallel json
+  - repair json
+  - valid json
+tools:
+  - json_specialist
+  - json_specialist_batch
+  - read_file
+  - write_file
+  - run_command
+---
+
+# Structured JSON
+
+Use this skill when the user needs reliable JSON that follows an explicit schema, especially for repetitive chunk processing.
+
+## Workflow
+
+1. Define the smallest useful JSON Schema for the next data artifact.
+2. Pass only the task, focused instructions, minimal domain context, input, and schema to `json_specialist`.
+3. Use `json_specialist_batch` only when items are independent and can be processed in parallel without shared writes.
+4. Keep formatting, file writes, validation scripts, compilation, and project-specific orchestration in the main agent.
+5. After the tool returns, validate any project-specific invariants with local scripts before treating the data as complete.
+
+## Provider Strategy
+
+- Prefer provider-native structured output when available, such as JSON Schema or JSON object mode.
+- Keep a fallback parser/repair path for providers that do not support strict schema responses.
+- On validation failure, retry with the exact schema errors and only the smallest relevant source text.
+- For batch work, write candidate JSON per chunk first; promote it only after schema and semantic validators pass.
+- Keep schema versions in the artifact metadata so old reviewed outputs can be reused or selectively regenerated when prompts change.
+
+## Boundaries
+
+- Do not pass shell, browser, file policy, package-install, or agent-planning context into the JSON specialist.
+- Do not make schemas book-, app-, or project-specific inside AgInTiFlow core. Project schemas belong in the target repository.
+- Do not let parallel JSON workers share one mutable output file. Use shard-local outputs, atomic renames, and a serialized merge/promote step.

package/src/agent-runner.js

@@ -26,6 +26,7 @@ import {
 import { refreshCodebaseMap } from "./codebase-map.js";
 import { readImage, researchWrapper, webResearch } from "./perception-tools.js";
 import { searchWeb } from "./web-search.js";
+import { runJsonSpecialist, runJsonSpecialistBatch } from "./json-specialist.js";
 import { runWritingSpecialist } from "./writing-specialist.js";
 import { runParallelScouts, shouldRunParallelScouts } from "./parallel-scouts.js";
 import { readProjectInstructions } from "./project.js";
@@ -1214,6 +1215,26 @@ function sanitizeToolArgs(toolName, args) {
         typeof args.constraints === "string" ? `[${Buffer.byteLength(args.constraints, "utf8")} bytes sha256=${hashForLog(args.constraints)}]` : safeArgs.constraints,
     };
   }
+  if (toolName === "json_specialist") {
+    return {
+      ...safeArgs,
+      task: typeof args.task === "string" ? `[${Buffer.byteLength(args.task, "utf8")} bytes sha256=${hashForLog(args.task)}]` : safeArgs.task,
+      instructions:
+        typeof args.instructions === "string" ? `[${Buffer.byteLength(args.instructions, "utf8")} bytes sha256=${hashForLog(args.instructions)}]` : safeArgs.instructions,
+      context: typeof args.context === "string" ? `[${Buffer.byteLength(args.context, "utf8")} bytes sha256=${hashForLog(args.context)}]` : safeArgs.context,
+      inputText:
+        typeof args.inputText === "string" ? `[${Buffer.byteLength(args.inputText, "utf8")} bytes sha256=${hashForLog(args.inputText)}]` : safeArgs.inputText,
+      schemaJson:
+        typeof args.schemaJson === "string" ? `[${Buffer.byteLength(args.schemaJson, "utf8")} bytes sha256=${hashForLog(args.schemaJson)}]` : safeArgs.schemaJson,
+    };
+  }
+  if (toolName === "json_specialist_batch") {
+    return {
+      ...safeArgs,
+      defaults: args.defaults ? "[json specialist defaults redacted]" : safeArgs.defaults,
+      items: Array.isArray(args.items) ? `[${args.items.length} json specialist items]` : safeArgs.items,
+    };
+  }
   if (toolName === "apply_patch") {
     return {
       ...safeArgs,
@@ -1309,6 +1330,19 @@ export function sanitizeToolResult(result) {
     safeResult.draftTruncated = true;
     delete safeResult.draft;
   }
+  if (safeResult.toolName === "json_specialist" && safeResult.result !== undefined) {
+    const encoded = JSON.stringify(safeResult.result);
+    safeResult.resultBytes = Buffer.byteLength(encoded, "utf8");
+    if (safeResult.resultBytes > TOOL_RESULT_INLINE_CONTENT_BYTES) {
+      safeResult.resultPreview = encoded.slice(0, TOOL_RESULT_CONTENT_PREVIEW_CHARS);
+      safeResult.resultTruncated = true;
+      delete safeResult.result;
+    }
+  }
+  if (safeResult.toolName === "json_specialist_batch" && Array.isArray(safeResult.results)) {
+    safeResult.resultCount = safeResult.results.length;
+    safeResult.results = safeResult.results.map((item) => sanitizeToolResult(item));
+  }
   return safeResult;
 }
 
@@ -1584,6 +1618,20 @@ async function executeTool(browserState, toolCall, snapshot, config, store, obse
         observers.event(result.ok ? "tool.completed" : "tool.failed", eventResult);
         return result;
       }
+      case "json_specialist": {
+        const result = await runJsonSpecialist(args, config, store);
+        const eventResult = sanitizeToolResult(result);
+        await store.appendEvent(result.ok ? "tool.completed" : "tool.failed", eventResult);
+        observers.event(result.ok ? "tool.completed" : "tool.failed", eventResult);
+        return result;
+      }
+      case "json_specialist_batch": {
+        const result = await runJsonSpecialistBatch(args.items || [], { ...args, items: undefined }, config, store);
+        const eventResult = sanitizeToolResult(result);
+        await store.appendEvent(result.ok ? "tool.completed" : "tool.failed", eventResult);
+        observers.event(result.ok ? "tool.completed" : "tool.failed", eventResult);
+        return result;
+      }
       case "writing_specialist": {
         const result = await runWritingSpecialist(args, config, store);
         const eventResult = sanitizeToolResult(result);

package/src/guardrails.js

@@ -304,6 +304,66 @@ export function checkToolUse({ toolName, args, snapshot, config }) {
     return { allowed: true };
   }
 
+  if (toolName === "json_specialist") {
+    const task = String(args.task || args.prompt || "").trim();
+    if (!task) return { allowed: false, reason: "JSON specialist requires task.", category: "json-specialist" };
+    if (!args.schema && !String(args.schemaJson || "").trim()) {
+      return { allowed: false, reason: "JSON specialist requires schema or schemaJson.", category: "json-specialist" };
+    }
+    const provider = String(args.provider || process.env.AGINTI_JSON_PROVIDER || "").trim();
+    if (provider && !["deepseek", "openai", "qwen", "venice", "mock"].includes(provider)) {
+      return { allowed: false, reason: `Unknown JSON specialist provider: ${provider}`, category: "json-specialist" };
+    }
+    const payloadBytes = Buffer.byteLength(
+      [
+        task,
+        args.instructions,
+        args.requirements,
+        args.context,
+        args.inputText,
+        args.source,
+        args.content,
+        args.schemaJson,
+        args.inputJson ? JSON.stringify(args.inputJson) : "",
+        args.schema ? JSON.stringify(args.schema) : "",
+      ]
+        .filter(Boolean)
+        .join("\n"),
+      "utf8"
+    );
+    if (payloadBytes > 220_000) {
+      return {
+        allowed: false,
+        reason: "JSON specialist payload is too large. Split the source into smaller independent items or save inputs to files and pass a focused excerpt.",
+        category: "json-specialist",
+      };
+    }
+    return { allowed: true };
+  }
+
+  if (toolName === "json_specialist_batch") {
+    const items = Array.isArray(args.items) ? args.items : [];
+    if (items.length === 0) return { allowed: false, reason: "JSON specialist batch requires items.", category: "json-specialist" };
+    if (items.length > 32) return { allowed: false, reason: "JSON specialist batch is limited to 32 items per tool call.", category: "json-specialist" };
+    const concurrency = Number(args.concurrency || 4);
+    if (Number.isFinite(concurrency) && concurrency > 16) {
+      return { allowed: false, reason: "JSON specialist batch concurrency is limited to 16.", category: "json-specialist" };
+    }
+    const provider = String(args.provider || args.defaults?.provider || process.env.AGINTI_JSON_PROVIDER || "").trim();
+    if (provider && !["deepseek", "openai", "qwen", "venice", "mock"].includes(provider)) {
+      return { allowed: false, reason: `Unknown JSON specialist provider: ${provider}`, category: "json-specialist" };
+    }
+    const payloadBytes = Buffer.byteLength(JSON.stringify({ defaults: args.defaults || {}, items }), "utf8");
+    if (payloadBytes > 360_000) {
+      return {
+        allowed: false,
+        reason: "JSON specialist batch payload is too large. Use fewer items or smaller chunk text per call.",
+        category: "json-specialist",
+      };
+    }
+    return { allowed: true };
+  }
+
   if (toolName === "generate_image") {
     if (!config.allowAuxiliaryTools) {
       return { allowed: false, reason: "Auxiliary tools are disabled for this run.", category: "auxiliary-tools" };

package/src/json-specialist.js

@@ -0,0 +1,422 @@
+import crypto from "node:crypto";
+import { createChatCompletion, createClient } from "./model-client.js";
+import { getProviderDefaults } from "./model-routing.js";
+import { redactSensitiveText, redactValue } from "./redaction.js";
+
+const MAX_INLINE_PREVIEW = 1600;
+const JSON_PROVIDERS = new Set(["openai", "deepseek", "qwen", "venice", "mock"]);
+
+function compact(value = "", limit = MAX_INLINE_PREVIEW) {
+  const text = redactSensitiveText(String(value || "").trim());
+  if (text.length <= limit) return text;
+  return `${text.slice(0, Math.max(0, limit - 24))} ... [truncated]`;
+}
+
+function parseJsonValue(content = "") {
+  const text = String(content || "").trim();
+  if (!text) return { ok: false, error: "empty JSON response" };
+  const fenced = text.match(/```(?:json)?\s*([\s\S]*?)```/i);
+  const candidates = [fenced?.[1], text].filter(Boolean);
+  for (const candidate of candidates) {
+    try {
+      return { ok: true, value: JSON.parse(candidate.trim()) };
+    } catch {
+      // Try a balanced excerpt below.
+    }
+    const starts = ["{", "["]
+      .map((char) => ({ char, index: candidate.indexOf(char) }))
+      .filter((item) => item.index >= 0)
+      .sort((a, b) => a.index - b.index);
+    for (const start of starts) {
+      const close = start.char === "{" ? "}" : "]";
+      const end = candidate.lastIndexOf(close);
+      if (end <= start.index) continue;
+      try {
+        return { ok: true, value: JSON.parse(candidate.slice(start.index, end + 1)) };
+      } catch {
+        // Keep looking.
+      }
+    }
+  }
+  return { ok: false, error: "response was not parseable JSON" };
+}
+
+function normalizeSchema(raw) {
+  let schema = raw;
+  let name = "";
+  let strict = true;
+  if (typeof schema === "string" && schema.trim()) {
+    schema = JSON.parse(schema);
+  }
+  if (schema?.schema && typeof schema.schema === "object" && !schema.type) {
+    name = String(schema.name || "").trim();
+    strict = schema.strict !== false;
+    schema = schema.schema;
+  }
+  if (!schema || typeof schema !== "object" || Array.isArray(schema)) {
+    throw new Error("JSON specialist requires a JSON Schema object.");
+  }
+  return { schema, name, strict };
+}
+
+function normalizeInputValue(args = {}) {
+  if (args.inputJson !== undefined) return redactValue(args.inputJson);
+  if (args.input !== undefined) return redactValue(args.input);
+  const text = args.inputText ?? args.source ?? args.content ?? "";
+  return redactSensitiveText(String(text || ""));
+}
+
+function normalizeResponseFormat(value = "") {
+  const normalized = String(value || "auto").trim().toLowerCase();
+  if (["auto", "json_schema", "json_object", "prompt"].includes(normalized)) return normalized;
+  return "auto";
+}
+
+function normalizeJsonRequest(args = {}) {
+  const normalizedSchema = normalizeSchema(args.schemaJson || args.schema);
+  const task = redactSensitiveText(String(args.task || args.prompt || "").trim());
+  const instructions = redactSensitiveText(String(args.instructions || args.requirements || "").trim());
+  const context = redactSensitiveText(String(args.context || "").trim());
+  const temperature = Number.isFinite(Number(args.temperature)) ? Math.min(Math.max(Number(args.temperature), 0), 1.2) : 0;
+  const maxTokens = Number.isFinite(Number(args.maxTokens)) ? Math.max(256, Math.floor(Number(args.maxTokens))) : 4096;
+  const provider = String(args.provider || process.env.AGINTI_JSON_PROVIDER || "").trim();
+  const model = String(args.model || process.env.AGINTI_JSON_MODEL || "").trim();
+  const schemaName = String(args.schemaName || normalizedSchema.name || "aginti_structured_output")
+    .trim()
+    .replace(/[^A-Za-z0-9_-]/g, "_")
+    .slice(0, 64) || "aginti_structured_output";
+  return {
+    task,
+    instructions,
+    context,
+    input: normalizeInputValue(args),
+    schema: normalizedSchema.schema,
+    schemaName,
+    strict: args.strict === undefined ? normalizedSchema.strict : args.strict !== false,
+    responseFormat: normalizeResponseFormat(args.responseFormat),
+    fallbackOnInvalid: args.fallbackOnInvalid !== false,
+    temperature,
+    maxTokens,
+    provider,
+    model,
+  };
+}
+
+function jsonSystemPrompt() {
+  return [
+    "You are the isolated AgInTiFlow JSON Specialist.",
+    "You transform only the supplied task, input, and schema into structured JSON.",
+    "You do not know or discuss AgInTiFlow internals, shell tools, browser tools, file policies, planning, package installs, or execution constraints.",
+    "Never include commentary, markdown fences, prose explanations, or partial objects.",
+    "Return one JSON value that satisfies the provided schema.",
+  ].join(" ");
+}
+
+function jsonUserPrompt(request, attempt) {
+  return JSON.stringify(
+    {
+      boundary:
+        "This is the complete context visible to the JSON specialist. Ignore absent agent/runtime details. Produce only schema-valid JSON.",
+      task: request.task,
+      instructions: request.instructions,
+      context: request.context,
+      input: request.input,
+      json_schema: request.schema,
+      output_contract: {
+        mode: attempt,
+        strict: request.strict,
+        requirement: "Return exactly one JSON value matching json_schema. Do not wrap it in markdown.",
+      },
+    },
+    null,
+    2
+  );
+}
+
+function allowedTypes(schema) {
+  if (!schema || schema.type === undefined) return [];
+  return Array.isArray(schema.type) ? schema.type : [schema.type];
+}
+
+function typeMatches(value, type) {
+  if (type === "array") return Array.isArray(value);
+  if (type === "object") return value && typeof value === "object" && !Array.isArray(value);
+  if (type === "integer") return Number.isInteger(value);
+  if (type === "number") return typeof value === "number" && Number.isFinite(value);
+  if (type === "string") return typeof value === "string";
+  if (type === "boolean") return typeof value === "boolean";
+  if (type === "null") return value === null;
+  return true;
+}
+
+function validateSchema(value, schema, path = "$") {
+  if (!schema || typeof schema !== "object") return [];
+  if (Array.isArray(schema.anyOf) && schema.anyOf.length > 0) {
+    const variants = schema.anyOf.map((variant) => validateSchema(value, variant, path));
+    return variants.some((errs) => errs.length === 0) ? [] : [`${path}: did not match anyOf`];
+  }
+  if (Array.isArray(schema.oneOf) && schema.oneOf.length > 0) {
+    const matches = schema.oneOf.filter((variant) => validateSchema(value, variant, path).length === 0).length;
+    return matches === 1 ? [] : [`${path}: did not match exactly one oneOf variant`];
+  }
+  const errors = [];
+  if (schema.const !== undefined && JSON.stringify(value) !== JSON.stringify(schema.const)) {
+    errors.push(`${path}: expected const ${JSON.stringify(schema.const)}`);
+  }
+  if (Array.isArray(schema.enum) && !schema.enum.some((item) => JSON.stringify(item) === JSON.stringify(value))) {
+    errors.push(`${path}: value is not in enum`);
+  }
+  const types = allowedTypes(schema);
+  if (types.length > 0 && !types.some((type) => typeMatches(value, type))) {
+    errors.push(`${path}: expected type ${types.join("|")}`);
+    return errors;
+  }
+  if (Array.isArray(value)) {
+    if (Number.isFinite(schema.minItems) && value.length < schema.minItems) errors.push(`${path}: fewer than minItems`);
+    if (Number.isFinite(schema.maxItems) && value.length > schema.maxItems) errors.push(`${path}: more than maxItems`);
+    if (schema.items) {
+      value.forEach((item, index) => errors.push(...validateSchema(item, schema.items, `${path}[${index}]`)));
+    }
+  } else if (value && typeof value === "object") {
+    const properties = schema.properties && typeof schema.properties === "object" ? schema.properties : {};
+    for (const key of schema.required || []) {
+      if (!(key in value)) errors.push(`${path}.${key}: required property missing`);
+    }
+    for (const [key, child] of Object.entries(properties)) {
+      if (key in value) errors.push(...validateSchema(value[key], child, `${path}.${key}`));
+    }
+    if (schema.additionalProperties === false) {
+      for (const key of Object.keys(value)) {
+        if (!(key in properties)) errors.push(`${path}.${key}: additional property not allowed`);
+      }
+    }
+  }
+  return errors;
+}
+
+function mockValueForSchema(schema) {
+  if (!schema || typeof schema !== "object") return {};
+  if (schema.const !== undefined) return schema.const;
+  if (Array.isArray(schema.enum) && schema.enum.length > 0) return schema.enum[0];
+  const type = allowedTypes(schema)[0] || (schema.properties ? "object" : "string");
+  if (type === "string") return "mock";
+  if (type === "integer") return 1;
+  if (type === "number") return 1;
+  if (type === "boolean") return true;
+  if (type === "null") return null;
+  if (type === "array") {
+    const count = Number.isFinite(schema.minItems) ? Math.max(1, schema.minItems) : 1;
+    return Array.from({ length: count }, () => mockValueForSchema(schema.items || { type: "string" }));
+  }
+  const result = {};
+  const properties = schema.properties && typeof schema.properties === "object" ? schema.properties : {};
+  const keys = new Set([...(schema.required || []), ...Object.keys(properties).slice(0, 6)]);
+  for (const key of keys) result[key] = mockValueForSchema(properties[key] || { type: "string" });
+  return result;
+}
+
+function attemptsForRequest(request, provider) {
+  if (request.responseFormat === "prompt") return ["prompt"];
+  if (request.responseFormat === "json_schema") return ["json_schema", "prompt"];
+  if (request.responseFormat === "json_object") return ["json_object", "prompt"];
+  if (["openai", "deepseek", "qwen"].includes(provider)) return ["json_schema", "json_object", "prompt"];
+  return ["json_object", "prompt"];
+}
+
+function responseFormatForAttempt(request, attempt) {
+  if (attempt === "json_schema") {
+    return {
+      type: "json_schema",
+      json_schema: {
+        name: request.schemaName,
+        strict: request.strict,
+        schema: request.schema,
+      },
+    };
+  }
+  if (attempt === "json_object") return { type: "json_object" };
+  return null;
+}
+
+function errorLooksLikeUnsupportedResponseFormat(error) {
+  const message = [error?.message, error?.error?.message, error?.response?.data?.error?.message]
+    .filter(Boolean)
+    .join(" ");
+  return /response_format|json_schema|json_object|unsupported|invalid request|400/i.test(message);
+}
+
+function parsedRawPreview(rawContent = "") {
+  return rawContent ? compact(rawContent, 1200) : "";
+}
+
+export async function runJsonSpecialist(args = {}, config = {}, store = null) {
+  let request;
+  try {
+    request = normalizeJsonRequest(args);
+  } catch (error) {
+    return {
+      ok: false,
+      toolName: "json_specialist",
+      reason: redactSensitiveText(error instanceof Error ? error.message : String(error)),
+    };
+  }
+  if (!request.task) {
+    return {
+      ok: false,
+      toolName: "json_specialist",
+      reason: "task is required.",
+    };
+  }
+
+  const startedAt = new Date().toISOString();
+  const requestFingerprint = crypto.createHash("sha256").update(JSON.stringify(redactValue(request))).digest("hex");
+  let model = request.model || config.model || "";
+  let provider = request.provider || config.provider || "";
+  let rawContent = "";
+  const attemptNotes = [];
+
+  try {
+    let result;
+    let validationErrors = [];
+    let usedResponseFormat = "mock";
+    if (config.provider === "mock" || provider === "mock") {
+      provider = "mock";
+      model = request.model || config.model || "mock-agent";
+      result = mockValueForSchema(request.schema);
+      validationErrors = validateSchema(result, request.schema);
+    } else {
+      if (provider && !JSON_PROVIDERS.has(provider)) {
+        throw new Error(`Unknown JSON specialist provider: ${provider}`);
+      }
+      const providerDefaults = request.provider ? getProviderDefaults(request.provider) : {};
+      const jsonConfig = {
+        ...config,
+        ...providerDefaults,
+        provider: provider || config.provider,
+        model: model || providerDefaults.model || config.model,
+      };
+      model = jsonConfig.model;
+      provider = jsonConfig.provider;
+      const client = createClient(jsonConfig);
+      const attempts = attemptsForRequest(request, provider);
+      for (const attempt of attempts) {
+        const responseFormat = responseFormatForAttempt(request, attempt);
+        const payload = {
+          model: jsonConfig.model,
+          temperature: request.temperature,
+          max_tokens: request.maxTokens,
+          messages: [
+            { role: "system", content: jsonSystemPrompt() },
+            { role: "user", content: jsonUserPrompt(request, attempt) },
+          ],
+          ...(responseFormat ? { response_format: responseFormat } : {}),
+        };
+        try {
+          const response = await createChatCompletion(client, payload, jsonConfig, `json specialist ${attempt} request`);
+          rawContent = response.choices[0]?.message?.content || "";
+          const parsed = parseJsonValue(rawContent);
+          if (!parsed.ok) {
+            attemptNotes.push(`${attempt}: ${parsed.error}`);
+            if (request.fallbackOnInvalid && attempt !== attempts.at(-1)) continue;
+            throw new Error(parsed.error);
+          }
+          const errors = validateSchema(parsed.value, request.schema);
+          if (errors.length > 0) {
+            attemptNotes.push(`${attempt}: ${errors.slice(0, 4).join("; ")}`);
+            if (request.fallbackOnInvalid && attempt !== attempts.at(-1)) continue;
+          }
+          result = parsed.value;
+          validationErrors = errors;
+          usedResponseFormat = attempt;
+          break;
+        } catch (error) {
+          const message = redactSensitiveText(error instanceof Error ? error.message : String(error));
+          attemptNotes.push(`${attempt}: ${message}`);
+          if (attempt !== attempts.at(-1) && (request.fallbackOnInvalid || errorLooksLikeUnsupportedResponseFormat(error))) continue;
+          throw error;
+        }
+      }
+    }
+
+    const artifact = {
+      version: 1,
+      generatedAt: new Date().toISOString(),
+      startedAt,
+      provider,
+      model,
+      responseFormat: usedResponseFormat,
+      requestFingerprint,
+      request: redactValue(request),
+      result: redactValue(result),
+      validationErrors,
+      attemptNotes,
+      rawPreview: parsedRawPreview(rawContent),
+    };
+    const artifactPath = store
+      ? await store.saveJsonArtifact(`json-specialist-${Date.now()}.json`, artifact).catch(() => "")
+      : "";
+    return {
+      ok: validationErrors.length === 0,
+      toolName: "json_specialist",
+      provider,
+      model,
+      responseFormat: usedResponseFormat,
+      args: {
+        task: request.task,
+        schemaName: request.schemaName,
+        responseFormat: request.responseFormat,
+        provider: request.provider,
+        requestFingerprint,
+      },
+      artifactPath,
+      result,
+      validationErrors,
+      attemptNotes,
+      rawPreview: parsedRawPreview(rawContent),
+    };
+  } catch (error) {
+    return {
+      ok: false,
+      toolName: "json_specialist",
+      provider,
+      model,
+      args: {
+        task: request.task,
+        schemaName: request.schemaName,
+        responseFormat: request.responseFormat,
+        provider: request.provider,
+        requestFingerprint,
+      },
+      error: redactSensitiveText(error instanceof Error ? error.message : String(error)),
+      attemptNotes,
+      rawPreview: parsedRawPreview(rawContent),
+    };
+  }
+}
+
+export async function runJsonSpecialistBatch(tasks = [], options = {}, config = {}, store = null) {
+  const items = Array.isArray(tasks) ? tasks : [];
+  const concurrency = Math.min(Math.max(Number(options.concurrency) || 4, 1), 32);
+  const results = new Array(items.length);
+  let nextIndex = 0;
+
+  async function worker() {
+    while (nextIndex < items.length) {
+      const index = nextIndex;
+      nextIndex += 1;
+      const task = items[index] || {};
+      results[index] = await runJsonSpecialist({ ...options.defaults, ...task }, config, store);
+    }
+  }
+
+  await Promise.all(Array.from({ length: Math.min(concurrency, items.length) }, () => worker()));
+  return {
+    ok: results.every((item) => item?.ok),
+    toolName: "json_specialist_batch",
+    count: results.length,
+    succeeded: results.filter((item) => item?.ok).length,
+    failed: results.filter((item) => !item?.ok).length,
+    concurrency,
+    results,
+  };
+}

package/src/model-client.js

@@ -567,6 +567,27 @@ function mockWritingSpecialistToolForGoal(goal = "", taskProfile = "") {
   });
 }
 
+function mockJsonSpecialistToolForGoal(goal = "") {
+  const text = String(goal || "");
+  if (!/\b(json|schema|structured output|extract structured|valid object|valid array)\b/i.test(text)) return null;
+  return mockToolCall("json_specialist", {
+    task: text.replace(/\s+/g, " ").slice(0, 800) || "Return structured JSON.",
+    schemaName: "mock_structured_output",
+    schema: {
+      type: "object",
+      properties: {
+        summary: { type: "string" },
+        complete: { type: "boolean" },
+      },
+      required: ["summary", "complete"],
+      additionalProperties: false,
+    },
+    inputText: text.slice(0, 1200),
+    responseFormat: "prompt",
+    provider: "mock",
+  });
+}
+
 function mockPreviewToolForGoal(goal = "") {
   const text = String(goal).toLowerCase();
   if (!/(open|preview|view|browser|website|web\s*site)/.test(text)) return null;
@@ -676,6 +697,7 @@ export async function createPlan(client, config, state) {
             ? "web_search is available for lightweight snippets. web_research is available for auditable sourced research with persisted artifacts; use mode=snippets by default and mode=openai only when hosted OpenAI web research is needed and configured. Prefer these tools over opening a search engine in the browser."
             : "web_search is disabled for this run.",
           "For substantial writing work such as novels, chapters, books, scripts, essays, LaTeX manuscripts, or research-paper prose, plan to call writing_specialist with only the writing brief/canon/style/draft context. The main agent should handle files, citations, checks, and Markdown/LaTeX/Final Draft formatting after the isolated writing draft returns.",
+          "For repetitive schema-bound extraction, annotation, conversion, or validation tasks, use json_specialist with only the task, input, schema, and focused instructions. It calls the model directly for strict JSON, tries provider-native structured output when supported, and keeps agent/runtime/tool context out of the specialist prompt.",
           config.allowFileTools
             ? "read_image is available for workspace-local or allowed remote screenshots/images using OpenAI vision when OPENAI_API_KEY is configured. It returns typed visual observations and persists a perception artifact; if credentials are missing, report the blocker instead of guessing from the filename."
             : "",
@@ -729,6 +751,66 @@ export async function requestNextStep(client, config, messages) {
         },
       },
     },
+    {
+      type: "function",
+      function: {
+        name: "json_specialist",
+        description:
+          "Call an isolated schema-only LLM context for strict JSON extraction, annotation, classification, conversion, or validation. Pass only the task, input, schema, and focused instructions; do not pass shell/browser/file policy, agent runtime, or broad planning context. The tool tries provider-native structured JSON where supported and falls back to prompt-and-validate parsing.",
+        parameters: {
+          type: "object",
+          properties: {
+            task: { type: "string", description: "Focused JSON task. Required." },
+            instructions: { type: "string", description: "Additional schema-specific rules or quality constraints." },
+            context: { type: "string", description: "Minimal domain context needed for the JSON transformation." },
+            inputText: { type: "string", description: "Source text or serialized input to transform." },
+            inputJson: { type: "object", description: "Source object to transform.", additionalProperties: true },
+            schema: { type: "object", description: "JSON Schema object for the expected output.", additionalProperties: true },
+            schemaJson: { type: "string", description: "JSON-stringified schema alternative when schema is easier to pass as text." },
+            schemaName: { type: "string", description: "Short schema name for provider-native structured output." },
+            responseFormat: {
+              type: "string",
+              enum: ["auto", "json_schema", "json_object", "prompt"],
+              description: "Native structured-output preference. auto tries native JSON schema/object then prompt fallback.",
+            },
+            fallbackOnInvalid: { type: "boolean", description: "Try a weaker fallback if native structured output is unsupported or invalid. Defaults true." },
+            strict: { type: "boolean", description: "Use strict provider schema mode where supported. Defaults true." },
+            temperature: { type: "number", description: "Defaults to 0 for deterministic structured data." },
+            maxTokens: { type: "integer", description: "Maximum completion tokens for the JSON specialist call." },
+            provider: {
+              type: "string",
+              enum: ["deepseek", "openai", "qwen", "venice", "mock"],
+              description: "Optional provider override. Defaults to AGINTI_JSON_PROVIDER or current provider.",
+            },
+            model: { type: "string", description: "Optional model override. Defaults to AGINTI_JSON_MODEL or current model." },
+          },
+          required: ["task"],
+          additionalProperties: false,
+        },
+      },
+    },
+    {
+      type: "function",
+      function: {
+        name: "json_specialist_batch",
+        description:
+          "Run many isolated json_specialist requests concurrently for chunked structured-data work. Each item is one independent schema-bound task. Use only for independent items with no shared writes or ordering dependency.",
+        parameters: {
+          type: "object",
+          properties: {
+            concurrency: { type: "integer", description: "Parallel request count, clamped by runtime guardrails." },
+            defaults: { type: "object", description: "Default json_specialist arguments applied to every item.", additionalProperties: true },
+            items: {
+              type: "array",
+              description: "Independent json_specialist argument objects.",
+              items: { type: "object", additionalProperties: true },
+            },
+          },
+          required: ["items"],
+          additionalProperties: false,
+        },
+      },
+    },
     {
       type: "function",
       function: {
@@ -1393,6 +1475,11 @@ export async function requestNextStep(client, config, messages) {
       }
     }
 
+    const jsonTool = mockJsonSpecialistToolForGoal(config.goal);
+    if (jsonTool) {
+      return mockChatResponse("Mock mode will exercise the isolated JSON specialist.", [jsonTool]);
+    }
+
     const writingTool = mockWritingSpecialistToolForGoal(config.goal, config.taskProfile);
     if (writingTool) {
       return mockChatResponse("Mock mode will exercise the isolated writing specialist before file or format work.", [writingTool]);

package/src/step-budget-controller.js

@@ -6,6 +6,8 @@ const PROGRESS_TOOL_NAMES = new Set([
   "apply_patch",
   "generate_image",
   "inspect_project",
+  "json_specialist",
+  "json_specialist_batch",
   "open_workspace_file",
   "preview_workspace",
   "read_file",

package/src/task-profiles.js

@@ -132,6 +132,13 @@ export const TASK_PROFILES = {
       "Bias toward supervising another agent or long-running task instead of doing the target work directly. Define acceptance criteria, give the student agent normal user-level prompts, monitor progress through tmux/session logs/artifacts, independently verify claims, record evidence, and convert repeated failures into reusable AgInTiFlow skills, tools, policies, tests, or profile improvements.",
     tools: ["shell", "files", "canvas", "inspect_project", "tmux"],
   },
+  pipeline: {
+    id: "pipeline",
+    label: "Self-healing pipeline",
+    prompt:
+      "Bias toward keeping long-running writer, reviewer, repairer, monitor, queue, ETL, batch generation, and build pipelines observable, resumable, and moving from raw inputs to final artifacts. Diagnose from status files, logs, manifests, tmux panes, heartbeats, and timestamps before intervening. Distinguish healthy waiting from stalls. Preserve valid artifacts, quarantine stale or invalid outputs, and patch project-owned scripts/prompts/validators in small reversible steps only when evidence supports it. For new pipelines, create a local contract with source manifest, stable chunks, schemas, validators, runners, checkpoint builds, and completion evidence. Choose sequential, parallel, async, or review-gated designs according to the user's request and the project's existing architecture. Verify with focused checks, checkpoint artifacts, then restart only the affected session and record the exact resume command.",
+    tools: ["inspect_project", "search_files", "read_file", "apply_patch", "shell", "tmux", "sandbox"],
+  },
   app: {
     id: "app",
     label: "App builder",
@@ -391,6 +398,16 @@ const PROFILE_ALIASES = {
   curriculum: "supervision",
   selfsupervision: "supervision",
   "self-supervision": "supervision",
+  pipeline: "pipeline",
+  pipelines: "pipeline",
+  autorepair: "pipeline",
+  "auto-repair": "pipeline",
+  "self-healing": "pipeline",
+  stuck: "pipeline",
+  stalled: "pipeline",
+  monitor: "pipeline",
+  queue: "pipeline",
+  reviewer: "pipeline",
   cpp: "c-cpp",
   "c++": "c-cpp",
   clang: "c-cpp",
@@ -432,6 +449,7 @@ export function defaultMaxStepsForProfile(value = "auto") {
   if (profile === "android") return 60;
   if (profile === "latex") return 30;
   if (profile === "supervision") return 40;
+  if (profile === "pipeline") return 44;
   if (profile === "aaps") return 36;
   if (["devops", "security"].includes(profile)) return 36;
   if (