@lazyingart/agintiflow 0.20.145 → 0.20.151

package/README.md

@@ -240,7 +240,7 @@ The website keeps the visual walkthrough in a carousel so this README can stay f
 | SCS mode | Optional Student-Committee-Supervisor quality gate for complicated or risky tasks. |
 | AAPS adapter | Optional `@lazyingart/aaps` integration for `.aaps` workflow init, validate, parse, compile, dry-run, and run commands. |
 | Image generation | Optional GRS AI and Venice image tools with saved manifests and canvas artifact previews. |
-| Skill library | Built-in Markdown skills for code, websites, Android/iOS, Python, Rust, Java, LaTeX, writing, reviews, GitHub, AAPS, and more. |
+| Skill library | Built-in Markdown skills for code, websites, Android/iOS, Python, Rust, Java, LaTeX, writing, reviews, source ingestion/OCR, structured JSON, autonomous artifact pipelines, GitHub, AAPS, and more. |
 | Skill Mesh | Optional strict skill recording/sharing for reviewed reusable skill packs. If unused, AgInTiFlow runs normally without background sharing. |
 | Multilingual UI | CLI and docs language support for English, Japanese, Simplified/Traditional Chinese, Korean, French, Spanish, Arabic, Vietnamese, German, and Russian. |
 

package/docs/autonomous-artifact-pipelines.md

@@ -0,0 +1,36 @@
+# Autonomous Artifact Pipelines
+
+AgInTiFlow can run a project from raw inputs to final artifacts when the work is expressed as a local, resumable pipeline. This pattern is for jobs such as source conversion, book generation, report building, dataset annotation, media preparation, or any workflow where partial outputs are valuable and failure recovery matters.
+
+## Contract
+
+Each pipeline should create these project-local files or equivalents:
+
+- Source manifest: raw files, hashes, roles, languages, extraction method, and caveats.
+- Derived inputs: Markdown, text, tables, images, or structured bundles created from the raw files.
+- Task manifest: stable chunk IDs, source locations, dependencies, prompt/schema version, and output paths.
+- Schema and validator: the exact artifact shape plus semantic checks that define a promotable output.
+- Runners: writer, reviewer, repairer, monitor, merge, compile/export, and status commands.
+- Completion report: counts, first missing ID, failed/quarantined items, latest previews, final artifact paths, and resume commands.
+
+The target repository owns its schemas, prompts, chunk policy, and rendering code. AgInTiFlow owns the behavior: inspect, create missing scripts, run observable sessions, preserve valid work, validate, repair, compile, and report evidence.
+
+## Roles
+
+The writer creates candidate artifacts. It should never be the only quality gate.
+
+The validator promotes candidates only after schema and project-specific checks pass.
+
+The reviewer inspects valid-looking artifacts for missing source units, source drift, repeated filler, malformed annotations, suspicious all-one-style output, and other known quality failures. It writes candidate fixes or failed-only repair requests.
+
+The repairer runs independently of the writer. It can wake from status files, handle failed or quarantined chunks, retry with exact validator errors, reduce chunk size, or escalate to a stronger model when the project allows it. If the error is mechanical, it should repair locally first: punctuation restoration from source text, token splitting, default metadata, schema migration, and renderer wrapping should not consume model calls.
+
+The monitor is gentle. It waits through healthy progress and provider limits, restarts only on hard evidence of stall or crash, and records each decision.
+
+## Concurrency
+
+Parallelism is optional. When used, each worker needs deterministic shard ownership, separate logs, atomic writes, and no direct compile responsibility. Merge, promotion, compilation, publishing, and commits should be serialized unless the project already has a safe coordinator.
+
+## Completion
+
+A run is complete only when the final artifact was built from the current manifest and the status report shows full coverage or intentional quarantine. A successful tmux pane, a page count, or a single preview file is not enough.

package/docs/self-healing-pipelines.md

@@ -0,0 +1,23 @@
+# Self-Healing Pipelines
+
+AgInTiFlow treats long-running writers, reviewers, repairers, monitors, queues, ETL jobs, and batch generators as resumable pipelines rather than one-shot commands.
+
+Use the `pipeline` task profile or rely on automatic skill selection when a prompt mentions a stalled writer, monitor, reviewer, queue worker, failed chunks, retry passes, stale claims, or tmux supervision.
+
+## Operating Model
+
+The agent should first read status files, manifests, logs, and tmux panes. It should compare progress across observations before calling a job stalled unless a hard error is visible. Healthy provider waits, rate limits, and long compile steps should usually be left alone.
+
+When repair is justified, the agent should patch project-owned scripts or prompts in small reversible changes. Preferred repairs include failed-only retry modes, bounded retry passes, atomic writes, stale claim cleanup, idempotent compile commands, clear status JSON, heartbeat files, and durable logs.
+
+Concurrency is a tool, not a product stance. AgInTiFlow should choose sequential, parallel, async, or review-gated operation from the user's request and the local project evidence. It should not force a sharded design into projects that do not need it.
+
+Review and repair are separate from writing. A reviewer should detect missing source units, repeated filler, malformed structured data, source drift, and known quality failures, then produce candidate repairs or failed-only requests. A repairer should be able to run independently of the writer, wake from status files, run bounded passes, and exit without blocking healthy progress.
+
+## Boundaries
+
+AgInTiFlow should not embed project-specific schemas in its core. A book writer, data pipeline, or build system owns its own validators and artifact layout. AgInTiFlow provides the reusable behavior: diagnose, preserve valid work, patch the local workflow, verify, build checkpoint artifacts, restart only affected sessions, and report exact resume commands.
+
+## Verification
+
+After a repair, the agent should run syntax checks for changed scripts, perform a dry-run or bounded batch when safe, inspect counters and first-missing IDs, and keep unrelated tmux sessions running. If the same symptom repeats, the agent should improve the project workflow or a reusable AgInTiFlow skill instead of repeatedly sending manual nudges.

package/docs/skills-and-tools.md

@@ -12,7 +12,7 @@ AgInTiFlow separates **skills** from **tools** so the agent can stay general whi
 
 ## Built-In Skills
 
-The package ships built-in skills for code engineering, website/app building, LaTeX manuscripts, books, Microsoft Word documents, image generation, GitHub maintenance, system maintenance, tmux session control, Android, R/Stan, Python, C/C++, shell scripting, AAPS, novel writing, and supervision/student-agent training.
+The package ships built-in skills for code engineering, website/app building, LaTeX manuscripts, books, Microsoft Word documents, image generation, GitHub maintenance, system maintenance, source ingestion/OCR, structured JSON, autonomous artifact pipelines, tmux session control, Android, R/Stan, Python, C/C++, shell scripting, AAPS, novel writing, and supervision/student-agent training.
 
 List them from a project:
 
@@ -78,3 +78,5 @@ For schema-bound structured data, prefer:
 - `json_specialist_batch` for independent chunks that can be requested in parallel without shared writes.
 
 The JSON specialist receives only the task, focused instructions, minimal context, input, and JSON Schema. It tries provider-native structured output (`json_schema` or JSON object mode) when available, then falls back to prompt-and-validate parsing.
+
+For raw-input-to-final-output work, prefer the autonomous artifact pipeline pattern. The target project should own its source manifest, chunk manifest, schemas, validators, runner scripts, reviewer/repairer logic, and compiler/exporter. AgInTiFlow should create or patch those project-local pieces, run them in observable sessions, and verify checkpoint artifacts before declaring completion.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lazyingart/agintiflow",
-  "version": "0.20.145",
+  "version": "0.20.151",
   "type": "module",
   "description": "AgInTiFlow is a project-aware agent workspace for hybrid wet-dry R&D, hardware-aware intelligence, software automation, and industrial workflows.",
   "license": "Apache-2.0",

package/scripts/smoke-capabilities.js

@@ -129,6 +129,10 @@ try {
     capabilities.tools?.taskProfiles?.some((profile) => profile.id === "supervision"),
     "capabilities did not report supervision task profile"
   );
+  assert(
+    capabilities.tools?.taskProfiles?.some((profile) => profile.id === "pipeline"),
+    "capabilities did not report pipeline task profile"
+  );
   for (const profileId of ["docs", "data", "qa", "database", "devops", "security", "slides", "education", "java", "ios", "go", "rust", "dotnet", "php", "ruby"]) {
     assert(
       capabilities.tools?.taskProfiles?.some((profile) => profile.id === profileId),
@@ -138,6 +142,7 @@ try {
   const qaProfile = listTaskProfiles().find((profile) => profile.id === "qa");
   assert(qaProfile, "QA profile is missing");
   assert(defaultMaxStepsForProfile("qa") >= 40, "QA profile step budget is too low for verification and cleanup");
+  assert(defaultMaxStepsForProfile("pipeline") >= 44, "pipeline profile step budget is too low for repair/verify/resume loops");
   assert(!/misleading failing test/i.test(qaProfile.prompt), "QA profile still encourages misleading test fixtures");
   assert(/do not stage fake bugs/i.test(qaProfile.prompt), "QA profile does not discourage fake staged failures");
   assert(
@@ -168,6 +173,10 @@ try {
     capabilities.tools?.skills?.some((skill) => skill.id === "supervision-student"),
     "capabilities did not report built-in supervision skill"
   );
+  assert(
+    capabilities.tools?.skills?.some((skill) => skill.id === "self-healing-pipeline"),
+    "capabilities did not report built-in self-healing pipeline skill"
+  );
   for (const skillId of ["data-analysis", "docs-knowledge", "qa-testing", "database", "devops-deployment", "security-review", "presentation-slides", "writing-editing", "java-jvm", "ios-swift", "go", "rust", "dotnet-csharp", "php", "ruby"]) {
     assert(
       capabilities.tools?.skills?.some((skill) => skill.id === skillId),

package/scripts/smoke-skills.js

@@ -21,6 +21,7 @@ const ids = new Set(skills.map((skill) => skill.id));
 assert(skills.length >= 27, "expected built-in skills to load");
 for (const required of [
   "aaps",
+  "autonomous-artifact-pipeline",
   "code",
   "code-review",
   "data-analysis",
@@ -41,6 +42,9 @@ for (const required of [
   "rust",
   "php",
   "security-review",
+  "self-healing-pipeline",
+  "source-ingestion",
+  "structured-json",
   "system-maintenance",
   "supervision-student",
   "tmux-session",
@@ -57,6 +61,13 @@ assert(selectedIds("edit a Microsoft Word docx and preserve the original").inclu
 assert(selectedIds("generate a logo image with grsai nanobanana").includes("image-generation"), "image prompt did not select image-generation");
 assert(selectedIds("git status commit push with gh").includes("github-maintenance"), "git prompt did not select github-maintenance");
 assert(selectedIds("monitor a long running tmux session").includes("tmux-session"), "tmux prompt did not select tmux-session");
+assert(selectedIds("repair a stuck writer monitor pipeline and retry failed chunks").includes("self-healing-pipeline"), "stuck pipeline prompt did not select self-healing-pipeline");
+assert(
+  selectedIds("from raw pdf epub sources create markdown chunks json and compile final pdf in tmux with monitor and auto repair").includes(
+    "autonomous-artifact-pipeline"
+  ),
+  "raw-source artifact pipeline prompt did not select autonomous-artifact-pipeline"
+);
 assert(selectedIds("create an .aaps example for @lazyingart/aaps").includes("aaps"), "AAPS prompt did not select aaps");
 assert(selectedIds("debug a C++ CMake build").includes("c-cpp"), "C++ prompt did not select c-cpp");
 assert(selectedIds("set up Stan and CmdStanR reproducibly").includes("r-stan"), "Stan prompt did not select r-stan");
@@ -68,6 +79,8 @@ assert(selectedIds("debug a C# dotnet web API").includes("dotnet-csharp"), ".NET
 assert(selectedIds("fix a PHP Laravel composer project").includes("php"), "PHP prompt did not select php");
 assert(selectedIds("repair a Ruby Rails app with RSpec").includes("ruby"), "Ruby prompt did not select ruby");
 assert(selectedIds("clean a CSV dataset and make plots").includes("data-analysis"), "data prompt did not select data-analysis");
+assert(selectedIds("convert scanned PDF EPUB and image sources to markdown with OCR").includes("source-ingestion"), "source ingestion prompt did not select source-ingestion");
+assert(selectedIds("use json schema to fetch valid structured json for each chunk in parallel").includes("structured-json"), "structured JSON prompt did not select structured-json");
 assert(selectedIds("write README API docs and a tutorial").includes("docs-knowledge"), "docs prompt did not select docs-knowledge");
 assert(selectedIds("fix failing tests and add regression coverage").includes("qa-testing"), "QA prompt did not select qa-testing");
 const qaSkill = skills.find((skill) => skill.id === "qa-testing");

package/skills/autonomous-artifact-pipeline/SKILL.md

@@ -0,0 +1,71 @@
+---
+id: autonomous-artifact-pipeline
+label: Autonomous Artifact Pipeline
+description: Turn raw inputs into validated final artifacts through a resumable, observable pipeline with ingestion, chunking, writers, reviewers, repairers, monitors, and checkpoint builds.
+triggers:
+  - raw source to final output
+  - autonomous pipeline
+  - artifact pipeline
+  - final artifact
+  - prepare chunks
+  - meta tasks
+  - writer reviewer repairer
+  - tmux monitor
+  - resumable batch
+  - compile output
+  - auto repair
+tools:
+  - inspect_project
+  - read_file
+  - write_file
+  - apply_patch
+  - run_command
+  - tmux_start_session
+  - tmux_capture_pane
+  - tmux_send_keys
+  - send_to_canvas
+---
+# Autonomous Artifact Pipeline
+
+Use this skill when the user wants AgInTiFlow to start from raw materials and keep working until durable final artifacts exist, such as books, reports, datasets, PDFs, apps, slides, media bundles, or generated JSON corpora.
+
+## Pipeline Contract
+
+Before launching long work, create or identify a project-local contract:
+
+1. Source manifest: raw paths, hashes, source roles, language/type, extraction method, and caveats.
+2. Derived inputs: Markdown, text, images, tables, or structured bundles produced from the raw sources.
+3. Task manifest: stable chunk IDs, source location, dependency order, prompt/schema version, and output paths.
+4. Artifact schema: JSON Schema or other validator-owned shape for each generated unit.
+5. Runners: resumable writer, reviewer, repairer, monitor, merge, compile/export, and status commands.
+6. Completion evidence: counters, first missing item, failed IDs, current previews, final artifact paths, and resume commands.
+
+Project-specific schemas, prompts, layouts, and compilers belong in the target repository. AgInTiFlow provides the orchestration pattern and should generate or patch local scripts when they are missing.
+
+## Execution Pattern
+
+- Inspect the repository and instructions first. Preserve raw inputs and do not overwrite reviewed outputs.
+- Convert raw files into durable intermediate inputs before asking a model to generate downstream artifacts.
+- Split work into deterministic chunks that survive reruns. If chunk policy changes, map old outputs by stable source IDs instead of restarting from zero.
+- Use isolated structured-data calls for repetitive JSON units. Keep prompts focused on the chunk, schema, source references, and validation errors.
+- Add deterministic canonicalizers before model retry when failures are representational, such as punctuation normalization, stable token splitting, missing metadata backfill, schema version migration, or renderer-specific wrapping. Do not spend provider calls on repairs a local script can prove.
+- Run writers in tmux or another observable background process. Each worker must have disjoint claims, atomic output writes, and shard-local logs.
+- Keep review and repair asynchronous but safe. Reviewers may produce candidate fixes while writers continue; only validators or merge scripts promote candidates.
+- Compile or export checkpoint previews after successful merge batches and always at final completion.
+- Commit reusable scripts, manifests, validators, templates, and stable checkpoints when the project expects git tracking.
+
+## Autorepair Behavior
+
+A robust pipeline has an independent repair path that is not blocked by the main writer:
+
+- Heartbeats record active worker, current chunk, last success, last failure, and provider wait state.
+- Provider/rate-limit failures wait with backoff and retry at long intervals.
+- Schema/parse failures are repaired with the exact validator error and the smallest useful input.
+- Mechanical validation failures are repaired locally first when the canonical form is derivable from source text or schema rules.
+- Semantic/source-drift failures are retried with smaller chunks or stronger source references.
+- Repeated failures are quarantined with reasons, then handled by a bounded failed-only repair pass.
+- Monitor intervention is gentle: observe healthy progress, restart only on hard error, stale claim, repeated no-progress window, or missing child process.
+
+## Done Criteria
+
+Do not call the task complete until the final artifact was built from the current manifest and the status report shows complete or intentionally quarantined coverage. A partial PDF, stale page count, or successful worker log is not enough.

package/skills/bilingual-interlinear-book/SKILL.md

@@ -27,13 +27,14 @@ Use this skill when the task asks for a paired-language book, ruby/furigana/piny
 
 1. Inspect repository instructions, existing scripts, book plans, and ignored paths before editing.
 2. Keep original PDFs/EPUBs in source folders and do not commit large source media unless the repository explicitly tracks them.
-3. Convert source books to durable Markdown first. Keep raw and cleaned Markdown separate when OCR or EPUB extraction is noisy.
+3. Convert source books to durable Markdown first. Treat PDF, EPUB, image, JSON/wiki, and scanned sources as ingestion problems; create a source manifest with hashes, method, role, language, and caveats before generation. Keep raw and cleaned Markdown separate when OCR or EPUB extraction is noisy.
 4. Split cleaned Markdown into stable paragraph- or chapter-scoped chunks with `manifest.json` and `chunks.jsonl`. Use source paragraph IDs that survive reruns. If a paragraph is too large for reliable provider output, split it into ordered subchunks at sentence or clause boundaries while preserving the original source order and recording `split_from_chunk_id`, `split_part`, and `split_part_count`.
 5. When retuning chunk size or repairing split logic, merge any existing `split_from_chunk_id` groups back to the original paragraph text first, then split again. Do not repeatedly split already-split parts, and do not discard valid reviewed chunks unless validation proves they no longer match the manifest.
 6. Write resumable per-chunk JSON artifacts. Never overwrite a valid reviewed chunk unless a validator or prompt version requires regeneration; move stale chunks out of the compile path.
-7. Generate or repair annotations with a provider worker loop, not a monolithic prompt. Each chunk should validate independently before promotion. If JSON is malformed or validation fails, retry the chunk with the exact validator errors before marking it failed.
-8. Compile preview PDFs periodically and at the end. For paired-language books, compile both directions when renderers exist, plus color and blackwhite variants when color is supported.
-9. Run the writer and monitor in observable tmux sessions with status files, logs, retry/backoff for provider limits, and clear resume commands.
+7. Generate or repair annotations with a provider worker loop, not a monolithic prompt. Each chunk should validate independently before promotion. If JSON is malformed or validation fails, retry the chunk with the exact validator errors before marking it failed. Use the structured JSON workflow for repetitive chunk output.
+8. Keep writer, reviewer, repairer, monitor, merge, and compile roles explicit. Writers produce candidates; validators promote; reviewers check semantic quality and request fixes; repairers handle failed-only or quarantined chunks; monitors observe and resume gently.
+9. Compile preview PDFs periodically and at the end. For paired-language books, compile both directions when renderers exist, plus color and blackwhite variants when color is supported.
+10. Run long jobs in observable tmux sessions with status files, logs, retry/backoff for provider limits, and clear resume commands.
 
 ## JSON Quality Gates
 

package/skills/self-healing-pipeline/SKILL.md

@@ -0,0 +1,73 @@
+---
+id: self-healing-pipeline
+label: Self-Healing Pipeline
+description: Diagnose stalled long-running writer, reviewer, monitor, queue, ETL, build, and generation pipelines; patch project-owned scripts; verify; and resume without discarding valid work.
+triggers:
+  - self healing
+  - autorepair
+  - auto repair
+  - stuck pipeline
+  - stalled pipeline
+  - writer monitor
+  - reviewer monitor
+  - queue worker
+  - long running pipeline
+  - retry failed
+  - resume failed
+tools:
+  - inspect_project
+  - read_file
+  - search_files
+  - apply_patch
+  - run_command
+  - tmux_list_sessions
+  - tmux_capture_pane
+  - tmux_send_keys
+  - tmux_start_session
+---
+# Self-Healing Pipeline
+
+Use this skill when a project has a durable worker, writer, reviewer, monitor, queue, batch generator, ETL, build loop, or tmux job that must keep moving after malformed output, provider limits, crashes, bad chunks, stale locks, or compile failures.
+
+## Diagnose Before Repair
+
+1. Read project instructions, runner scripts, status files, manifests, logs, and current tmux panes.
+2. Classify the symptom as one of: healthy wait, provider/rate-limit wait, validation failure, deterministic data/schema failure, script crash, stale lock/claim, compile/render failure, missing dependency, or monitor failure.
+3. Compare progress counters across two observations before declaring a stall unless the logs show a hard error.
+4. Preserve valid outputs. Never delete reviewed artifacts, manifests, checkpoints, or source inputs unless validation proves they are stale and the project has a quarantine path.
+
+## Repair Pattern
+
+- Patch project-owned scripts, prompts, validators, or monitors only after the logs identify a repeatable failure.
+- Prefer small resumability upgrades: `--failed-only`, bounded retry passes, stale-claim cleanup, atomic writes, checkpoint status, idempotent compile commands, and clear resume commands.
+- Keep writer/reviewer/monitor responsibilities separate. The writer should produce and validate; the reviewer should repair quality; the monitor should observe, compile, restart, or queue the next bounded run.
+- Make monitors gentle: wait on healthy progress, restart only after explicit stop/stall/error evidence, and write a durable decision log.
+- If parallel workers exist, require disjoint output paths or claim files, atomic promotion, and merge-stage validation before compiling.
+- A reviewer is not just a promoter. It should inspect valid-looking outputs for source drift, missing units, repeated filler, malformed annotations, and known quality failures, then write candidate fixes or failed-only repair requests.
+- A companion repairer should be independent from the main writer process. It can stay dormant, wake from status files or monitor decisions, run bounded repair passes, and exit without blocking healthy writer progress.
+- If the same failure repeats after a local nudge, improve the runner, validator, prompt, or status model rather than relying on manual chat intervention.
+
+## Parallel And Async Options
+
+Parallel and async designs are optional implementation patterns, not a default preference. Use them when the user asks for concurrency, the existing project already has a parallel pipeline, or the evidence shows a sequential bottleneck that can be partitioned safely.
+
+- Sequential processing is often the safest default for small jobs, fragile prompts, scarce quota, or unclear ownership boundaries.
+- If using sharded writer/fetcher workers for independent JSON/data chunks, give each worker a deterministic shard, separate log file, and no compile responsibility.
+- An async reviewer/promoter loop can validate and promote completed candidate files while writers continue, but it must use atomic writes, locks, or merge directories so it cannot race with active writers.
+- Keep compilation, publishing, and git commits out of parallel workers unless the project already has a safe, serialized mechanism for those steps.
+- If a worker stalls on one bad chunk, it should mark the chunk failed and continue its shard. Failed-only repair passes should be bounded and observable.
+- When increasing concurrency, check provider quota/rate-limit behavior. If rate limits appear, reduce worker count or add backoff rather than letting every worker retry aggressively.
+
+## Verification And Resume
+
+After a repair:
+
+1. Run syntax checks for changed scripts.
+2. Run a dry-run or small bounded batch when safe.
+3. Verify status counters, first missing item, failed IDs, and output timestamps.
+4. Restart only the affected tmux session, not unrelated jobs.
+5. Compile or export a checkpoint artifact when the pipeline has a renderer or build command.
+6. Record the exact resume command, current status, logs inspected, remaining failed items, and any quarantined artifacts.
+7. Commit reusable script/profile/prompt fixes when the project expects git tracking.
+
+The goal is not to hide failures. The goal is to keep the pipeline observable, resumable, and able to recover from known classes of failure without overwriting good work.

package/skills/source-ingestion/SKILL.md

@@ -0,0 +1,63 @@
+---
+id: source-ingestion
+label: Source Ingestion And OCR
+description: Inspect mixed input files and convert PDFs, EPUBs, images, scans, web/JSON sources, archives, or unknown documents into useful text, Markdown, manifests, or reviewable artifacts.
+triggers:
+  - source
+  - sources
+  - input file
+  - convert to markdown
+  - extract text
+  - pdf
+  - ocr
+  - scanned
+  - epub
+  - image text
+  - recognition
+  - markdown
+tools:
+  - inspect_project
+  - read_file
+  - write_file
+  - run_command
+  - web_search
+  - send_to_canvas
+---
+# Source Ingestion And OCR
+
+Use this skill when the user gives arbitrary source files or asks to read, recognize, OCR, convert, or prepare inputs for another workflow. The task is to make the files usable without assuming they are already text.
+
+## Operating Loop
+
+1. Inventory inputs first. List paths, extensions, sizes, likely language, and whether each file is original media or derived output. Keep original files untouched.
+2. Probe before converting:
+   - `file`, `stat`, `pdfinfo`, `pdftotext`, `exiftool` when available.
+   - EPUB: inspect the archive or use an existing EPUB-to-Markdown script/tool.
+   - PDF: try text-layer extraction first; if empty or garbage, mark as image-only and choose OCR.
+   - Images/scans: inspect dimensions/orientation and run OCR only after deciding language and page segmentation.
+   - JSON/HTML/wiki manifests: read the manifest, follow local `html`, `pdf`, `iiif`, or source fields, and prefer structured extraction over OCR.
+   - Archives/directories: expand or enumerate into a durable work folder only when needed.
+3. Choose the simplest reliable route. Prefer existing project scripts and installed tools before adding dependencies. Install only when policy allows and record the command.
+4. Write durable outputs near the project workflow, usually `books/<id>/sources/markdown/`, `ocr/`, `artifacts/`, or a user-specified path. Use descriptive names; do not overwrite reviewed outputs unless asked.
+5. Produce a manifest for nontrivial ingestion. Include source path, sha256, extraction method, status (`complete`, `requires_ocr`, `failed`, `pending`), language, page/chapter counts, output path, and caveats.
+6. Validate the result externally: line/character counts, heading counts, boilerplate/debris checks, sample excerpts, and a no-text-layer check for PDFs marked `requires_ocr`.
+
+## OCR And Recognition Strategy
+
+- Treat OCR as a pipeline, not a guess: render pages, crop/deskew if needed, choose language (`chi_sim`, `chi_tra`, `jpn`, `jpn_vert`, `eng`, or combinations), test a small page range, then scale up.
+- For large scans, create resumable page-level outputs and a manifest before full OCR. Do not run a fragile all-pages command without logs and resume paths.
+- For vertical Japanese or classical Chinese scans, expect tool tuning. If OCR quality is poor, save page images and report that manual/model-assisted correction is required rather than fabricating text.
+- Preserve page references in OCR Markdown (`## Page N`) until the text is reviewed; only collapse into chapters after quality checks.
+
+## Expected Outputs
+
+When the downstream task expects Markdown, JSON, or a source bundle, create exactly that shape plus a report:
+
+- Markdown: clean body text with stable headings and paragraphs.
+- JSON: schema-valid and validated, not prose pretending to be JSON.
+- Source bundle: roles, source hashes, extraction status, and activation rules.
+- Report: what was converted, what was not, tools used, validation commands, and residual risks.
+
+## Safety Rules
+
+Do not claim a scanned PDF or image was converted if the output is empty, repeated headers, mojibake, or OCR garbage. Mark it `requires_ocr` or `failed` with evidence. Do not silently delete real content while removing boilerplate. Do not commit original large source media unless the repository already tracks that category and the user asked for it.

package/skills/structured-json/SKILL.md

@@ -1,6 +1,22 @@
 ---
-name: structured-json
-description: Use isolated schema-bound JSON generation for extraction, annotation, conversion, classification, or chunked data production without mixing in agent runtime context.
+id: structured-json
+label: Structured JSON Generation
+description: Use isolated schema-bound JSON generation for extraction, annotation, conversion, classification, repair, or chunked data production without mixing in agent runtime context.
+triggers:
+  - structured json
+  - json schema
+  - schema-bound
+  - json specialist
+  - json fetcher
+  - parallel json
+  - repair json
+  - valid json
+tools:
+  - json_specialist
+  - json_specialist_batch
+  - read_file
+  - write_file
+  - run_command
 ---
 
 # Structured JSON
@@ -15,8 +31,17 @@ Use this skill when the user needs reliable JSON that follows an explicit schema
 4. Keep formatting, file writes, validation scripts, compilation, and project-specific orchestration in the main agent.
 5. After the tool returns, validate any project-specific invariants with local scripts before treating the data as complete.
 
+## Provider Strategy
+
+- Prefer provider-native structured output when available, such as JSON Schema or JSON object mode.
+- Keep a fallback parser/repair path for providers that do not support strict schema responses.
+- On validation failure, retry with the exact schema errors and only the smallest relevant source text.
+- For batch work, write candidate JSON per chunk first; promote it only after schema and semantic validators pass.
+- Keep schema versions in the artifact metadata so old reviewed outputs can be reused or selectively regenerated when prompts change.
+- Before retrying the provider, classify whether the failure is semantic or mechanical. Use local canonicalization for deterministic fixes such as token splitting, punctuation restoration from source text, missing default fields, role aliases, or renderer wrappers.
+
 ## Boundaries
 
 - Do not pass shell, browser, file policy, package-install, or agent-planning context into the JSON specialist.
 - Do not make schemas book-, app-, or project-specific inside AgInTiFlow core. Project schemas belong in the target repository.
-- Prefer provider-native structured output when available, but keep fallback parsing enabled unless the user explicitly wants hard failure on unsupported response formats.
+- Do not let parallel JSON workers share one mutable output file. Use shard-local outputs, atomic renames, and a serialized merge/promote step.

package/src/task-profiles.js

@@ -132,6 +132,13 @@ export const TASK_PROFILES = {
       "Bias toward supervising another agent or long-running task instead of doing the target work directly. Define acceptance criteria, give the student agent normal user-level prompts, monitor progress through tmux/session logs/artifacts, independently verify claims, record evidence, and convert repeated failures into reusable AgInTiFlow skills, tools, policies, tests, or profile improvements.",
     tools: ["shell", "files", "canvas", "inspect_project", "tmux"],
   },
+  pipeline: {
+    id: "pipeline",
+    label: "Self-healing pipeline",
+    prompt:
+      "Bias toward keeping long-running writer, reviewer, repairer, monitor, queue, ETL, batch generation, and build pipelines observable, resumable, and moving from raw inputs to final artifacts. Diagnose from status files, logs, manifests, tmux panes, heartbeats, and timestamps before intervening. Distinguish healthy waiting from stalls. Preserve valid artifacts, quarantine stale or invalid outputs, and patch project-owned scripts/prompts/validators in small reversible steps only when evidence supports it. For new pipelines, create a local contract with source manifest, stable chunks, schemas, validators, runners, checkpoint builds, and completion evidence. Choose sequential, parallel, async, or review-gated designs according to the user's request and the project's existing architecture. Verify with focused checks, checkpoint artifacts, then restart only the affected session and record the exact resume command.",
+    tools: ["inspect_project", "search_files", "read_file", "apply_patch", "shell", "tmux", "sandbox"],
+  },
   app: {
     id: "app",
     label: "App builder",
@@ -391,6 +398,16 @@ const PROFILE_ALIASES = {
   curriculum: "supervision",
   selfsupervision: "supervision",
   "self-supervision": "supervision",
+  pipeline: "pipeline",
+  pipelines: "pipeline",
+  autorepair: "pipeline",
+  "auto-repair": "pipeline",
+  "self-healing": "pipeline",
+  stuck: "pipeline",
+  stalled: "pipeline",
+  monitor: "pipeline",
+  queue: "pipeline",
+  reviewer: "pipeline",
   cpp: "c-cpp",
   "c++": "c-cpp",
   clang: "c-cpp",
@@ -432,6 +449,7 @@ export function defaultMaxStepsForProfile(value = "auto") {
   if (profile === "android") return 60;
   if (profile === "latex") return 30;
   if (profile === "supervision") return 40;
+  if (profile === "pipeline") return 44;
   if (profile === "aaps") return 36;
   if (["devops", "security"].includes(profile)) return 36;
   if (