turnkit 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3db08f0653a468f22daa74070c7b1a290f02b44beaf6d35c950f3d36d86dc224
-  data.tar.gz: 18d3b2df593865ba53e0863d0e2b061f8744bdbbf78409b40d525c3ba47b5070
+  metadata.gz: d8ca23035231ccf58df8389aa2e845bb3e8874b1a23d33a8e8ffb8b793c533fe
+  data.tar.gz: 1f1bafaf7eb330caa119504846b9960989f116780b5ea364fdbe4e7c47869db2
 SHA512:
-  metadata.gz: dd2216c9e2275dd737d6e1cf7dadd97053ca5afb6992939791669f29e047f3e44eb45f492a09b4494222117428baf07ffd9aa60352f9363b0a8b3eedb9c6ab54
-  data.tar.gz: 3b8f1a45bc98506dc09a9a6703aacf1f614fb7649c3c64942c98e902aff1db9232f986aad33c077be51fec1623d82abf4506d32698b34e3079ff775b9a1eb38b
+  metadata.gz: dbe5569a73d9118776e207567a515642d9fc59923de34b5084499623f36798e9c826832bf35d7f8105b9272252edb6da244faf896f9509e59e571d9db7f9022f
+  data.tar.gz: 31b2decb3c2336eafd7f943eaff23cf149b88c67067b75c785f9cd71960349a9f89a20da5dfd1d415fb1f7e083f9e061514f359568c4a5b92fc44426a8e05fcd

data/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 0.6.0 - 2026-09-06
+
+### Added
+
+- Independent background submission through Active Job, registered agent reconstruction, durable messages/completion callbacks, inbox/outbox queries, and worker-free joins.
+- Parallel background sub-agents with persisted parent/child lineage and ordered results; opt-in launch, send-message, and wait tools.
+- Persisted execution phases, fenced claims, root-wide budget reservations, and recovery of missed enqueues and abandoned workers without replaying unknown external effects.
+- PostgreSQL concurrency/process-death tests and a reversible `turnkit:upgrade` migration generator.
+- Agent-scoped context and skills, extensible specialist factories, authorization hooks, and fenced cancellation with explicit descendant handling.
+- Live recipe, deep-research and rare-earth policy examples, plus prepared orb dependencies and OIDC-based RubyGems releases.
+
+### Fixed
+
+- Reject conflicting delivery-key reuse without disclosing another conversation's payload; authorize implicit joins and detect conversation-lane wait cycles.
+- Bound inline reconciliation queries and preserve replay-safe tool keys and skill activation across recovery.
+- Forward image references and normalize RubyLLM provider failures as terminal model errors instead of repeatedly retrying permanent failures.
+
+### Breaking
+
+- Existing ActiveRecord installations must run `turnkit:upgrade` and migrate; turn records gain `submitted_at` and `claim_token`, plus delivery and wait tables.
+- Custom stores must implement the new transactional coordination contract. Reconciled inline workers lose write authority; `stale` is no longer provisional.
+- Background execution requires Active Job 7.2+ and a persistent queue backend. `async: true` remains previewable pending work; call `perform_later` to submit.
+
 ## 0.5.0 - 2026-07-22
 
 ### Breaking

data/README.md

@@ -548,6 +548,21 @@ end
 
 Require an image before completion with `TurnKit::OutputPolicy.require_image`.
 
+For editing or style references, override `input_images(**arguments)` and
+`mask(**arguments)` in your `ImageTool` subclass, just like `prompt` and
+`metadata`. Both default to `nil` and are forwarded to the image provider.
+Declare the corresponding tool parameters yourself; accepted sources, reference
+limits, and mask support depend on the provider. Local paths must exist on the
+executing worker. Generate only when the application/user requested it; use
+`ViewMediaTool` for inspection, not image generation.
+
+Image results retain URL or base64 bytes, including in tool results. Provider
+URLs may expire: use the event hook above to copy images into application-owned
+storage for durable user-facing links. TurnKit does not host attachments or
+automatically make generated images visually available to the next text model
+call. Base64 tool results also consume prompt space; for large images, prefer a
+custom tool that stores the artifact and returns its application-owned URL.
+
 ### Media analysis
 
 Analyze existing images, PDFs, audio, or video inside a durable turn with
@@ -727,6 +742,161 @@ puts turn.output_text
 
 Use sub-agents for isolated child conversations.
 
+#### Oracle- and Librarian-style specialists
+
+Use ordinary agents, not a second specialist runtime. Give each specialist its
+own model, instructions, description (when the parent should invoke it), and
+explicit tool list, then include it in the parent's `sub_agents`:
+
+- **Oracle:** instructions for a specific unresolved decision, the evidence
+  already checked, constraints, and a recommendation. Supply only scoped
+  read/search tools; the parent owns implementation and verification.
+- **Librarian:** instructions for external repository understanding and a full
+  evidence-backed report. Supply authenticated repository read/search, history,
+  diff, and issue-reading tools appropriate to the application. A role name does
+  not provide GitHub access. Exclude repository and issue mutation tools.
+- **Painter:** use `ImageTool` for an actual image-provider call rather than a
+  text agent pretending to generate an image.
+
+Children receive the explicit `task`, not the parent's conversation history or
+tool list. Their complete final text, structured `output_data`, status/error,
+and conversation/turn IDs return through the parent's matching tool result;
+intermediate child messages remain in the child conversation. Inline calls
+block; background calls use the same durable fan-out/join machinery below.
+
+This is a context boundary, not an OS or credential sandbox. Global context
+contributors and custom clients can supply additional context/capabilities;
+scope those deliberately. Enforce read-only access in the supplied tools and
+credentials, not merely in prompts. Likewise, enforce user approval for image
+generation in application tool exposure/authorization when it is a requirement.
+TurnKit does not ship Amp's private tools, repository service, or attachment host.
+
+### Background execution and agent messaging
+
+Durable background execution uses Active Record/Active Job 7.2+, `ActiveRecordStore`, and a
+persistent backend such as Solid Queue or Sidekiq. Configure the backend in your
+application; TurnKit does not start worker processes. The database owns pending
+work, while jobs are wake-up signals. `MemoryStore` is for inline use and tests.
+
+Existing installations must run `bin/rails generate turnkit:upgrade` and migrate
+before using this version. New installations use `turnkit:install` as usual.
+
+Register agent definitions at application boot **in every web and worker
+process**. Registration also registers configured sub-agents. Names must identify
+the same agent configuration in every process; use Rails `to_prepare` when
+definitions are reloadable. TurnKit persists IDs, model, options, and lineage,
+not Ruby clients, tools, closures, or executable configuration.
+Subject `to_prompt` data is snapshotted in conversation metadata for worker
+reconstruction. Use live-context contributors with application-owned identifiers
+when background turns need freshly loaded domain objects rather than that snapshot.
+
+```ruby
+require "turnkit/job"
+
+writer = TurnKit::Agent.new(name: "writer", instructions: "Draft concise copy.")
+editor = TurnKit.register(TurnKit::Agent.new(
+  name: "editor",
+  sub_agents: [writer],
+  tools: [TurnKit::LaunchAgentTool, TurnKit::SendMessageTool, TurnKit::WaitTool]
+))
+
+run = editor.run("Research and draft the announcement.", async: true)
+run.perform_later
+
+# Another process can reconstruct records using the registered definitions.
+run = TurnKit::Run.new(TurnKit.load_turn(run.id))
+run.reload.status
+```
+
+`async: true` still means **prepare pending work**, not enqueue it. This preserves
+preview-only use. `perform_later` persists submission before enqueueing and
+returns immediately. Background work uses the application-wide `TurnKit.store`.
+Worker clients and tools must be safe for the concurrency configured in your job
+backend. Do not launch unjoined Ruby threads from tools.
+
+In a background turn, a group of consecutive sub-agent tool calls launches
+independent child jobs. The parent becomes `waiting` and releases its worker;
+once all children finish, it resumes with tool results in call order. A failed
+child returns a structured failure, not an implicit parent failure. Ordinary
+tools remain sequential, and terminal tools prevent later calls from running.
+Inline turns use the same engine but execute sub-agents synchronously.
+
+`LaunchAgentTool` starts a configured sub-agent without waiting. Its optional
+`callback: true` requests a completion message in the launching conversation.
+`WaitTool` joins submitted turns without holding a worker. These coordination
+tools are opt-in; the application owns authorization and should expose only
+the destinations/actions appropriate for that agent.
+
+Application code can send messages and request completion callbacks too:
+
+```ruby
+parent = TurnKit.load_conversation(parent_conversation_id)
+child = writer.run("Investigate the issue.", async: true)
+child.perform_later(callback: parent)
+
+parent.send_message(other_conversation_id, "Please check the latest results.",
+  key: "request-123:check-results")
+parent.inbox
+parent.outbox
+
+# Attach dependencies before enqueueing, avoiding a race with an eager worker.
+summary = editor.run("Summarize the joined results.", async: true)
+summary.wait_for(child).perform_later
+```
+
+Delivery keys are globally unique idempotency keys. Retrying the same key returns
+the original delivery; it does not replace its payload. Inbox, outbox, and
+completion callbacks use the same persisted delivery rows. Delivery appends one
+message and records the need for a continuation transactionally. Messages to a
+running or waiting conversation do not change its current input snapshot: they
+are consumed by a later turn. Background turns in one conversation execute
+serially. Use `TurnKit.load_conversation` to address a conversation independently
+of a worker's fenced execution context.
+
+Application-level `wait_for` supplies joined results as turn-local input before
+the first model call. Attach waits only to pending turns. Wait targets must be
+submitted or finished, and must not form dependency cycles. Waiting for unfinished
+work in the same conversation is rejected because execution there is serial.
+
+#### Recovery and operational requirements
+
+Schedule `TurnKit::ReconcileJob` using your backend's recurring-job facility
+(for example every minute). This repairs missed enqueues, undelivered messages,
+ready joins, and abandoned workers. `TurnKit.reconcile_stale!` also recovers
+submitted work while retaining the inline stale-turn behavior described below.
+
+- Claims are fenced: after revocation or completion, an old execution cannot
+  persist another model response, tool result, heartbeat, or completion.
+- Recovery reuses committed model responses and tool results. An interrupted
+  model request may be reissued and charged again by the provider.
+- Started external tool calls without results become `interrupted`, with unknown
+  outcomes. They are **not replayed**. The built-in durable coordination tools can
+  resume using their existing child IDs/delivery keys.
+- Waiting is not a stale heartbeat. Root timeout still applies, including queue
+  and wait time from initial submission. It is a cooperative limit, not a hard
+  process kill. Use provider/tool timeouts for blocking external calls.
+- Root iteration and tool-count limits are reserved under a database lock across
+  parallel children. Spend limits use observed cost; already-running requests can
+  overshoot them.
+- `on_event` remains an inline observation hook, not a durable callback. Background
+  infrastructure exceptions reach the job backend rather than being converted
+  into success. Monitor failed jobs and run reconciliation.
+- TurnKit fences its own persistence, not arbitrary external side effects. Tools
+  should use `context.turn.store` for execution-owned TurnKit writes. External
+  services need their own idempotency keys where appropriate.
+
+The PostgreSQL integration suite exercises real row locks and worker-process
+death. Run it against a dedicated test database:
+
+```sh
+TURNKIT_TEST_DATABASE_URL=postgresql:///turnkit_test bundle exec rake test
+```
+
+It creates/rebuilds only `turnkit_test_*` and `turnkit_upgrade_test_*` test tables.
+Without the variable, database tests explicitly skip. Custom stores must implement
+the transactional `Store` contract, including reentrant `atomic`, deliveries,
+waits, and submitted-turn queries; a mutex alone is not cross-process durability.
+
 ### Context Compaction
 
 Disable compaction:
@@ -791,7 +961,8 @@ process). Run this periodically:
 TurnKit.reconcile_stale!
 ```
 
-Reconciliation atomically marks pending and running turns whose last heartbeat
+For unsubmitted inline work, reconciliation atomically marks pending and running
+turns whose last heartbeat
 is older than `TurnKit.timeout` as `stale`, so it never overwrites a turn that
 was concurrently claimed, heartbeated, or completed. Each stale turn's
 unfinished tool executions become `interrupted`, and a synthetic error tool
@@ -799,10 +970,10 @@ result is appended for any unresolved tool call so the conversation can be
 continued. TurnKit never reruns an interrupted tool — whether its side effect
 happened is unknown, so the continued model is told not to assume either way.
 
-Reconciliation does not cancel or reassign the underlying process. If the
-original worker is still alive (its heartbeats were merely late), it continues
-over the synthetic interrupted result and later replaces `stale` with its
-actual `completed` or `failed` outcome — treat `turn.stale` as provisional.
+Reconciliation revokes the original worker's commit authority. A late worker
+cannot replace `stale` with its own outcome. It does not kill the underlying
+process or undo external side effects. Continue an inline stale conversation
+with a new turn; submitted background work is resumed automatically.
 
 ## Options
 
@@ -877,6 +1048,11 @@ agent = TurnKit::Agent.new(
 
 ## Upgrading
 
+See [runtime hardening](docs/runtime-hardening.md) for scoped context, authorization,
+cancellation, cycle-safe waits, replay-safe effects and bounded maintenance, and
+[specialists and skills](docs/specialists.md) for the extensible Oracle, Librarian
+and Painter factories and skill-owned tools.
+
 See the [0.4.2 Upgrade Guide](UPGRADE_TO_0_4_2.md) for the full API migration checklist.
 
 Rails installs from older versions may need `output_data` for structured output,
@@ -906,6 +1082,12 @@ find lib test examples -type f -name '*.rb' -print0 | xargs -0 ruby -c
 
 Open a pull request.
 
+## Releasing
+
+Maintainers: follow [the release guide](docs/releasing.md) for RubyGems trusted
+publishing setup, version and lockfile updates, and the explicit tag-push procedure.
+Pushing a `v*` tag triggers tests and publication through the `release` environment.
+
 ## License
 
 Use this gem under the MIT License.

data/UPGRADE.md

@@ -1,5 +1,62 @@
 # Upgrade Guide
 
+## 0.6.0: Runtime hardening
+
+Configure global context/skills at boot, before constructing agents; defaults are now
+snapshotted. Use agent-scoped `context_contributors:` and run-scoped `context:` for requests.
+Specialist factories disable global inheritance and validate read-only skill tools.
+Skill-owned tools activate only after successful loading in the current turn.
+
+Configure `TurnKit.authorization_policy` and pass authenticated `principal:` values for
+multi-user applications. The default remains trusted application code. Submitted cancellation
+is terminal and fenced, with explicit `descendants: :retain` or `:cascade` semantics.
+
+PostgreSQL is required for ActiveRecord wait-graph locking. Custom stores must implement
+atomic graph operations and bounded actionable queries. Earlier durable-schema adopters
+need the two indexes in [runtime hardening](docs/runtime-hardening.md); new install/upgrade
+migrations already contain them. That guide also documents stable tool idempotency keys,
+opt-in replay safety and the limits of external-effect recovery.
+
+Delivery key reuse with different routing or payload now raises `ToolError` instead of
+returning the original message. Implicit subagent joins honor `:wait` policy. Cycle
+checks conservatively include all unfinished turns in a conversation's serial lane.
+Custom stores also need `list_stale_inline_turns(before:, limit:)`; public inline
+reconciliation now drains bounded batches rather than all history in one call.
+
+## 0.6.0: Durable background execution
+
+Stop existing workers before upgrading: older workers do not respect claim
+fencing. Generate and apply the additive migration, then restart the application
+and workers together:
+
+```sh
+bin/rails generate turnkit:upgrade
+bin/rails db:migrate
+```
+
+Use `--table-prefix your_prefix` if your installation uses custom table names.
+The generator adds delivery/wait models and a reversible migration; it does not
+replace existing models or delete conversation history. For custom model names,
+pass `delivery_class:` and `wait_class:` to `ActiveRecordStore` alongside the
+existing model-class options.
+
+Background execution requires Active Job 7.2+ with a persistent queue adapter,
+agent registration at boot in every worker, and a recurring
+`TurnKit::ReconcileJob`. See [background execution](README.md#background-execution-and-agent-messaging)
+for examples and operational semantics. `async: true` does not enqueue work.
+
+Custom stores need reentrant, rollback-capable `atomic(conversation_id)` with
+cross-process locking for durable use, submitted-turn queries, delivery CRUD
+with a unique idempotency key, and idempotent wait relations. Turn statuses now
+include `waiting`; turn records carry `submitted_at` and `claim_token`.
+The base store implements inline stale reconciliation using these primitives.
+
+Do not rely on a late worker overwriting a reconciled stale turn. Its claim is
+now revoked. Execution-owned writes must go through `context.turn.store`, not a
+global/raw store. Background limits are rooted in persisted run configuration;
+timeout includes queue and wait time after submission. Inline use needs no job
+backend, but ActiveRecord schemas and custom store contracts still need updating.
+
 ## 0.4.2
 
 TurnKit 0.4.2 is a pre-1.0 surface cleanup. Update these call sites before upgrading:

data/lib/generators/turnkit/install/templates/create_turnkit_tables.rb

@@ -31,6 +31,8 @@ class CreateTurnkitTables < ActiveRecord::Migration[7.1]
       t.json :error
       t.text :output_text
       t.json :output_data
+      t.datetime :submitted_at
+      t.string :claim_token
       t.datetime :started_at
       t.datetime :heartbeat_at
       t.datetime :completed_at
@@ -40,6 +42,34 @@ class CreateTurnkitTables < ActiveRecord::Migration[7.1]
       t.index :conversation_uid
       t.index :root_turn_uid
       t.index [ :status, :heartbeat_at ]
+      t.index [ :status, :submitted_at, :updated_at ], name: "index_<%= table_prefix %>_turns_on_maintenance"
+    end
+
+    create_table :<%= table_prefix %>_deliveries do |t|
+      t.string :uid, null: false
+      t.string :source_conversation_uid, null: false
+      t.string :destination_conversation_uid, null: false
+      t.string :source_turn_uid
+      t.string :key, null: false
+      t.json :payload, null: false, default: {}
+      t.string :message_uid
+      t.datetime :delivered_at
+      t.timestamps
+
+      t.index :uid, unique: true
+      t.index :key, unique: true
+      t.index [ :source_conversation_uid, :created_at ]
+      t.index [ :destination_conversation_uid, :delivered_at ]
+      t.index [ :delivered_at, :created_at ], name: "index_<%= table_prefix %>_deliveries_on_pending"
+    end
+
+    create_table :<%= table_prefix %>_waits do |t|
+      t.string :turn_uid, null: false
+      t.string :target_turn_uid, null: false
+      t.timestamps
+
+      t.index [ :turn_uid, :target_turn_uid ], unique: true
+      t.index :target_turn_uid
     end
 
     create_table :<%= table_prefix %>_messages do |t|

data/lib/generators/turnkit/install/templates/delivery.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Turnkit
+  class Delivery < ApplicationRecord
+    self.table_name = "<%= table_prefix %>_deliveries"
+  end
+end

data/lib/generators/turnkit/install/templates/initializer.rb

@@ -8,6 +8,11 @@
 #   TurnKit::ActiveRecordStore.new(conversation_class: "My::Conversation", ...)
 TurnKit.store = TurnKit::ActiveRecordStore.new
 
+# Background work: require "turnkit/job", configure a persistent Active Job
+# backend, register agents with TurnKit.register at boot in every process,
+# and schedule TurnKit::ReconcileJob in your recurring-job facility.
+# async: true only prepares a run; perform_later submits it.
+
 # TurnKit.default_model = "claude-sonnet-4-5"
 # TurnKit.max_iterations = 25
 # TurnKit.timeout = 300

data/lib/generators/turnkit/install/templates/wait.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Turnkit
+  class Wait < ApplicationRecord
+    self.table_name = "<%= table_prefix %>_waits"
+  end
+end

data/lib/generators/turnkit/install_generator.rb

@@ -23,6 +23,8 @@ module TurnKit
         template "turn.rb", "app/models/turnkit/turn.rb"
         template "message.rb", "app/models/turnkit/message.rb"
         template "tool_execution.rb", "app/models/turnkit/tool_execution.rb"
+        template "delivery.rb", "app/models/turnkit/delivery.rb"
+        template "wait.rb", "app/models/turnkit/wait.rb"
       end
 
       def create_ai_directories

data/lib/generators/turnkit/upgrade/templates/add_turnkit_durable_orchestration.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+class AddTurnkitDurableOrchestration < ActiveRecord::Migration[7.1]
+  def change
+    add_column :<%= table_prefix %>_turns, :submitted_at, :datetime
+    add_column :<%= table_prefix %>_turns, :claim_token, :string
+    add_index :<%= table_prefix %>_turns, [ :status, :submitted_at, :updated_at ], name: "index_<%= table_prefix %>_turns_on_maintenance"
+
+    create_table :<%= table_prefix %>_deliveries do |t|
+      t.string :uid, null: false
+      t.string :source_conversation_uid, null: false
+      t.string :destination_conversation_uid, null: false
+      t.string :source_turn_uid
+      t.string :key, null: false
+      t.json :payload, null: false, default: {}
+      t.string :message_uid
+      t.datetime :delivered_at
+      t.timestamps
+
+      t.index :uid, unique: true
+      t.index :key, unique: true
+      t.index [ :source_conversation_uid, :created_at ]
+      t.index [ :destination_conversation_uid, :delivered_at ]
+      t.index [ :delivered_at, :created_at ], name: "index_<%= table_prefix %>_deliveries_on_pending"
+    end
+
+    create_table :<%= table_prefix %>_waits do |t|
+      t.string :turn_uid, null: false
+      t.string :target_turn_uid, null: false
+      t.timestamps
+
+      t.index [ :turn_uid, :target_turn_uid ], unique: true
+      t.index :target_turn_uid
+    end
+  end
+end

data/lib/generators/turnkit/upgrade_generator.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/migration"
+
+module TurnKit
+  module Generators
+    class UpgradeGenerator < Rails::Generators::Base
+      include Rails::Generators::Migration
+
+      namespace "turnkit:upgrade"
+      source_root File.expand_path("upgrade/templates", __dir__)
+      source_paths << File.expand_path("install/templates", __dir__)
+
+      class_option :table_prefix, type: :string, default: "turnkit", desc: "Database table prefix."
+
+      def copy_models
+        template "delivery.rb", "app/models/turnkit/delivery.rb"
+        template "wait.rb", "app/models/turnkit/wait.rb"
+      end
+
+      def copy_migration
+        migration_template "add_turnkit_durable_orchestration.rb", "db/migrate/add_turnkit_durable_orchestration.rb"
+      end
+
+      def self.next_migration_number(dirname)
+        Time.now.utc.strftime("%Y%m%d%H%M%S")
+      end
+
+      private
+        def table_prefix = options[:table_prefix]
+    end
+  end
+end