ask-tools-shell 0.3.4 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6c8152ae1dcc82318421fde8c5a890ada7a6d4f11435a6e642e0724a06bab80e
-  data.tar.gz: ce7bb3ea38eb7514ae0f4e44e9cd4adfc1ebed5cf413cc9b424b74e17ce183a8
+  metadata.gz: 8805be0a64d1a42f4b0eae3600c81ce0e61565fa19aaf1fd835d36fbb6fa9b58
+  data.tar.gz: fe497b0c0604cc2c9a90cd02bc17c31ff8f46140ac88a67e9697b5b9cd15f7ee
 SHA512:
-  metadata.gz: 284a6529ddae8a8099ab8d5af6d53fa2e3f53d97780c6ed9b94a26cb8eb6773500c12b1efb9f4087f3f415dc85b91b8724ecad2483ce511ca5c7e863d0dd00c8
-  data.tar.gz: a1d656f3851f88949869dc9f4aa1591f79c2ae394fa065f4e8767a55af4c004f63f84df06ef88f11bbfec27c5919b7374cc278f8eb97245f93c02799143b8e6e
+  metadata.gz: defeb70d692dc1da34ad21300ed52dff8496df4fd28c3365b7724d9e3a6c113afbdd7816619cfa0f9017479d9af1a061fb0ef82c140110f0e46209df8829bdc6
+  data.tar.gz: e855c29b7b8ed782dab2fccb0764c3eebe4b5b5b7af039aff713fc746c31b3b70f77b1a47abc250a54b217a3bacba4642bef97aa3f409e5f52ee80043a8f0610

data/CHANGELOG.md

@@ -1,3 +1,63 @@
+## [0.5.0] - 2026-08-10
+
+### Added
+- **Read tool engineering pass** — reads are the bill for building context,
+  so every decision inside `Ask::Tools::Read` is a token-budget decision:
+  - **Three ceilings, not one**: the 2000-line window (existing), a byte
+    budget (128 KB of output by default, `ASK_TOOLS_SHELL_READ_BYTE_BUDGET`
+    to override) for wide files, and a per-line clamp (2000 chars) for
+    minified bundles. Truncated reads return ok with a **precomputed resume
+    offset** — no pagination arithmetic for the model.
+  - **Named recovery, facts not errors**: empty files, past-EOF offsets,
+    binary files (mime note, never garbage bytes), and PDFs (pdftotext hint)
+    all return ok with a one-line answer instead of an error.
+  - **Streaming reads**: `File.foreach` with an early break at the budget —
+    a 400 MB log costs one read, not one load. "Is there more file?" is only
+    answered when it can be (peek, never guess).
+  - **Strict input repair**: `offset`/`limit` accept `"2000"` and `2.0` but
+    reject `"2abc"` and `1.5` instead of silently reading the wrong window.
+  - **Device blocklist**: `/dev/zero`, `/dev/urandom`, `/dev/stdin`,
+    `/dev/fd/*`, `/proc/*/fd/*` refused by name before any I/O — a read can
+    never hang on them.
+  - **Filename repair**: NFD/NFC, narrow NBSP, and curly-quote variants are
+    retried for the model; then "did you mean?" (substring + bounded
+    Levenshtein ≤ 2, catches `AGENT.md` → `AGENTS.md`).
+  - **Self-expiring dedup**: re-reading the same unchanged (path, mtime,
+    size, offset, limit) window returns a one-line "already in context" stub
+    — consumed on use, complete reads only, kill-switch
+    `ASK_TOOLS_SHELL_READ_NO_CACHE=1`.
+  - **Partial-view ledger** (`Ask::Tools::Shell::FileLedger`): Read records
+    what it showed; **Write refuses to overwrite a partially-read unchanged
+    file** ("re-read the full file first"), and Edit records a full read so
+    the invariant can't deadlock. Ledger entries auto-invalidate when the
+    file's mtime/size change.
+  - **Hygiene**: BOM stripped, CRLF → LF, invalid UTF-8 replaced instead of
+    raising (a read never crashes on bytes).
+
+## [0.4.0] - 2026-08-05
+
+### Added
+- **`Ask::Tools::Repl`** — evaluate Ruby code in a persistent session (the
+  RLM / recursive language model pattern). A long-lived plain-ruby kernel
+  subprocess keeps state across calls: locals, `require`s, and defined
+  methods survive between evaluations, so the model composes capabilities as
+  code against a working environment instead of re-bootstrapping each time.
+  - Framed newline-delimited JSON protocol over stdin/stdout with
+    request/response id matching; concurrent calls to a session serialize.
+  - Per-evaluation timeout kills the session (state is lost, kernel
+    respawns fresh on next call); idle sessions recycle after
+    `Repl.idle_timeout` (default 300s).
+  - Named sessions shared process-wide (`session:` param, default
+    `"default"`); `reset: true` discards state; `Repl.close_session` /
+    `Repl.close_all` manage lifetimes; `at_exit` cleanup.
+  - Sessions are isolated subprocesses — a crash in one session can't take
+    others down, and a dead session is respawned transparently with one
+    retry.
+  - Kernel spawn strips bundler env vars (RUBYOPT, GEM_HOME, etc.) so the
+    session is plain ruby and sees globally installed gems — consistent
+    with the one-shot `Code` tool.
+- Registered `repl` in `Shell::TOOLS` / `Shell.all`.
+
 ## [0.3.4] - 2026-06-25
 
 ### Fixed

data/README.md

@@ -1,171 +1,72 @@
 # ask-tools-shell
 
-Shell, filesystem, and code execution tools for AI agents. Part of the ask-rb ecosystem.
+[![Gem Version](https://badge.fury.io/rb/ask-tools-shell.svg)](https://badge.fury.io/rb/ask-tools-shell)
 
-Provides **Bash**, **Read**, **Write**, **Edit**, **Glob**, **Grep**, and **Code** — the execution tools every agent needs.
+Shell, filesystem, and code execution tools for AI agents. Ships 9 tools: Bash, Read, Write, Edit, Glob, Grep, Code, Repl, and ApplyPatch. Bash and Code execute through ask-sandbox-providers; Repl runs a persistent plain-ruby kernel; the rest operate directly on the local filesystem.
+
+## Installation
 
 ```ruby
 gem "ask-tools-shell"
 ```
 
-## Dependencies
-
-- **ask-tools** ~> 0.1 (provides `Ask::Tool` base class and `Ask::Result`)
-
----
-
 ## Quick Start
 
 ```ruby
 require "ask-tools-shell"
 
-# List all available tools
 Ask::Tools::Shell.all.map(&:name)
-# => ["bash", "read", "write", "edit", "glob", "grep", "code"]
+# => ["bash", "read", "write", "edit", "glob", "grep", "code", "repl", "apply_patch"]
 
-# Use a tool standalone
 result = Ask::Tools::Bash.new.call(command: "echo hello")
-result.ok?              # => true
-result.output[:stdout]  # => "hello\n"
-result.output[:exit_code] # => 0
+result.ok?                 # => true
+result.output[:stdout]     # => "hello\n"
+result.output[:exit_code]  # => 0
 ```
 
----
+## The tools
 
-## Tools
+| Tool | Parameters | Notes |
+|---|---|---|
+| `Ask::Tools::Bash` | `command`, `timeout` (30), `workdir` | Runs via `Ask::Sandbox.provider`; returns `{ stdout, stderr, exit_code, timed_out }`, output truncated to 100KB |
+| `Ask::Tools::Read` | `path`, `offset` (0-indexed), `limit` (2000) | Reads files with line numbers, or lists a directory |
+| `Ask::Tools::Write` | `path`, `content` | Creates parent directories automatically |
+| `Ask::Tools::Edit` | `path`, `old_string`, `new_string`, `replace_all` | Exact string replacement |
+| `Ask::Tools::Glob` | `pattern`, `path` | Up to 1000 files, newest first |
+| `Ask::Tools::Grep` | `pattern`, `path`, `include` | Regex search; 100 matches max, skips `.git`, `node_modules`, `vendor`, `.bundle`, `tmp`, `log` |
+| `Ask::Tools::Code` | `code` | Runs Ruby via `Ask::Sandbox.provider`; returns `{ stdout, stderr, exit_code }` |
+| `Ask::Tools::Repl` | `code`, `session`, `reset` | Evaluates Ruby in a persistent session — state (variables, requires, methods) survives across calls; timeouts kill the session and respawn fresh |
+| `Ask::Tools::ApplyPatch` | `patchText` | Applies unified diffs inside a `*** Begin Patch` / `*** End Patch` envelope (Add File, Update File, Delete File sections) |
 
-### `Ask::Tools::Bash`
+## Sandboxed execution
 
-Execute shell commands in a sandboxed temp directory.
-
-| Param | Type | Required | Default | Description |
-|-------|------|----------|---------|-------------|
-| `command` | `string` | Yes | — | The bash command to execute |
-| `timeout` | `integer` | No | 30 | Timeout in seconds |
-| `workdir` | `string` | No | temp dir | Working directory |
-
-Returns `{ stdout, stderr, exit_code, timed_out }`. Output truncated to 100KB. Process killed on timeout.
+`Bash` and `Code` run through `Ask::Sandbox.provider` (ask-sandbox-providers), which defaults to the Local provider. Switch to stronger isolation:
 
 ```ruby
-Ask::Tools::Bash.new.call(command: "ls -la", timeout: 10)
+Ask::Sandbox.provider = :docker
 ```
 
-### `Ask::Tools::Read`
+`Repl` is a durable control environment (a persistent subprocess that must
+keep state) and is not sandboxed — don't point it at untrusted code.
 
-Read file contents with line numbers, or list directory entries.
+### Code vs Repl
 
-| Param | Type | Required | Default | Description |
-|-------|------|----------|---------|-------------|
-| `path` | `string` | Yes | — | Absolute path to file or directory |
-| `offset` | `integer` | No | 0 | Starting line number (0-indexed) |
-| `limit` | `integer` | No | 2000 | Maximum lines to read |
+`Code` runs one Ruby snippet in a sandboxed subprocess and forgets it.
+`Repl` keeps a session alive so variables and methods survive across calls.
 
-```ruby
-Ask::Tools::Read.new.call(path: "/etc/hosts")
-Ask::Tools::Read.new.call(path: "large.log", offset: 100, limit: 50)
-```
-
-### `Ask::Tools::Write`
-
-Write content to a file. Creates parent directories automatically.
+Use `Code` for isolated one-off snippets and for code you don't trust (the
+sandbox is the safety boundary). Use `Repl` for multi-step work: load data
+and define helpers once, then keep working with them.
 
-| Param | Type | Required | Default | Description |
-|-------|------|----------|---------|-------------|
-| `path` | `string` | Yes | — | Absolute path to write to |
-| `content` | `string` | Yes | — | File content (max 500KB) |
-
-```ruby
-Ask::Tools::Write.new.call(path: "/tmp/hello.txt", content: "Hello, World!")
-```
+## Full documentation
 
-### `Ask::Tools::Edit`
-
-Replace exact text in a file. Uses exact string matching.
-
-| Param | Type | Required | Default | Description |
-|-------|------|----------|---------|-------------|
-| `path` | `string` | Yes | — | Absolute path to the file |
-| `old_string` | `string` | Yes | — | Exact text to replace |
-| `new_string` | `string` | Yes | — | Replacement text |
-| `replace_all` | `boolean` | No | false | Replace all occurrences |
-
-```ruby
-Ask::Tools::Edit.new.call(path: "file.rb", old_string: "foo", new_string: "bar")
-Ask::Tools::Edit.new.call(path: "file.rb", old_string: "x", new_string: "y", replace_all: true)
-```
-
-### `Ask::Tools::Glob`
-
-Find files matching a glob pattern, sorted by modification time (newest first).
-
-| Param | Type | Required | Default | Description |
-|-------|------|----------|---------|-------------|
-| `pattern` | `string` | Yes | — | Glob pattern (e.g. `**/*.rb`) |
-| `path` | `string` | No | current dir | Base directory |
-
-Max 1000 results.
-
-```ruby
-Ask::Tools::Glob.new.call(pattern: "**/*.rb", path: "/path/to/project")
-```
-
-### `Ask::Tools::Grep`
-
-Search file contents using a regex pattern.
-
-| Param | Type | Required | Default | Description |
-|-------|------|----------|---------|-------------|
-| `pattern` | `string` | Yes | — | Regex pattern to search for |
-| `path` | `string` | No | current dir | Directory to search |
-| `include` | `string` | No | `**/*` | File pattern filter (e.g. `*.rb`) |
-
-Max 100 matches. Line content capped at 500 chars. Skips `.git`, `node_modules`, `vendor`, `.bundle`, `tmp`, `log`.
-
-```ruby
-Ask::Tools::Grep.new.call(pattern: "TODO", path: ".", include: "*.rb")
-```
-
-### `Ask::Tools::Code`
-
-Write and execute Ruby code in a subprocess. Uses gems already available in the environment.
-
-| Param | Type | Required | Default | Description |
-|-------|------|----------|---------|-------------|
-| `code` | `string` | Yes | — | Ruby source code to execute |
-
-Returns `{ stdout, stderr, exit_code }`. Output truncated to 100KB.
-
-```ruby
-Ask::Tools::Code.new.call(code: <<~RUBY)
-  puts "Hello from Ruby!"
-  result = 2 + 2
-  puts "2 + 2 = #{result}"
-RUBY
-```
-
----
-
-## Using Tools with an Agent
-
-```ruby
-require "ask-tools-shell"
-
-# All tools
-tools = Ask::Tools::Shell.all
-
-# Find by name
-bash = Ask::Tools["bash"]
-bash.call(command: "date")
-```
-
----
+The full ask-rb documentation lives at https://ask-rb.github.io/ask-docs. [ask-tools in depth](https://ask-rb.github.io/ask-docs/core/tools) covers the shell tools, the ApplyPatch format, and sandbox configuration. API reference: https://ask-rb.github.io/ask-docs/reference/api.
 
 ## Development
 
-```bash
+```
 bundle install
 bundle exec rake test
-gem build ask-tools-shell.gemspec
 ```
 
 ## License

data/lib/ask/tools/shell/edit.rb

@@ -95,6 +95,10 @@ module Ask
         end
 
         raw = operations.read_file(path)
+        # Edit reads the whole file, so the model has seen all of it — record
+        # that before the write so a later Write isn't blocked by a stale
+        # partial view (and so the ledger reflects what was actually shown).
+        Shell::FileLedger.record(path, partial: false, lines_seen: [0, raw.count("\n") + 1])
         bom, content = Shell.strip_bom(raw)
         original_ending = Shell.detect_line_ending(content)
         content = Shell.normalize_line_endings(content)

data/lib/ask/tools/shell/file_ledger.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+require "monitor"
+
+module Ask
+  module Tools
+    module Shell
+      # Records what Read has shown of each file, so Write can refuse to
+      # destroy content the model never saw. Entries are validated against
+      # the file's current mtime/size — a changed file invalidates its
+      # entry, so a stale ledger can never block a write it shouldn't.
+      #
+      # Class-level on purpose: a partial view is a fact about the world,
+      # not about one tool instance (agent sessions new up fresh tool
+      # instances, and the invariant must survive across them).
+      class FileLedger
+        Entry = Struct.new(:path, :mtime, :size, :partial, :lines_seen, keyword_init: true)
+
+        @entries = {}
+        @mutex = Monitor.new
+
+        class << self
+          # Record what a read showed of a file.
+          # @param partial [Boolean] true when the view was clamped/truncated
+          # @param lines_seen [Array(Integer, Integer)] [start, stop) line
+          #   indices shown, 0-indexed
+          def record(path, partial:, lines_seen:)
+            path = File.expand_path(path)
+            @mutex.synchronize do
+              @entries[path] = Entry.new(
+                path: path,
+                mtime: File.mtime(path),
+                size: File.size(path),
+                partial: partial,
+                lines_seen: lines_seen
+              )
+            end
+          rescue Errno::ENOENT
+            nil # file vanished mid-read; nothing to record
+          end
+
+          # The entry for a path, or nil when the file changed since the read.
+          def entry_for(path)
+            path = File.expand_path(path)
+            @mutex.synchronize do
+              entry = @entries[path]
+              next nil unless entry
+
+              current = File.stat(path)
+              (current.mtime == entry.mtime && current.size == entry.size) ? entry : nil
+            end
+          rescue Errno::ENOENT
+            nil
+          end
+
+          # True when the file was only partially read and hasn't changed
+          # since. Write consults this before overwriting.
+          def partially_seen?(path)
+            entry = entry_for(path)
+            !entry.nil? && entry.partial
+          end
+
+          def reset!
+            @mutex.synchronize { @entries.clear }
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ask/tools/shell/grep.rb

@@ -34,7 +34,13 @@ module Ask
 
         Dir.glob(File.join(base, glob)).each do |file|
           next unless File.file?(file)
-          next if EXCLUDE_DIRS.any? { |d| file.include?("/#{d}/") }
+
+          # Exclude by path segment relative to the search root, not by
+          # absolute-path substring: the old check matched any path
+          # containing "/tmp/" (e.g. a search rooted under the system tmp
+          # dir on Linux), excluding every file.
+          relative = file.delete_prefix("#{base}/")
+          next if relative.split("/").any? { |seg| EXCLUDE_DIRS.include?(seg) }
 
           begin
             File.readlines(file).each_with_index do |line, i|

data/lib/ask/tools/shell/read.rb

@@ -5,64 +5,332 @@ require "fileutils"
 module Ask
   module Tools
     # Read file contents with line numbers, or list directory contents.
-    # Output is truncated to 2000 lines.
+    #
+    # Engineered for token budgets. Three ceilings stop the three shapes of
+    # hostile file — the long file (line window), the wide file (byte
+    # budget), the minified bundle (per-line clamp):
+    #
+    #   max_lines      = 2000 lines  — the window
+    #   byte_budget    = 128 KB      — chars of output returned
+    #   max_line_chars = 2000        — per-line clamp
+    #
+    # Truncation is a fact, not an error: reads that stop short return ok
+    # with a precomputed resume offset, so the model never does pagination
+    # arithmetic and never treats a fact about the world as a failure.
+    #
+    # The other decisions that make a read cheap instead of expensive:
+    #   - strict offset/limit repair (never silently mangle "2abc" into 2)
+    #   - device blocklist — /dev/zero would hang a read forever
+    #   - filename repair: NFD/NFC, narrow NBSP, curly quotes, did-you-mean
+    #   - a self-expiring dedup stub for unchanged re-reads (consumed on use,
+    #     complete reads only, kill-switchable)
+    #   - a partial-view ledger that Write consults before overwriting
     class Read < Ask::Tool
       description "Read the contents of a file or list a directory. " \
-                   "Files are displayed with line numbers. " \
-                   "Output is truncated to 2000 lines."
+                   "Files are displayed with line numbers. Output is bounded " \
+                   "to 2000 lines and 128 KB, with resume hints when truncated."
 
       param :path, type: :string, desc: "Absolute path to the file or directory", required: true
       param :offset, type: :integer, desc: "Starting line number (0-indexed)", required: false
       param :limit, type: :integer, desc: "Maximum number of lines to read", required: false
 
-      MAX_LINES = 2000
+      DEFAULT_MAX_LINES = 2000
+      DEFAULT_BYTE_BUDGET = 128_000
+      DEFAULT_MAX_LINE_CHARS = 2000
+
+      # Device files that never end or block forever — refused by name
+      # before any I/O, so a read can never hang on them.
+      DEVICE_PATHS = %w[
+        /dev/zero /dev/random /dev/urandom
+        /dev/stdin /dev/stdout /dev/stderr
+      ].freeze
+
+      MIME_TYPES = {
+        ".png" => "image/png", ".jpg" => "image/jpeg", ".jpeg" => "image/jpeg",
+        ".gif" => "image/gif", ".webp" => "image/webp", ".svg" => "image/svg+xml",
+        ".pdf" => "application/pdf", ".zip" => "application/zip",
+        ".gz" => "application/gzip", ".mp3" => "audio/mpeg", ".mp4" => "video/mp4"
+      }.freeze
+
+      attr_reader :max_lines, :byte_budget, :max_line_chars, :dedup_enabled
+      attr_writer :max_lines, :byte_budget, :max_line_chars
+
+      def initialize
+        super
+        @max_lines = DEFAULT_MAX_LINES
+        @byte_budget = (ENV["ASK_TOOLS_SHELL_READ_BYTE_BUDGET"] || DEFAULT_BYTE_BUDGET).to_i
+        @max_line_chars = DEFAULT_MAX_LINE_CHARS
+        @dedup_enabled = ENV["ASK_TOOLS_SHELL_READ_NO_CACHE"] != "1"
+        @dedup = {}
+      end
 
       def execute(path:, offset: nil, limit: nil)
         path = File.expand_path(path)
 
+        if device_path?(path)
+          return Ask::Result.error(message: "Refusing to read device file: #{path} (can block forever).")
+        end
+
         unless File.exist?(path)
-          return Ask::Result.error(message: "Path does not exist: #{path}")
+          return Ask::Result.error(message: missing_path_message(path))
         end
 
         if File.directory?(path)
-          entries = Dir.children(path).sort
-          entries.map! do |e|
-            full = File.join(path, e)
-            "#{e}#{File.directory?(full) ? '/' : ''}"
-          end
-          return Ask::Result.ok(data: entries.join("\n"), metadata: { type: "directory", count: entries.size })
+          return directory_listing(path)
         end
 
         unless File.file?(path)
           return Ask::Result.error(message: "Not a file: #{path}")
         end
 
-        if File.size(path) > 1_000_000
-          return Ask::Result.error(
-            message: "File too large (#{File.size(path)} bytes). Use offset/limit to read portions."
+        offset = coerce_int("offset", offset)
+        return Ask::Result.error(message: offset) if offset.is_a?(String)
+        limit = coerce_int("limit", limit)
+        return Ask::Result.error(message: limit) if limit.is_a?(String)
+
+        offset ||= 0
+        limit ||= @max_lines
+        return Ask::Result.error(message: "Invalid offset: #{offset} (must be >= 0).") if offset.negative?
+        return Ask::Result.error(message: "Invalid limit: #{limit} (must be >= 1).") if limit < 1
+
+        special = sniff(path)
+        return special if special
+
+        read = read_lines(path, offset, limit)
+        partial_view = read[:more] || read[:clamped].positive?
+
+        if @dedup_enabled && !partial_view && read[:lines].any?
+          key = [path, File.mtime(path).to_f, File.size(path), offset, limit]
+          if @dedup.key?(key)
+            @dedup.delete(key) # self-expiring: one stub, then real content again
+            return Ask::Result.ok(
+              data: "File unchanged since last read — content is already in context.",
+              metadata: { dedup: true }
+            )
+          end
+          @dedup[key] = true
+        end
+
+        Shell::FileLedger.record(path, partial: partial_view, lines_seen: [offset, offset + read[:lines].size])
+
+        data, resume_offset = format_output(path, offset, read)
+
+        metadata = {
+          total_lines: read[:more] || (read[:lines].empty? && read[:saw_any]) ? nil : offset + read[:lines].size,
+          start_line: read[:lines].empty? ? nil : offset + 1,
+          end_line: offset + read[:lines].size,
+          truncated: read[:more],
+          partial_view: partial_view,
+          clamped_lines: read[:clamped],
+          resume_offset: resume_offset
+        }
+        metadata.delete(:resume_offset) unless read[:more]
+        Ask::Result.ok(data: data, metadata: metadata)
+      end
+
+      private
+
+      # ── the three ceilings ─────────────────────────────────────────────
+
+      # Stream the file line by line (never load the whole thing), stopping
+      # at the line window, the byte budget, or EOF. Returns
+      # { lines:, more:, clamped:, mid_cut:, saw_any: }.
+      #
+      # "Is there more file?" is only answered when it can be: the window
+      # break proves it by having skipped a line; the budget break proves it
+      # by holding a line that didn't fit; at EOF there is none.
+      def read_lines(path, offset, limit)
+        selected = []
+        more = false
+        clamped = 0
+        mid_cut = false
+        output_len = 0
+        saw_any = false
+
+        File.foreach(path, chomp: true, invalid: :replace, undef: :replace).with_index do |line, i|
+          saw_any = true
+          next if i < offset
+
+          if selected.size >= limit
+            more = true
+            break
+          end
+
+          line = line.delete_suffix("\r") # CRLF → LF
+          line = Shell.strip_bom(line).last if i.zero? && offset.zero?
+
+          if line.length > @max_line_chars
+            line = line[0, @max_line_chars] + "…[clamped at #{@max_line_chars} chars]"
+            clamped += 1
+          end
+
+          numbered = "#{i + 1}: #{line}"
+          cost = numbered.length + 1
+          if output_len + cost > @byte_budget
+            if selected.empty?
+              # One line that outgrows the whole budget: show a slice rather
+              # than silence — silence is the most expensive thing a tool can
+              # return.
+              room = [@byte_budget - output_len - 2, 1].max
+              selected << "#{i + 1}: #{line[0, room]}…"
+              clamped += 1
+              mid_cut = true
+            end
+            more = true
+            break
+          end
+
+          output_len += cost
+          selected << numbered
+        end
+
+        { lines: selected, more: more, clamped: clamped, mid_cut: mid_cut, saw_any: saw_any }
+      end
+
+      # ── named recovery: facts, not errors ──────────────────────────────
+
+      def format_output(path, offset, read)
+        if read[:lines].empty?
+          return read[:saw_any] ?
+            ["Offset #{offset} is past the end of the file — retry with a smaller offset.", nil] :
+            ["File is empty (0 lines).", nil]
+        end
+
+        data = read[:lines].join("\n")
+        resume = nil
+        if read[:more]
+          # A mid-line cut resumes ON the line that was cut (it is the last
+          # line shown); every other truncation resumes on the next line.
+          resume = read[:mid_cut] ? offset + read[:lines].size - 1 : offset + read[:lines].size
+          data << "\n... (more lines) — resume with offset=#{resume}"
+        end
+        [data, resume]
+      end
+
+      def directory_listing(path)
+        entries = Dir.children(path).sort
+        entries.map! do |e|
+          full = File.join(path, e)
+          "#{e}#{File.directory?(full) ? '/' : ''}"
+        end
+        Ask::Result.ok(data: entries.join("\n"), metadata: { type: "directory", count: entries.size })
+      end
+
+      # Peek the head of the file: PDF magic first (it's also binary), then
+      # a null byte means binary. Returns an Ask::Result for special formats.
+      def sniff(path)
+        head = File.open(path, "rb") { |f| f.read(1024) } || ""
+        if head.start_with?("%PDF")
+          return Ask::Result.ok(
+            data: "PDF document (#{File.size(path)} bytes) — extract text with pdftotext.",
+            metadata: { format: "pdf" }
           )
         end
+        if head.include?("\x00")
+          mime = MIME_TYPES[File.extname(path).downcase] || "application/octet-stream"
+          return Ask::Result.ok(
+            data: "Binary file (#{mime}, #{File.size(path)} bytes) — content not shown.",
+            metadata: { format: "binary", mime: mime }
+          )
+        end
+        nil
+      end
+
+      # ── input repair ───────────────────────────────────────────────────
+
+      # Coerce the value an LLM actually sent into an integer, or return an
+      # error message string. Accepts Integer, "2000", 2.0 — rejects "2abc"
+      # and 1.5 rather than silently reading a wrong window.
+      def coerce_int(name, value)
+        case value
+        when nil then nil
+        when Integer then value
+        when String
+          value.match?(/\A-?\d+\z/) ? value.to_i : "Invalid #{name}: #{value.inspect} — expected an integer."
+        when Float
+          value == value.to_i ? value.to_i : "Invalid #{name}: #{value.inspect} — expected an integer."
+        else
+          "Invalid #{name}: #{value.inspect} — expected an integer."
+        end
+      end
+
+      # ── device blocklist ───────────────────────────────────────────────
+
+      def device_path?(path)
+        DEVICE_PATHS.include?(path) ||
+          path.start_with?("/dev/fd/") ||
+          path.match?(%r{\A/proc/\d+/fd/}) ||
+          path.match?(%r{\A/proc/\d+/task/\d+/fd/})
+      end
+
+      # ── filename repair ────────────────────────────────────────────────
+
+      def missing_path_message(path)
+        base = File.basename(path)
+        dir = File.dirname(path)
+
+        # The model can't see byte-level differences: narrow NBSP vs space,
+        # NFD vs NFC, straight vs curly quotes. Retry the candidates for it.
+        match = filename_variants(base).find { |c| File.exist?(File.join(dir, c)) }
+        if match
+          return "Path does not exist: #{path} — a close match exists: " \
+                 "#{File.join(dir, match)} (different characters; use that exact path)."
+        end
 
-        lines = File.readlines(path, chomp: true)
-        total = lines.size
-        offset_val = offset.to_i.clamp(0, total)
-        limit_val = limit || MAX_LINES
+        if File.directory?(dir)
+          suggestions = did_you_mean(base, Dir.children(dir))
+          unless suggestions.empty?
+            return "Path does not exist: #{path} — did you mean: " \
+                   "#{suggestions.map { |s| File.join(dir, s) }.join(", ")}?"
+          end
+        end
 
-        selected = lines[offset_val, limit_val]
-        truncated = selected.size < (total - offset_val)
+        "Path does not exist: #{path}"
+      end
 
-        result = selected.each_with_index.map do |line, i|
-          "#{offset_val + i + 1}: #{line}"
-        end.join("\n")
+      def filename_variants(name)
+        variants = []
+        [["\u202F", " "], ["\u00A0", " "]].each do |special, plain| # narrow NBSP / NBSP
+          variants << name.gsub(special, plain) if name.include?(special)
+          variants << name.gsub(plain, special) if name.include?(plain)
+        end
+        variants << name.unicode_normalize(:nfd) unless name.unicode_normalized?(:nfd)
+        variants << name.unicode_normalize(:nfc) unless name.unicode_normalized?(:nfc)
+        [["'", "\u2018"], ["'", "\u2019"], ['"', "\u201C"], ['"', "\u201D"]].each do |straight, curly|
+          variants << name.gsub(straight, curly) if name.include?(straight)
+          variants << name.gsub(curly, straight) if name.include?(curly)
+        end
+        variants.uniq.reject { |v| v == name }
+      end
+
+      def did_you_mean(name, siblings)
+        others = siblings.reject { |s| s == name }
+        if name.length >= 3
+          sub = others.select { |s| s.include?(name) || name.include?(s) }
+          return sub.sort_by(&:length).first(3) unless sub.empty?
+        end
+        others
+          .select { |s| levenshtein(name, s) <= 2 }
+          .sort_by { |s| levenshtein(name, s) }
+          .first(3)
+      end
 
-        result << "\n... (#{total - offset_val - selected.size} more lines)" if truncated
+      # Bounded Wagner–Fischer; cheap because we bail on length gap > max.
+      def levenshtein(a, b, max: 2)
+        return 0 if a == b
+        return max + 1 if (a.length - b.length).abs > max
 
-        Ask::Result.ok(data: result, metadata: {
-          total_lines: total,
-          start_line: offset_val + 1,
-          end_line: offset_val + selected.size,
-          truncated: truncated
-        })
+        row = (0..b.length).to_a
+        a.each_char do |ac|
+          prev = row[0]
+          row[0] = prev + 1
+          b.each_char.with_index do |bc, j|
+            cur = row[j + 1]
+            row[j + 1] = [cur + 1, row[j] + 1, prev + (ac == bc ? 0 : 1)].min
+            prev = cur
+          end
+        end
+        row[b.length]
       end
     end
   end

data/lib/ask/tools/shell/repl/kernel_script.rb

@@ -0,0 +1,89 @@
+# frozen_string_literal: true
+
+# Persistent Ruby kernel for Ask::Tools::Shell::Repl.
+#
+# A standalone plain-Ruby script run as a long-lived subprocess. Reads framed
+# JSON requests from stdin, evaluates each snippet into a persistent binding,
+# and writes a framed JSON response to stdout.
+#
+# Protocol (newline-delimited JSON):
+#   request:  {"id": 1, "code": "1 + 1"}
+#   response: {"id": 1, "result": "2", "stdout": "", "stderr": "", "error": null}
+#
+# The binding persists across requests, so locals, requires, and defined
+# methods survive between calls. stdout/stderr are captured per evaluation.
+
+require "json"
+require "stringio"
+
+# Die immediately on TERM (sent by the parent to shut the session down).
+# exit! bypasses the rescue Exception below, which would otherwise swallow
+# the SignalException raised mid-eval and keep the kernel alive.
+trap("TERM") { exit!(0) }
+
+def read_frame
+  line = $stdin.gets
+  return nil if line.nil?
+
+  JSON.parse(line)
+rescue JSON::ParserError
+  { "error" => "invalid request frame" }
+end
+
+def write_frame(frame)
+  $stdout.puts(JSON.generate(frame))
+  $stdout.flush
+end
+
+def capture_output
+  old_out = $stdout
+  old_err = $stderr
+  out = StringIO.new
+  err = StringIO.new
+  $stdout = out
+  $stderr = err
+  yield
+  [out.string, err.string]
+ensure
+  $stdout = old_out
+  $stderr = old_err
+end
+
+def safe_inspect(value)
+  value.inspect
+rescue StandardError
+  "#<#{value.class}>"
+end
+
+binding = TOPLEVEL_BINDING
+
+loop do
+  request = read_frame
+  break if request.nil?
+
+  id = request["id"]
+  code = request["code"].to_s
+
+  result = nil
+  stdout = ""
+  stderr = ""
+  error = nil
+
+  begin
+    stdout, stderr = capture_output do
+      result = binding.eval(code)
+    end
+  rescue Exception => e # rubocop:disable Lint/RescueException
+    error = "#{e.class}: #{e.message}"
+    trace = (e.backtrace || []).first(10).join("\n")
+    stderr += trace
+  end
+
+  write_frame(
+    "id" => id,
+    "result" => safe_inspect(result),
+    "stdout" => stdout,
+    "stderr" => stderr,
+    "error" => error
+  )
+end

data/lib/ask/tools/shell/repl.rb

@@ -0,0 +1,357 @@
+# frozen_string_literal: true
+
+require "json"
+require "monitor"
+require "io/wait"
+
+module Ask
+  module Tools
+    # Evaluate Ruby code in a persistent, long-lived subprocess.
+    #
+    # Unlike {Code} (which spawns a fresh `ruby -e` per call), Repl keeps a
+    # kernel process alive across calls and evaluates every snippet into the
+    # same binding — so state (locals, requires, defined methods) survives
+    # between calls. This is the RLM (recursive language model) pattern: the
+    # model composes capabilities as code against a persistent environment.
+    #
+    # @example
+    #   repl = Ask::Tools::Repl.new
+    #   repl.call(code: 'require "json"; data = JSON.parse(%q({"a": 1}))')
+    #   repl.call(code: "data['a'] + 1")       # => 2 — `data` still exists
+    #   repl.call(code: "def double(x); x * 2; end")
+    #   repl.call(code: "double(21)")          # => 42
+    #
+    # Sessions are named and shared process-wide: calling with the same
+    # +session+ name from any tool instance reaches the same kernel. A
+    # session is closed by +reset: true+, {Repl.close_session}, or after
+    # {Repl.idle_timeout} seconds without use.
+    #
+    # @note The kernel runs with the caller's permissions (like {Code}).
+    #   It is a durable control environment, not a security sandbox.
+    class Repl < Ask::Tool
+      description "Evaluate Ruby code in a persistent session. " \
+                   "State (variables, requires, defined methods) survives " \
+                   "across calls in the same session. " \
+                   "Use sessions to keep working context alive."
+
+      param :code, type: :string, desc: "Ruby source code to evaluate", required: true
+      param :session, type: :string, desc: "Session name; state persists per name", required: false
+      param :reset, type: :boolean, desc: "Discard session state before evaluating", required: false
+
+      # Timeout for a single evaluation, in seconds.
+      DEFAULT_EVAL_TIMEOUT = 30
+
+      # Sessions idle longer than this are closed on next access.
+      DEFAULT_IDLE_TIMEOUT = 300
+
+      # Env vars that would drag the parent's bundler context into the
+      # kernel subprocess (RUBYOPT=-rbundler/setup restricts $LOAD_PATH to
+      # the parent's Gemfile). Nil overrides remove them at spawn so the
+      # session is plain ruby, like the one-shot Code tool's sandbox.
+      BUNDLER_ENV = %w[
+        RUBYOPT RUBYLIB BASH_ENV GEM_PATH GEM_HOME
+        BUNDLE_GEMFILE BUNDLE_PATH BUNDLE_BIN_PATH BUNDLER_SETUP
+        BUNDLER_VERSION BUNDLE_WITHOUT BUNDLE_FROZEN BUNDLE_DEPLOYMENT
+        BUNDLE_LOCKFILE BUNDLE_APP_CONFIG
+      ].freeze
+
+      class << self
+        # @return [Integer] seconds a session may sit unused before it is
+        #   closed on next access
+        attr_accessor :idle_timeout
+
+        # @return [Integer] seconds a single evaluation may take
+        attr_accessor :eval_timeout
+      end
+
+      self.idle_timeout = DEFAULT_IDLE_TIMEOUT
+      self.eval_timeout = DEFAULT_EVAL_TIMEOUT
+
+      @registry = {}
+      @registry_mutex = Monitor.new
+
+      class << self
+        # The kernel for +session+, spawning one if needed (or after reset).
+        #
+        # @param session [String]
+        # @param reset [Boolean] discard existing session state
+        # @return [Kernel]
+        def kernel_for(session, reset: false)
+          @registry_mutex.synchronize do
+            close_session(session) if reset
+            kernel = @registry[session]
+            if kernel.nil? || kernel.dead?
+              kernel = Kernel.new(session: session)
+              @registry[session] = kernel
+            elsif kernel.idle_seconds > idle_timeout
+              kernel.close
+              kernel = Kernel.new(session: session)
+              @registry[session] = kernel
+            end
+            kernel
+          end
+        end
+
+        # Close and forget a session's kernel.
+        #
+        # @param session [String]
+        # @return [void]
+        def close_session(session)
+          @registry_mutex.synchronize do
+            kernel = @registry.delete(session)
+            kernel&.close
+          end
+          nil
+        end
+
+        # Close every session kernel. Called at exit; call it explicitly to
+        # free subprocesses early.
+        #
+        # @return [void]
+        def close_all
+          @registry_mutex.synchronize do
+            @registry.each_value(&:close)
+            @registry.clear
+          end
+          nil
+        end
+
+        # @return [Array<String>] active session names
+        def sessions
+          @registry_mutex.synchronize { @registry.keys }
+        end
+      end
+
+      at_exit { Repl.close_all }
+
+      def execute(code:, session: "default", reset: false)
+        build_result(eval_code(session, code, reset: reset), session)
+      rescue Kernel::TimeoutError => e
+        Ask::Result.error(message: "REPL session '#{session}' timed out (#{e.timeout}s); session state was lost")
+      rescue Kernel::DeadError => e
+        # The session died (crash, external kill, closed stdin). Respawn a
+        # fresh kernel and retry once; only give up if it dies again.
+        begin
+          build_result(eval_code(session, code), session)
+        rescue Kernel::DeadError
+          Ask::Result.error(message: "REPL session '#{session}' died repeatedly: #{e.message}")
+        end
+      end
+
+      private
+
+      def eval_code(session, code, reset: false)
+        Repl.kernel_for(session, reset: reset).eval(code, timeout: Repl.eval_timeout)
+      end
+
+      def build_result(result, session)
+        if result["error"]
+          Ask::Result.error(
+            message: result["error"],
+            metadata: { session: session, stdout: result["stdout"], stderr: result["stderr"] }
+          )
+        else
+          Ask::Result.ok(data: {
+            result: result["result"],
+            stdout: result["stdout"],
+            stderr: result["stderr"],
+            session: session
+          })
+        end
+      end
+
+      # A single long-lived Ruby subprocess executing the kernel script.
+      #
+      # Communicates over newline-delimited JSON on stdin/stdout. One eval
+      # at a time per kernel; concurrent calls serialize on an internal
+      # monitor.
+      class Kernel
+        class Error < StandardError; end
+        class TimeoutError < Error
+          attr_reader :timeout
+
+          def initialize(timeout)
+            @timeout = timeout
+            super("evaluation exceeded #{timeout}s")
+          end
+        end
+        class DeadError < Error; end
+
+        # @return [String] session name this kernel belongs to
+        attr_reader :session
+
+        # @return [Time] last time an evaluation completed
+        attr_reader :last_used
+
+        def initialize(session:, ruby: "ruby")
+          @session = session
+          @script = File.expand_path("repl/kernel_script.rb", __dir__)
+          @monitor = Monitor.new
+          @last_used = Time.now
+          @next_id = 0
+          spawn_process(ruby)
+        end
+
+        # @return [Float] seconds since the last evaluation
+        def idle_seconds
+          Time.now - @last_used
+        end
+
+        # @return [Boolean] whether the kernel process is gone. Reaps the
+        #   child if it already exited (zombies count as dead).
+        def dead?
+          return true if @closed
+          return true unless @pid
+
+          _, status = Process.waitpid(@pid, Process::WNOHANG)
+          if status.nil?
+            false
+          else
+            @pid = nil
+            true
+          end
+        rescue Errno::ECHILD, Errno::ESRCH, Errno::EINTR
+          @pid = nil
+          true
+        end
+
+        # Evaluate +code+ in the persistent binding.
+        #
+        # @param code [String]
+        # @param timeout [Integer] max seconds for this evaluation
+        # @return [Hash] {"result" => String, "stdout" => String,
+        #   "stderr" => String, "error" => String or nil}
+        # @raise [TimeoutError] evaluation exceeded +timeout+; the kernel
+        #   was killed and session state lost
+        # @raise [DeadError] the kernel process died
+        def eval(code, timeout: Repl.eval_timeout)
+          @monitor.synchronize do
+            raise DeadError, "process not running" if dead?
+
+            id = (@next_id += 1)
+            write_frame("id" => id, "code" => code)
+            response = read_frame(id, timeout)
+            @last_used = Time.now
+            response
+          end
+        end
+
+        # Terminate the kernel process and close pipes. TERM is normally
+        # enough (the kernel script exits on it); KILL is the fallback for
+        # user code that overrode the trap or wedged the VM.
+        #
+        # @return [void]
+        def close
+          @monitor.synchronize do
+            return if @closed
+
+            @closed = true
+            if @pid
+              begin
+                Process.kill("TERM", @pid)
+              rescue Errno::ESRCH, Errno::ECHILD
+                @pid = nil
+              end
+              wait_for_exit(2)
+              if @pid
+                begin
+                  Process.kill("KILL", @pid)
+                rescue Errno::ESRCH, Errno::ECHILD
+                  @pid = nil
+                end
+                wait_for_exit(1)
+              end
+            end
+            @in_w.close unless @in_w.closed?
+            @out_r.close unless @out_r.closed?
+            @err_r.close unless @err_r.closed?
+          end
+          nil
+        end
+
+        private
+
+        # Poll until the child exits or +seconds+ elapse. Reaps the child
+        # when it does. Avoids blocking +Process.wait+ which Timeout cannot
+        # always interrupt.
+        #
+        # @param seconds [Numeric]
+        # @return [void]
+        def wait_for_exit(seconds)
+          deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds
+          loop do
+            _, status = Process.waitpid(@pid, Process::WNOHANG)
+            return if status
+
+            break if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline
+
+            sleep 0.01
+          end
+        rescue Errno::ECHILD, Errno::ESRCH, Errno::EINTR
+          @pid = nil
+        end
+
+        def spawn_process(ruby)
+          @in_r, @in_w = IO.pipe
+          @out_r, @out_w = IO.pipe
+          @err_r, @err_w = IO.pipe
+
+          # The kernel is a plain ruby environment: remove bundler plumbing
+          # inherited from the parent (RUBYOPT=-rbundler/setup etc.) so the
+          # session sees globally installed gems, not the parent's Gemfile
+          # subset. Note: Process.spawn *merges* its env hash with the
+          # parent environment — only explicit nil values delete keys.
+          env = BUNDLER_ENV.to_h { |key| [key, nil] }
+
+          @pid = Process.spawn(
+            env, ruby, @script,
+            in: @in_r, out: @out_w, err: @err_w
+          )
+          @in_r.close
+          @out_w.close
+          @err_w.close
+        rescue StandardError
+          close
+          raise
+        end
+
+        def write_frame(frame)
+          @in_w.write(JSON.generate(frame) + "\n")
+          @in_w.flush
+        rescue Errno::EPIPE, IOError => e
+          raise DeadError, e.message
+        end
+
+        def read_frame(expected_id, timeout)
+          deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout
+
+          loop do
+            remaining = deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            if remaining <= 0
+              close
+              raise TimeoutError, timeout
+            end
+
+            ready = @out_r.wait_readable(remaining)
+            unless ready
+              close
+              raise TimeoutError, timeout
+            end
+
+            line = @out_r.gets
+            if line.nil?
+              err = @err_r.read
+              close
+              raise DeadError, "kernel exited unexpectedly#{err.empty? ? "" : ": #{err.strip}"}"
+            end
+
+            frame = JSON.parse(line)
+            return frame if frame["id"] == expected_id
+          rescue JSON::ParserError
+            next
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ask/tools/shell/version.rb

@@ -3,7 +3,7 @@
 module Ask
   module Tools
     module Shell
-      VERSION = "0.3.4"
+      VERSION = "0.5.0"
     end
   end
 end

data/lib/ask/tools/shell/write.rb

@@ -24,6 +24,19 @@ module Ask
           )
         end
 
+        # Never destroy what the model never saw: if Read only showed part of
+        # this file and it hasn't changed since, the rest may hold something
+        # the overwrite would silently erase.
+        if Shell::FileLedger.partially_seen?(path)
+          entry = Shell::FileLedger.entry_for(path)
+          seen = entry.lines_seen
+          return Ask::Result.error(
+            message: "Refusing to overwrite #{path}: only part of the file has been read " \
+                     "(lines #{seen.first + 1}–#{seen.last} shown, more lines exist). " \
+                     "Re-read the full file first."
+          )
+        end
+
         dir = File.dirname(path)
         FileUtils.mkdir_p(dir) unless File.directory?(dir)
 

data/lib/ask/tools/shell.rb

@@ -4,17 +4,19 @@ require_relative "shell/version"
 require "ask/tools/tool"
 require_relative "shell/bash"
 require_relative "shell/read"
+require_relative "shell/file_ledger"
 require_relative "shell/write"
 require_relative "shell/edit"
 require_relative "shell/glob"
 require_relative "shell/grep"
 require_relative "shell/code"
 require_relative "shell/apply_patch"
+require_relative "shell/repl"
 
 module Ask
   module Tools
     module Shell
-      TOOLS = [Bash, Read, Write, Edit, Glob, Grep, Code, ApplyPatch].freeze
+      TOOLS = [Bash, Read, Write, Edit, Glob, Grep, Code, ApplyPatch, Repl].freeze
 
       def self.all
         TOOLS.map(&:new)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ask-tools-shell
 version: !ruby/object:Gem::Version
-  version: 0.3.4
+  version: 0.5.0
 platform: ruby
 authors:
 - Kaka Ruto
@@ -96,10 +96,13 @@ files:
 - lib/ask/tools/shell/bash.rb
 - lib/ask/tools/shell/code.rb
 - lib/ask/tools/shell/edit.rb
+- lib/ask/tools/shell/file_ledger.rb
 - lib/ask/tools/shell/file_mutation_queue.rb
 - lib/ask/tools/shell/glob.rb
 - lib/ask/tools/shell/grep.rb
 - lib/ask/tools/shell/read.rb
+- lib/ask/tools/shell/repl.rb
+- lib/ask/tools/shell/repl/kernel_script.rb
 - lib/ask/tools/shell/version.rb
 - lib/ask/tools/shell/write.rb
 homepage: https://github.com/ask-rb/ask-tools-shell