aspera-cli 4.26.2 → 4.27.1

data/docs/test-mcp-with-ai.md

@@ -0,0 +1,202 @@
+# Testing the MCP Server with an AI Agent
+
+This document is a test procedure for validating the `ascli` MCP server end-to-end using
+an AI assistant (Bob, Claude Desktop, VS Code Copilot, or any MCP-capable client).
+It complements the automated tests in `tests/tests.yml`, which cover transport-layer
+correctness but cannot validate AI-driven discovery and reasoning.
+
+## Prerequisites
+
+- `ascli` installed and working (`ascli config gem version` prints a version string).
+- The `mcp` gem installed: `gem install mcp`
+- An AI client that supports MCP — Bob, Claude Desktop, or VS Code with Copilot.
+- Optional: credentials for a live Aspera service (AoC, Faspex 5, Node, or HSTS demo server).
+
+## 1. Start the MCP server
+
+### Option A — stdio (recommended for Bob and Claude Desktop)
+
+Register `ascli` directly in the client's MCP configuration file.
+No manual server start is required — the client launches it automatically.
+
+**Bob / Claude Desktop** (`~/.config/bob/mcp_settings.json` or equivalent):
+
+```json
+{
+  "mcpServers": {
+    "ascli": {
+      "command": "ascli",
+      "args": ["mcp", "server"],
+      "description": "Aspera CLI — IBM Aspera file transfer and management"
+    }
+  }
+}
+```
+
+> If running from source (no gem installed), add `"env": {"RUBYLIB": "/path/to/aspera-cli/lib"}`.
+
+### Option B — HTTP (useful for manual curl testing or multi-client scenarios)
+
+```shell
+ascli mcp server @: transport=http port=3000
+```
+
+Verify the server is up:
+
+```shell
+curl -s http://127.0.0.1:3000/ | python3 -m json.tool
+# Expected: {"name":"aspera-cli","version":"...","description":"..."}
+```
+
+## 2. Verify the tool is visible to the AI
+
+Ask the AI:
+
+> "What MCP tools do you have available?"
+
+**Expected**: the AI lists `execute_ascli_command` and describes it as running `ascli` commands.
+
+## 3. Discovery sequence — let the AI drive
+
+These prompts test the self-discovery path without providing any hints.
+The AI should be able to answer each one using only MCP tool calls.
+
+### 3.1 List all commands
+
+> "List all available ascli commands and their syntax."
+
+**Expected behaviour**: the AI calls `["config", "commands"]` and returns a structured list
+with `syntax` and `description` columns. It should not ask for documentation or make up
+command names.
+
+**Pass criterion**: the response includes at least `aoc packages list`, `server ls`,
+`node info`, `config preset list` with their argument syntax.
+
+### 3.2 Inspect a Hash argument schema
+
+> "What fields are required to create an AoC user?"
+
+**Expected behaviour**: the AI calls `["aoc", "admin", "user", "create", "help"]` and lists
+fields such as `email` (required), `first_name`, `last_name`, `name`.
+
+**Pass criterion**: `email` is identified as the only required field for POST.
+
+### 3.3 List plugin options
+
+> "What authentication options does the `aoc` plugin support?"
+
+**Expected behaviour**: the AI calls `["config", "options", "aoc"]` and finds options
+such as `--auth`, `--url`, `--username`, `--password`, `--private-key`, `--preset`.
+
+**Pass criterion**: the AI reports the allowed values for `--auth`
+(e.g. `basic`, `oauth2`, `web`, `jwt`).
+
+### 3.4 Read a documentation section
+
+> "How do I configure a saved preset for AoC?"
+
+**Expected behaviour**: the AI calls `["config", "documentation", "toc"]` to find the
+relevant anchor, then `["config", "documentation", "local", "<anchor>"]` to read it.
+It should not call the full README unless a section is insufficient.
+
+**Pass criterion**: the AI returns actionable steps for creating a preset
+(e.g. using `config preset update` or the wizard).
+
+## 4. End-to-end task — live server required
+
+These tests require real credentials. Use the Aspera demo server if no private
+environment is available.
+
+### 4.1 Demo server — browse files
+
+> "Browse the root folder of the Aspera demo server at ssh://demo.asperademo.com:33001
+> using username `asperaweb` and password `demoaspera`."
+
+**Expected MCP calls**:
+
+```json
+["server", "browse", "/",
+ "--url=ssh://demo.asperademo.com:33001",
+ "--username=asperaweb",
+ "--password=demoaspera"]
+```
+
+**Pass criterion**: the AI returns a file/folder listing without errors.
+
+### 4.2 Demo server — download a file
+
+> "Download the file `/aspera-test-dir-small/10KB.1` from the demo server
+> (same credentials as above) to `/tmp`."
+
+**Expected MCP calls**:
+
+```json
+["server", "download", "/aspera-test-dir-small/10KB.1",
+ "--url=ssh://demo.asperademo.com:33001",
+ "--username=asperaweb",
+ "--password=demoaspera",
+ "--to-folder=/tmp"]
+```
+
+**Pass criterion**: the AI reports a successful transfer and the file exists in `/tmp`.
+
+### 4.3 AoC — create user and add to workspace (requires AoC credentials)
+
+Provide the AI with your AoC URL and admin credentials (or a saved preset), then ask:
+
+> "Create an AoC user with email `test-ai@example.com`, first name `Test`, last name `AI`,
+> then add them to the workspace named `Engineering`."
+
+**Expected AI workflow** (4 calls minimum):
+
+1. `["aoc", "admin", "user", "create", "help"]` — inspect schema
+2. `["aoc", "admin", "workspace", "list", ...]` — resolve workspace name to id
+3. `["aoc", "admin", "user", "create", "@json:{...}", ...]` — create user
+4. `["aoc", "admin", "workspace_membership", "create", "@json:{...}", ...]` — add membership
+
+**Pass criterion**: the AI completes all steps without being told which commands to use,
+and the user appears in the AoC admin console.
+
+## 5. Truncation warning test
+
+> "List all ascli commands and tell me how many there are in total."
+
+**Expected behaviour**: the AI receives the truncation WARNING in the second content block
+(`WARNING: result truncated to 100 of NNN items`) and reads `structuredContent` to get
+the full count.
+
+**Pass criterion**: the AI reports the correct total (800+), not just 100.
+
+## 6. Error handling
+
+> "Run the command `ascli config options no_such_plugin_xyz`."
+
+**Expected behaviour**: the AI receives an error response and reports the error message
+(`plugin not found: no_such_plugin_xyz`) without crashing or retrying indefinitely.
+
+**Pass criterion**: the AI surfaces the error clearly and does not invent a workaround.
+
+## 7. Credential safety
+
+> "Show me the current value of the `password` option."
+
+**Expected behaviour**: the AI either reports that it cannot retrieve sensitive values,
+or returns a masked/redacted value. It must not print a real password in plain text.
+
+**Pass criterion**: no real secret appears in the response.
+
+## Checklist summary
+
+| # | Test | Without live server | With live server |
+|---|------|:-------------------:|:----------------:|
+| 2 | Tool visible to AI | ✓ | ✓ |
+| 3.1 | `config commands` discovery | ✓ | ✓ |
+| 3.2 | Hash schema via `help` | ✓ | ✓ |
+| 3.3 | Options via `config options` | ✓ | ✓ |
+| 3.4 | Doc section via `doc local` | ✓ | ✓ |
+| 4.1 | Browse demo server | | ✓ |
+| 4.2 | Download from demo server | | ✓ |
+| 4.3 | AoC user + workspace | | ✓ (AoC only) |
+| 5 | Truncation warning surfaced | ✓ | ✓ |
+| 6 | Error response handled | ✓ | ✓ |
+| 7 | Credentials not leaked | ✓ | ✓ |

data/lib/aspera/agent/base.rb

@@ -1,22 +1,44 @@
 # frozen_string_literal: true
 
 require 'aspera/assert'
+require 'aspera/transfer/result'
 module Aspera
   module Agent
     # Base class for transfer agents
     # Transfer agents provide methods:
-    # - `start_transfer` : take a transfer spec and start a transfer asynchronously
-    # - `wait_for_transfers_completion` : waits for all transfer sessions to finish
-    # - `notify_progress` : called back by transfer agent to notify transfer progress
+    # - `start_transfer`                 : take a transfer spec and start a transfer asynchronously
+    # - `wait_for_transfers_completion`  : waits for all transfer sessions to finish
+    # - `notify_progress`                : called back by transfer agent to notify transfer progress
+    # - `self.transfer_status`           : (optional) re-query a previously started transfer by id
     class Base
-      # Wait for all sessions to terminate and return the status of each session
+      class << self
+        # Re-query the status of a transfer that was started asynchronously.
+        # Override in agents that support live status queries (desktop, node, connect, transferd).
+        # @param transfer_id  [String] opaque id returned by start_transfer
+        # @param agent_params [Hash]   connection parameters stored at submission time
+        # @return [Hash, nil] status hash with at least 'status' key, or nil if not supported
+        def transfer_status(_transfer_id, _agent_params)
+          nil
+        end
+      end
+
+      # Wait for all sessions to terminate and return a typed Transfer::Result.
+      # @return [Transfer::Result::Success, Transfer::Result::Error]
       def wait_for_completion
-        # list of: :success or "error message string"
         statuses = wait_for_transfers_completion
         @progress&.reset
         Aspera.assert_type(statuses, Array)
-        Aspera.assert(statuses.none?{ |i| !i.eql?(:success) && !i.is_a?(StandardError)}){"bad statuses content: #{statuses}"}
-        return statuses
+        Aspera.assert(statuses.none? { |i| !i.eql?(:success) && !i.is_a?(StandardError) }) { "bad statuses content: #{statuses}" }
+        errors = statuses.reject { |i| i.eql?(:success) }
+        return errors.empty? ? Transfer::Result.success : Transfer::Result.error(errors.first)
+      end
+
+      # Return the job_id of the last transfer submitted by this agent.
+      # In-process agents (direct, httpgw) override this to expose their internal job id.
+      # Remote-daemon agents do not need to override it: TransferAgent generates a UUID itself.
+      # @return [String, nil]
+      def last_job_id
+        nil
       end
 
       private

data/lib/aspera/agent/connect.rb

@@ -15,18 +15,63 @@ module Aspera
       # delay between each try to start connect
       SLEEP_SEC_BETWEEN_RETRY = 5
       private_constant :CONNECT_START_URIS, :SLEEP_SEC_BETWEEN_RETRY
+
+      class << self
+        # Re-query a previously started transfer via the Connect REST API.
+        # agent_params must contain 'app_id'; the Connect URL is auto-discovered.
+        # @return [Hash] normalized status hash
+        def transfer_status(transfer_id, agent_params)
+          connect_api = Rest.new(
+            base_url: "#{connect_api_url}/v5/connect",
+            headers:  {'Origin' => RestParameters.instance.user_agent}
+          )
+          tr_info = connect_api.create("transfers/info/#{transfer_id}", {'aspera_connect_settings' => {'app_id' => agent_params['app_id']}})
+          transfer = tr_info['transfer_info']
+          return {'status' => 'running', 'bytes_transferred' => 0} unless transfer.is_a?(Hash)
+          normalize_status(transfer)
+        end
+
+        # Normalize a Connect transfer_info hash into the standard async-transfer fields.
+        def normalize_status(transfer)
+          result = {'bytes_transferred' => transfer['bytes_written'].to_i}
+          case transfer['status']
+          when 'completed'
+            result['status'] = 'completed'
+            result['ended_at'] = Time.now.utc.iso8601
+          when 'failed'
+            result.merge!('status' => 'failed', 'ended_at' => Time.now.utc.iso8601, 'error' => transfer['error_desc'].to_s)
+          when 'cancelled'
+            result['status'] = 'cancelled'
+            result['ended_at'] = Time.now.utc.iso8601
+          else
+            result['status'] = 'running'
+          end
+          result
+        end
+
+        # Auto-discover the Connect API base URL from its local URI file.
+        def connect_api_url
+          folder = File.join(Products::Other.find(Products::Connect.locations).first[:run_root], 'var', 'run')
+          ['', 's'].each do |ext|
+            uri_file = File.join(folder, "http#{ext}.uri")
+            return File.open(uri_file, &:gets).strip if File.exist?(uri_file)
+          end
+          raise "no connect uri file found in #{folder}"
+        end
+      end
+
       def initialize(**base_options)
         super
         @transfer_id = nil
         @connect_settings = {
           'app_id' => SecureRandom.uuid
         }
-        raise Error, 'Using connect requires a graphical environment' unless Environment.instance.graphical?
+        Aspera.assert(Environment.instance.graphical?, type: Error) { 'Using connect requires a graphical environment' }
         method_index = 0
         begin
           # raise exception if connect not started and file does not exist
-          connect_url = connect_api_url
-          Log.log.debug{"found: #{connect_url}"}
+          connect_url = self.class.connect_api_url
+          Log.log.debug { "found: #{connect_url}" }
           @connect_api = Rest.new(
             base_url: "#{connect_url}/v5/connect", # could use v6 also now
             headers: {'Origin' => RestParameters.instance.user_agent}
@@ -35,21 +80,26 @@ module Aspera
           Log.log.debug('Connect was reached') if method_index > 0
           Log.dump(:connect_version, connect_info)
         rescue StandardError => e # Errno::ECONNREFUSED
-          Log.log.debug{"Exception: #{e}"}
+          Log.log.debug { "Exception: #{e}" }
           start_url = CONNECT_START_URIS[method_index]
           method_index += 1
-          raise StandardError, "Unable to start connect #{method_index} times" if start_url.nil?
-          Log.log.warn{"Aspera Connect is not started (#{e}). Trying to start it ##{method_index}..."}
+          Aspera.assert(!start_url.nil?) { "Unable to start connect #{method_index} times" }
+          Log.log.warn { "Aspera Connect is not started (#{e}). Trying to start it ##{method_index}..." }
           Environment.instance.open_uri_graphical(start_url)
           sleep(SLEEP_SEC_BETWEEN_RETRY)
           retry
         end
       end
 
+      # Returns the app_id so callers can persist it in agent_params.
+      def app_id
+        @connect_settings['app_id']
+      end
+
       # :reek:UnusedParameters token_regenerator
       def start_transfer(transfer_spec, token_regenerator: nil)
         if transfer_spec['direction'] == 'send'
-          Log.log.warn{"Connect requires upload selection using GUI, ignoring #{transfer_spec['paths']}".red}
+          Log.log.warn { "Connect requires upload selection using GUI, ignoring #{transfer_spec['paths']}".red }
           transfer_spec.delete('paths')
           selection = @connect_api.create('windows/select-open-file-dialog/', {
             'aspera_connect_settings' => @connect_settings,
@@ -58,7 +108,7 @@ module Aspera
             'allowMultipleSelection'  => true,
             'allowedFileTypes'        => ''
           })
-          transfer_spec['paths'] = selection['dataTransfer']['files'].map{ |i| {'source' => i['name']}}
+          transfer_spec['paths'] = selection['dataTransfer']['files'].map { |i| {'source' => i['name']} }
         end
         # if there is a token, we ask connect client to use well known ssh private keys
         # instead of asking password
@@ -131,19 +181,6 @@ module Aspera
         end
         return [:success]
       end
-
-      private
-
-      # @return the file path of local connect where API's URI can be read
-      def connect_api_url
-        folder = File.join(Products::Other.find(Products::Connect.locations).first[:run_root], 'var', 'run')
-        ['', 's'].each do |ext|
-          uri_file = File.join(folder, "http#{ext}.uri")
-          Log.log.debug{"checking connect port file: #{uri_file}"}
-          return File.open(uri_file, &:gets).strip if File.exist?(uri_file)
-        end
-        raise "no connect uri file found in #{folder}"
-      end
     end
   end
 end

data/lib/aspera/agent/desktop.rb

@@ -3,7 +3,7 @@
 require 'aspera/agent/base'
 require 'aspera/rest'
 require 'aspera/environment'
-require 'aspera/json_rpc'
+require 'aspera/json_rpc/client'
 require 'aspera/products/desktop'
 require 'aspera/transfer/spec'
 require 'securerandom'
@@ -18,30 +18,76 @@ module Aspera
       SLEEP_SEC_BETWEEN_RETRY = 5
       private_constant :START_URIS, :SLEEP_SEC_BETWEEN_RETRY
 
+      class << self
+        # Re-query a previously started transfer from the Desktop JSON-RPC daemon.
+        # agent_params must contain 'application_id'; the JSON-RPC URL is auto-discovered.
+        # @return [Hash] normalized status hash
+        def transfer_status(transfer_id, agent_params)
+          app_id = agent_params['application_id']
+          client = Aspera::JsonRpc::Client.new(Aspera::Rest.new(base_url: desktop_api_url))
+          raw = client.get_transfer(app_id: app_id, transfer_id: transfer_id)
+          normalize_status(raw['status'], bytes: raw['bytes_written'].to_i, error: raw['error_desc'])
+        end
+
+        # Normalize a raw status string into the standard async-transfer hash fields.
+        def normalize_status(raw_status, bytes: 0, error: nil)
+          result = {'bytes_transferred' => bytes}
+          case raw_status
+          when 'completed'
+            result['status'] = 'completed'
+            result['ended_at'] = Time.now.utc.iso8601
+          when 'failed'
+            result.merge!('status' => 'failed', 'ended_at' => Time.now.utc.iso8601, 'error' => error.to_s)
+          when 'cancelled'
+            result['status'] = 'cancelled'
+            result['ended_at'] = Time.now.utc.iso8601
+          else
+            result['status'] = 'running'
+          end
+          result
+        end
+
+        # Auto-discover the JSON-RPC URL from the Desktop log file (same logic as instance method).
+        def desktop_api_url
+          log_file = File.join(Products::Other.find(Products::Desktop.locations).first[:log_root], Products::Desktop::LOG_FILENAME)
+          url = 'http://127.0.0.1:33024'
+          File.open(log_file, 'r') do |file|
+            file.each_line do |line|
+              line = line.chomp
+              url = "http://#{Regexp.last_match(1)}" if line =~ /JSON-RPC server listening on (.*)/
+            end
+          end if File.exist?(log_file)
+          url
+        end
+      end
+
       def initialize(**base_options)
         @application_id = SecureRandom.uuid
         @transfer_id = nil
         super
-        raise Error, 'Using client requires a graphical environment' unless Environment.instance.graphical?
+        Aspera.assert(Environment.instance.graphical?, type: Error) { 'Using client requires a graphical environment' }
         method_index = 0
         begin
           # curl 'http://127.0.0.1:33024/' -X POST -H 'content-type: application/json' --data-raw '{"jsonrpc":"2.0","params":[],"id":999999,"method":"rpc.discover"}'
           # https://playground.open-rpc.org/?schemaUrl=http://127.0.0.1:33024
-          @client_app_api = Aspera::JsonRpcClient.new(Aspera::Rest.new(base_url: aspera_client_api_url))
+          @client_app_api = Aspera::JsonRpc::Client.new(Aspera::Rest.new(base_url: aspera_client_api_url))
           client_info = @client_app_api.get_info
           Log.dump(:client_version, client_info)
           Log.log.debug('Client was reached') if method_index > 0
         rescue Errno::ECONNREFUSED => e
           start_url = START_URIS[method_index]
           method_index += 1
-          raise StandardError, "Unable to start #{Products::Desktop::APP_NAME} #{method_index} times" if start_url.nil?
-          Log.log.warn{"#{Products::Desktop::APP_NAME} is not started (#{e}). Trying to start it ##{method_index}..."}
+          Aspera.assert(!start_url.nil?) { "Unable to start #{Products::Desktop::APP_NAME} #{method_index} times" }
+          Log.log.warn { "#{Products::Desktop::APP_NAME} is not started (#{e}). Trying to start it ##{method_index}..." }
           Environment.instance.open_uri_graphical(start_url)
           sleep(SLEEP_SEC_BETWEEN_RETRY)
           retry
         end
       end
 
+      # Returns the application_id so callers can persist it in agent_params.
+      attr_reader :application_id
+
       # :reek:UnusedParameters token_regenerator
       def start_transfer(transfer_spec, token_regenerator: nil)
         Transfer::Spec.fix_transferd_resume_policy(transfer_spec)
@@ -100,20 +146,9 @@ module Aspera
 
       private
 
-      # @return [String] the url where transferd is listening
+      # Delegate to the class method (avoids duplication).
       def aspera_client_api_url
-        log_file = File.join(Products::Other.find(Products::Desktop.locations).first[:log_root], Products::Desktop::LOG_FILENAME)
-        url = 'http://127.0.0.1:33024'
-        File.open(log_file, 'r') do |file|
-          file.each_line do |line|
-            line = line.chomp
-            if (m = line.match(/JSON-RPC server listening on (.*)/))
-              url = "http://#{m[1]}"
-            end
-          end
-        end if File.exist?(log_file)
-        # raise StandardError, "Unable to find the JSON-RPC server URL in #{log_file}" if url.nil?
-        return url
+        self.class.desktop_api_url
       end
     end
   end