cuboid 0.0.4 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a640876cd5ec28f9568947bef4c9cb0ebb222e62ec49690ebaaf74f32b17c5f8
-  data.tar.gz: 40189342279e0719ce0840ea41bfd3320057d60610e35039f2da9fd7ca9428ed
+  metadata.gz: '0686605598011432eb7f38026c2b2c4cc5c76e6272ecd38b2132a2c382114ab6'
+  data.tar.gz: c7c2e338fcd17aa6cc07c4902528bb1d7b9be3d01c5f3b30fffbbc6e40377dec
 SHA512:
-  metadata.gz: de7922d59f271ac39d4e709680b96795ce136c9153889f4592dccffda25bd17792b6080147b95fa4919a391fd8dac827f3a65322fbb51d1e369a91ff4b3e3236
-  data.tar.gz: d38bfa6117aff9234cee45917168627ef5a132525994cabe57418a035fa424197a25dd1e9aa098759eeb640125effd7025558ea108082ed473539f7ae9cef39b
+  metadata.gz: 00a62cc9358e08fab9f11fdea832f1b95f540529e1099a6a697aacc7dfe6bbfef7f3c6b24ec240d6e10e9d4331e03d1b03fb335906bb4aeafbc64bc3beacee8c
+  data.tar.gz: 211cd5f371b8b6bfdde1050d90253c2a408d4b532b98fd54eaa908c95ca28678bf5d8914a18a16da4fe8af467998bc321b419e963a045851729b7934f91b6068

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+# 0.1.0
+
+* `Dispatcher` => `Agent`
+* `Dispatcher#dispatch` => `Agent#spawn`
+* `neighbour` => `peer`

data/README.md

@@ -10,8 +10,8 @@ _Cloud_ within a _Cloud_.
 
 It offers:
 
-* Load-balancing of _**Instances**_ via a software mesh network (_**Grid**_) of _**Dispatchers**_.
-  * No need to setup a topology manually, _**Dispatchers**_ will reach
+* Load-balancing of _**Instances**_ via a network (_**Grid**_) of _**Agents**_.
+  * No need to setup a topology manually, _**Agents**_ will reach
     convergence on their own, just point them to an existing _**Grid**_ member.
   * Scaling up and down can be easily achieved by _plugging_ or _unplugging_ nodes.
   * Horizontal (`default`) and vertical workload distribution strategies available.
@@ -58,7 +58,7 @@ framework:
     * `:restore`
   * `instance_service_for( Symbol, Class )` -- Adds a custom _**Instance**_ RPC API.
   * `rest_service_for( Symbol, Module )` -- Hooks-up to the _**REST**_ service to provide a custom REST API.
-  * `dispatcher_service_for( Symbol, Class )` -- Hooks-up to the _**Dispatcher**_ to provide a custom RPC API.
+  * `agent_service_for( Symbol, Class )` -- Hooks-up to the _**Agent**_ to provide a custom RPC API.
   * `serialize_with( Module )` -- A serializer to be used for:
     * `#options`
     * `Report#data`
@@ -84,10 +84,10 @@ This is in order to enforce isolation (_state_, _data_, _fault_) between
 _**Applications**_, take advantage of _OS_ task management and generally keep
 things simple.
 
-### Dispatcher
+### Agent
 
-A _**Dispatcher**_ is a server which awaits for _**Instance**_ spawn requests
-(`dispatch` calls) upon which it spawns and passes the _**Instance**_'s
+A _**Agent**_ is a server which awaits for _**Instance**_ spawn requests
+(`spawn` calls) upon which it spawns and passes the _**Instance**_'s
 connection info to the client.
 
 The client can then proceed to use the _**Instance**_ to run and generally manage
@@ -95,7 +95,7 @@ the contained _**Application**_.
 
 #### Grid
 
-A _**Dispatcher**_ _**Grid**_ is a software mesh network of _**Dispatcher**_
+A _**Agent**_ _**Grid**_ is a software mesh network of _**Agent**_
 servers, aimed towards providing automated _load-balancing_ based on available
 system resources and each _**Application**_'s provisioning configuration.
 
@@ -104,11 +104,11 @@ providing any existing _**Grid**_ member upon start-up and the rest will be
 sorted out automatically.
 
 The network is _self-healing_ and will monitor _node_ connectivity, taking steps
-to ensure that neither server nor network conditions will disrupt dispatching.
+to ensure that neither server nor network conditions will disrupt spawning.
 
 ##### Scalability
 
-_**Dispatchers**_ can be easily _plugged_ to or _unplugged_ from the _**Grid**_
+_**Agents**_ can be easily _plugged_ to or _unplugged_ from the _**Grid**_
 to scale up or down as necessary.
 
 _Plugging_ happens at boot-time and _unplugging_ can take place via the available
@@ -125,12 +125,12 @@ The _**Scheduler**_ is a server which:
   5. Upon _**Application**_ completion stores report to disk.
   6. Shuts down the _**Instance**_.
 
-#### Dispatcher
+#### Agent
 
-The _**Scheduler**_ can be configured with a _**Dispatcher**_, upon which case,
+The _**Scheduler**_ can be configured with a _**Agent**_, upon which case,
 it will use it to spawn _**Instances**_.
 
-If the _**Dispatcher**_ is a _**Grid**_ member then the _**Scheduler**_ will
+If the _**Agent**_ is a _**Grid**_ member then the _**Scheduler**_ will
 also enjoy _load-balancing_ features. 
 
 ## APIs
@@ -154,7 +154,7 @@ RPC interface.
 A REST API is also available, taking advantage of HTTP sessions to make progress
 tracking easier.
 
-The REST interface is basically a web _**Dispatcher**_ and centralised point of
+The REST interface is basically a web _**Agent**_ and centralised point of
 management for the rest of the entities.
 
 Each _**Application**_ can extend upon this and expose an API via its _**REST**_ 

data/cuboid.gemspec

@@ -42,12 +42,8 @@ Gem::Specification.new do |s|
     # RPC serialization.
     s.add_dependency 'msgpack',             '1.1.0'
 
-    # Optimized JSON.
-    s.add_dependency 'oj',                  '3.11.5'
-    s.add_dependency 'oj_mimic_json',       '1.0.1'
-
     # Web server
-    s.add_dependency 'puma',                '5.5.2'
+    s.add_dependency 'puma',                '5.6.2'
 
     s.add_dependency 'rack',                '2.2.3'
     s.add_dependency 'rack-test'

data/lib/cuboid/application.rb

@@ -103,15 +103,15 @@ class Application
             @max_memory ||= 0
         end
 
-        def provision_disk( ram )
-            @max_disk = ram
+        def provision_disk( disk )
+            @max_disk = disk
         end
 
         def max_disk
-            @max_memory ||= 0
+            @max_disk ||= 0
         end
 
-        def instance_service_for(name, service )
+        def instance_service_for( name, service )
             instance_services[name] = service
         end
 
@@ -127,12 +127,12 @@ class Application
             @rest_services ||= {}
         end
 
-        def dispatcher_service_for( name, service )
-            dispatcher_services[name] = service
+        def agent_service_for( name, service )
+            agent_services[name] = service
         end
 
-        def dispatcher_services
-            @dispatcher_services ||= {}
+        def agent_services
+            @agent_services ||= {}
         end
 
         def handler_for( signal, handler )
@@ -183,8 +183,8 @@ class Application
             case type
             when :instance
                 const = :Instances
-            when :dispatcher
-                const = :Dispatchers
+            when :agent
+                const = :Agents
             when :scheduler
                 const = :Schedulers
             when :rest

data/lib/cuboid/option_groups/agent.rb

@@ -0,0 +1,54 @@
+module Cuboid::OptionGroups
+
+# Holds options for {RPC::Server::Agent} servers.
+#
+# @author Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
+class Agent < Cuboid::OptionGroup
+
+    STRATEGIES = Set.new([:horizontal, :vertical])
+
+    # @return   [String]
+    #   URL of a {RPC::Server::Agent}.
+    attr_accessor :url
+
+    # @return   [Array<Integer>]
+    #   Range of ports to use when spawning instances, first entry should be
+    #   the lowest port number, last the max port number.
+    attr_accessor :instance_port_range
+
+    # @return   [String]
+    #   The URL of a peering {RPC::Server::Agent}, applicable when
+    #   {RPC::Server::Agent} are connected to each other to form a Grid.
+    #
+    # @see RPC::Server::Agent::Node
+    attr_accessor :peer
+
+    # @return   [Float]
+    #   How regularly to check for peer statuses.
+    attr_accessor :ping_interval
+
+    # @return   [String]
+    #   Agent name.
+    attr_accessor :name
+
+    attr_accessor :strategy
+
+    set_defaults(
+        strategy:            :horizontal,
+        ping_interval:       5.0,
+        instance_port_range: [1025, 65535]
+    )
+
+    def strategy=( type )
+        return @strategy = defaults[:strategy] if !type
+
+        type = type.to_sym
+        if !STRATEGIES.include? type
+            fail ArgumentError, "Unknown strategy type: #{type}"
+        end
+
+        @strategy = type
+    end
+
+end
+end

data/lib/cuboid/option_groups/paths.rb

@@ -70,7 +70,7 @@ class Paths < Cuboid::OptionGroup
 
     # @!attribute executables
     #   @return [String]
-    #       System processes (instance, dispatcher, browser, etc.).
+    #       System processes (instance, agent, browser, etc.).
 
     def initialize
         @root = self.root_path

data/lib/cuboid/options.rb

@@ -76,7 +76,7 @@ class Options
     require_relative 'option_groups'
 
     TO_RPC_IGNORE = Set.new([
-        :instance, :rpc, :dispatcher, :queue, :paths,
+        :instance, :rpc, :agent, :queue, :paths,
         :snapshot, :report, :output, :system
     ])
 

data/lib/cuboid/processes/agents.rb

@@ -0,0 +1,154 @@
+module Cuboid
+module Processes
+
+#
+# Helper for managing {RPC::Server::Agent} processes.
+#
+# @author Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
+#
+class Agents
+    include Singleton
+    include Utilities
+
+    # @return   [Array<String>] URLs of all running Agents.
+    attr_reader :list
+
+    def initialize
+        @list = []
+        @agent_connections = {}
+    end
+
+    # Connects to a Agent by URL.
+    #
+    # @param    [String]    url URL of the Agent.
+    # @param    [Hash]    options Options for the RPC client.
+    #
+    # @return   [RPC::Client::Agent]
+    def connect( url, options = nil )
+        Arachni::Reactor.global.run_in_thread if !Arachni::Reactor.global.running?
+
+        fresh = false
+        if options
+            fresh = options.delete( :fresh )
+        end
+
+        if fresh
+            @agent_connections[url] = RPC::Client::Agent.new( url, options )
+        else
+            @agent_connections[url] ||= RPC::Client::Agent.new( url, options )
+        end
+    end
+
+    # @param    [Block] block   Block to pass an RPC client for each Agent.
+    def each( &block )
+        @list.each do |url|
+            block.call connect( url )
+        end
+    end
+
+    # Spawns a {RPC::Server::Agent} process.
+    #
+    # @param    [Hash]  options
+    #   To be passed to {Cuboid::Options#set}. Allows `address` instead of
+    #   `rpc_server_address` and `port` instead of `rpc_port`.
+    #
+    # @return   [RPC::Client::Agent]
+    def spawn( options = {} )
+        options = options.dup
+        fork = options.delete(:fork)
+
+        options[:ssl] ||= {
+          server: {},
+          client: {}
+        }
+
+        options = {
+            agent: {
+                name:      options[:name],
+                peer: options[:peer],
+                strategy:  options[:strategy],
+            },
+            rpc:        {
+                server_port:             options[:port]    || Utilities.available_port,
+                server_address:          options[:address] || '127.0.0.1',
+                server_external_address: options[:external_address],
+
+                ssl_ca:                 options[:ssl][:ca],
+                server_ssl_private_key: options[:ssl][:server][:private_key],
+                server_ssl_certificate: options[:ssl][:server][:certificate],
+                client_ssl_private_key: options[:ssl][:client][:private_key],
+                client_ssl_certificate: options[:ssl][:client][:certificate],
+            },
+            paths: {
+                application: options[:application] || Options.paths.application
+            }
+        }
+
+        if options[:rpc][:server_external_address].nil?
+            options[:rpc].delete :server_external_address
+        end
+
+        if options[:agent][:peer].nil?
+            options[:agent].delete :peer
+        end
+
+        pid = Manager.spawn( :agent, options: options, fork: fork )
+
+        url = "#{options[:rpc][:server_address]}:#{options[:rpc][:server_port]}"
+        while sleep( 0.1 )
+            begin
+                connect( url, connection_pool_size: 1, max_retries: 1 ).alive?
+                break
+            rescue => e
+                # ap e
+            end
+        end
+
+        @list << url
+        connect( url, fresh: true ).tap { |c| c.pid = pid }
+    end
+
+    def grid_spawn( options = {} )
+        d = spawn( options )
+        spawn( options.merge peer: d.url )
+    end
+
+    # @note Will also kill all Instances started by the Agent.
+    #
+    # @param    [String]    url URL of the Agent to kill.
+    def kill( url )
+        agent = connect( url )
+        Manager.kill_many agent.statistics['consumed_pids']
+        Manager.kill agent.pid
+    rescue => e
+        #ap e
+        #ap e.backtrace
+        nil
+    ensure
+        @list.delete( url )
+        @agent_connections.delete( url )
+    end
+
+    # Kills all {Agents #list}.
+    def killall
+        @list.dup.each do |url|
+            kill url
+        end
+    end
+
+    def self.method_missing( sym, *args, &block )
+        if instance.respond_to?( sym )
+            instance.send( sym, *args, &block )
+        else
+            super( sym, *args, &block )
+        end
+    end
+
+    def self.respond_to?( m )
+        super( m ) || instance.respond_to?( m )
+    end
+
+end
+
+end
+end

data/lib/cuboid/processes/executables/agent.rb

@@ -0,0 +1,5 @@
+require Options.paths.lib  + 'rpc/server/agent'
+
+Arachni::Reactor.global.run do
+    RPC::Server::Agent.new
+end

data/lib/cuboid/processes/helpers/agents.rb

@@ -0,0 +1,23 @@
+# @param (see Cuboid::Processes::Agents#spawn)
+# @return (see Cuboid::Processes::Agents#spawn)
+def agent_spawn( *args )
+    Cuboid::Processes::Agents.spawn( *args )
+end
+
+# @param (see Cuboid::Processes::Agents#kill)
+# @return (see Cuboid::Processes::Agents#kill)
+def agent_kill( *args )
+    Cuboid::Processes::Agents.kill( *args )
+end
+
+# @param (see Cuboid::Processes::Agents#killall)
+# @return (see Cuboid::Processes::Agents#killall)
+def agent_killall
+    Cuboid::Processes::Agents.killall
+end
+
+# @param (see Cuboid::Processes::Agents#connect)
+# @return (see Cuboid::Processes::Agents#connect)
+def agent_connect( *args )
+    Cuboid::Processes::Agents.connect( *args )
+end

data/lib/cuboid/processes/helpers/instances.rb

@@ -10,10 +10,10 @@ def instance_grid_spawn( *args )
     Cuboid::Processes::Instances.grid_spawn( *args )
 end
 
-# @param (see Cuboid::Processes::Instances#dispatcher_spawn)
-# @return (see Cuboid::Processes::Instances#dispatcher_spawn)
-def instance_dispatcher_spawn( *args )
-    Cuboid::Processes::Instances.dispatcher.spawn( *args )
+# @param (see Cuboid::Processes::Instances#agent_spawn)
+# @return (see Cuboid::Processes::Instances#agent_spawn)
+def instance_agent_spawn( *args )
+    Cuboid::Processes::Instances.agent.spawn( *args )
 end
 
 def instance_kill( url )

data/lib/cuboid/processes/helpers.rb

@@ -1,4 +1,4 @@
 require_relative 'helpers/processes'
-require_relative 'helpers/dispatchers'
+require_relative 'helpers/agents'
 require_relative 'helpers/instances'
 require_relative 'helpers/schedulers'

data/lib/cuboid/processes/instances.rb

@@ -21,7 +21,7 @@ class Instances
     #
     # Connects to a Instance by URL.
     #
-    # @param    [String]    url URL of the Dispatcher.
+    # @param    [String]    url URL of the Agent.
     # @param    [String]    token
     #   Authentication token -- only need be provided once.
     #
@@ -61,16 +61,28 @@ class Instances
     # @return   [RPC::Client::Instance, Integer]
     #   RPC client and PID.
     def spawn( options = {}, &block )
+        options = options.dup
         token = options.delete(:token) || Utilities.generate_token
         fork  = options.delete(:fork)
 
         port_range = options.delete( :port_range )
 
+        options[:ssl] ||= {
+          server: {},
+          client: {}
+        }
+
         options = {
             rpc:    {
                 server_socket:  options[:socket],
                 server_port:    options[:port]    || Utilities.available_port( port_range ),
-                server_address: options[:address] || '127.0.0.1'
+                server_address: options[:address] || '127.0.0.1',
+
+                ssl_ca:                 options[:ssl][:ca],
+                server_ssl_private_key: options[:ssl][:server][:private_key],
+                server_ssl_certificate: options[:ssl][:server][:certificate],
+                client_ssl_private_key: options[:ssl][:client][:private_key],
+                client_ssl_certificate: options[:ssl][:client][:certificate],
             },
             paths: {
                 application: options[:application]  || Options.paths.application
@@ -113,10 +125,10 @@ class Instances
         end
     end
 
-    # Starts {RPC::Server::Dispatcher} grid and returns a high-performance Instance.
+    # Starts {RPC::Server::Agent} grid and returns a high-performance Instance.
     #
     # @param    [Hash]  options
-    # @option options [Integer] :grid_size (3)  Amount of Dispatchers to spawn.
+    # @option options [Integer] :grid_size (3)  Amount of Agents to spawn.
     #
     # @return   [RPC::Client::Instance]
     def grid_spawn(options = {} )
@@ -124,23 +136,23 @@ class Instances
 
         last_member = nil
         options[:grid_size].times do |i|
-            last_member = Dispatchers.spawn(
-                neighbour: last_member ? last_member.url : last_member,
+            last_member = Agents.spawn(
+                peer: last_member ? last_member.url : last_member,
                 pipe_id:   Utilities.available_port.to_s + Utilities.available_port.to_s
             )
         end
 
         info = nil
-        info = last_member.dispatch while !info && sleep( 0.1 )
+        info = last_member.spawn while !info && sleep( 0.1 )
 
         connect( info['url'], info['token'] )
     end
 
-    # Starts {RPC::Server::Dispatcher} and returns an Instance.
+    # Starts {RPC::Server::Agent} and returns an Instance.
     #
     # @return   [RPC::Client::Instance]
-    def dispatcher_spawn
-        info = Dispatchers.spawn.dispatch
+    def agent_spawn
+        info = Agents.spawn.spawn
         connect( info['url'], info['token'] )
     end
 

data/lib/cuboid/processes/schedulers.rb

@@ -52,16 +52,29 @@ class Schedulers
     #
     # @return   [RPC::Client::Queue]
     def spawn( options = {} )
+        options = options.dup
         fork = options.delete(:fork)
 
+        options[:ssl] ||= {
+          server: {},
+          client: {}
+        }
+
         options = {
-            dispatcher: {
-                url: options[:dispatcher],
+            agent: {
+                url:      options[:agent],
+                strategy: options[:strategy]
             },
             rpc:        {
                 server_port:             options[:port]    || Utilities.available_port,
                 server_address:          options[:address] || '127.0.0.1',
-                server_external_address: options[:external_address]
+                server_external_address: options[:external_address],
+
+                ssl_ca:                 options[:ssl][:ca],
+                server_ssl_private_key: options[:ssl][:server][:private_key],
+                server_ssl_certificate: options[:ssl][:server][:certificate],
+                client_ssl_private_key: options[:ssl][:client][:private_key],
+                client_ssl_certificate: options[:ssl][:client][:certificate],
             },
             paths: {
               application: options[:application]  || Options.paths.application

data/lib/cuboid/processes.rb

@@ -3,11 +3,11 @@ require 'ostruct'
 
 lib = Cuboid::Options.paths.lib
 require lib + 'rpc/client/instance'
-require lib + 'rpc/client/dispatcher'
+require lib + 'rpc/client/agent'
 require lib + 'rpc/client/scheduler'
 
 lib = Cuboid::Options.paths.lib + 'processes/'
 require lib + 'manager'
-require lib + 'dispatchers'
+require lib + 'agents'
 require lib + 'instances'
 require lib + 'schedulers'

data/lib/cuboid/report.rb

@@ -1,4 +1,5 @@
 require_relative 'rpc/serializer'
+require 'time'
 
 module Cuboid
 

data/lib/cuboid/rest/server/instance_helpers.rb

@@ -5,10 +5,10 @@ class Server
 module InstanceHelpers
 
     @@instances   = {}
-    @@dispatchers = {}
+    @@agents = {}
 
     def get_instance
-        if dispatcher
+        if agent
             options = {
               owner:   self.class.to_s,
               helpers: {
@@ -18,7 +18,7 @@ module InstanceHelpers
                 }
             }
 
-            if (info = dispatcher.dispatch( options ))
+            if (info = agent.spawn( options ))
                 connect_to_instance( info['url'], info['token'] )
             end
         else
@@ -26,24 +26,24 @@ module InstanceHelpers
         end
     end
 
-    def dispatchers
-        @@dispatchers.keys
+    def agents
+        @@agents.keys
     end
 
-    def dispatcher
-        return if !Options.dispatcher.url
-        @dispatcher ||= connect_to_dispatcher( Options.dispatcher.url )
+    def agent
+        return if !Options.agent.url
+        @agent ||= connect_to_agent( Options.agent.url )
     end
 
-    def unplug_dispatcher( url )
-        connect_to_dispatcher( url ).node.unplug
+    def unplug_agent( url )
+        connect_to_agent( url ).node.unplug
 
-        c = @@dispatchers.delete( url )
+        c = @@agents.delete( url )
         c.close if c
     end
 
-    def connect_to_dispatcher( url )
-        @@dispatchers[url] ||= RPC::Client::Dispatcher.new( url )
+    def connect_to_agent( url )
+        @@agents[url] ||= RPC::Client::Agent.new( url )
     end
 
     def connect_to_instance( url, token )