cuboid 0.0.3 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cbcb75c293c3807aa184d72232133f7dd4adc4fe729d61e319885915a2c61393
-  data.tar.gz: b69dcab26139c027ebc3aed19c6ddd5bd5576ca9d60b80bcc2093c43dce45900
+  metadata.gz: edae28c56b33bd872843de41aaaa5babbe2ae9e24a127357c1e3d9fdcd347d72
+  data.tar.gz: 75d6a47a8eea4743c393a4f64468baf0f986f87ec488f971a83ebc649d3b6aa2
 SHA512:
-  metadata.gz: 63bd14b98fb47ac55c5d63083c7897b1befc9dabb6fd2299c9d677a3fae9f7218636a4a2a550d145a6ab5bc691d37943262c9c6445427521b5dc1b5969ffa9ee
-  data.tar.gz: e4b741b30fe68176fe86bc84b8e4e49e5ef7cb5d908eb19871ebd76d36ae53c9aeb4f6940382f81da68a962ae9c03262e0f1504a4f6cebcef759a0303d889b1a
+  metadata.gz: fea2bee694338842dcd1e21f6e8be63b0b61cf7efe6b9a3ac83122cfd4249bda373d40c8f884716b1ddccbe4c8a40d9d655c9e46cac0da802278d354c2d09622
+  data.tar.gz: e9f9ee9c1e3ad4d5710d9d4d587ad91f3bcf5cd71eb8dd2d94789a36760dcf7aa8e3d57816671b1c9d3a071d59c54b8d9fa2fd27466340c928684e47a641c6d7

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+# 0.1.0
+
+* `Dispatcher` => `Agent`
+* `Dispatcher#dispatch` => `Agent#spawn`
+* `neighbour` => `peer`

data/Gemfile

@@ -2,8 +2,6 @@ source 'https://rubygems.org'
 
 gem 'rake', '13.0.3'
 
-gem 'arachni-reactor',      path: '../../arachni-reactor'
-
 group :docs do
     gem 'yard'
     gem 'redcarpet'

data/README.md

@@ -10,10 +10,11 @@ _Cloud_ within a _Cloud_.
 
 It offers:
 
-* Load-balancing of _**Instances**_ via a software mesh network (_**Grid**_) of _**Dispatchers**_.
-  * No need to setup a topology manually, _**Dispatchers**_ will reach
+* Load-balancing of _**Instances**_ via a network (_**Grid**_) of _**Agents**_.
+  * No need to setup a topology manually, _**Agents**_ will reach
     convergence on their own, just point them to an existing _**Grid**_ member.
   * Scaling up and down can be easily achieved by _plugging_ or _unplugging_ nodes.
+  * Horizontal (`default`) and vertical workload distribution strategies available.
   * Fault tolerant -- one application per process (_**Instance**_).
   * Self-healing -- keeps an eye out for disappearing and also re-appearing members.
 * A clean and simple framework for application development.
@@ -57,7 +58,7 @@ framework:
     * `:restore`
   * `instance_service_for( Symbol, Class )` -- Adds a custom _**Instance**_ RPC API.
   * `rest_service_for( Symbol, Module )` -- Hooks-up to the _**REST**_ service to provide a custom REST API.
-  * `dispatcher_service_for( Symbol, Class )` -- Hooks-up to the _**Dispatcher**_ to provide a custom RPC API.
+  * `agent_service_for( Symbol, Class )` -- Hooks-up to the _**Agent**_ to provide a custom RPC API.
   * `serialize_with( Module )` -- A serializer to be used for:
     * `#options`
     * `Report#data`
@@ -83,10 +84,10 @@ This is in order to enforce isolation (_state_, _data_, _fault_) between
 _**Applications**_, take advantage of _OS_ task management and generally keep
 things simple.
 
-### Dispatcher
+### Agent
 
-A _**Dispatcher**_ is a server which awaits for _**Instance**_ spawn requests
-(`dispatch` calls) upon which it spawns and passes the _**Instance**_'s
+A _**Agent**_ is a server which awaits for _**Instance**_ spawn requests
+(`spawn` calls) upon which it spawns and passes the _**Instance**_'s
 connection info to the client.
 
 The client can then proceed to use the _**Instance**_ to run and generally manage
@@ -94,7 +95,7 @@ the contained _**Application**_.
 
 #### Grid
 
-A _**Dispatcher**_ _**Grid**_ is a software mesh network of _**Dispatcher**_
+A _**Agent**_ _**Grid**_ is a software mesh network of _**Agent**_
 servers, aimed towards providing automated _load-balancing_ based on available
 system resources and each _**Application**_'s provisioning configuration.
 
@@ -103,11 +104,11 @@ providing any existing _**Grid**_ member upon start-up and the rest will be
 sorted out automatically.
 
 The network is _self-healing_ and will monitor _node_ connectivity, taking steps
-to ensure that neither server nor network conditions will disrupt dispatching.
+to ensure that neither server nor network conditions will disrupt spawning.
 
 ##### Scalability
 
-_**Dispatchers**_ can be easily _plugged_ to or _unplugged_ from the _**Grid**_
+_**Agents**_ can be easily _plugged_ to or _unplugged_ from the _**Grid**_
 to scale up or down as necessary.
 
 _Plugging_ happens at boot-time and _unplugging_ can take place via the available
@@ -124,12 +125,12 @@ The _**Scheduler**_ is a server which:
   5. Upon _**Application**_ completion stores report to disk.
   6. Shuts down the _**Instance**_.
 
-#### Dispatcher
+#### Agent
 
-The _**Scheduler**_ can be configured with a _**Dispatcher**_, upon which case,
+The _**Scheduler**_ can be configured with a _**Agent**_, upon which case,
 it will use it to spawn _**Instances**_.
 
-If the _**Dispatcher**_ is a _**Grid**_ member then the _**Scheduler**_ will
+If the _**Agent**_ is a _**Grid**_ member then the _**Scheduler**_ will
 also enjoy _load-balancing_ features. 
 
 ## APIs
@@ -153,7 +154,7 @@ RPC interface.
 A REST API is also available, taking advantage of HTTP sessions to make progress
 tracking easier.
 
-The REST interface is basically a web _**Dispatcher**_ and centralised point of
+The REST interface is basically a web _**Agent**_ and centralised point of
 management for the rest of the entities.
 
 Each _**Application**_ can extend upon this and expose an API via its _**REST**_ 

data/cuboid.gemspec

@@ -42,10 +42,6 @@ Gem::Specification.new do |s|
     # RPC serialization.
     s.add_dependency 'msgpack',             '1.1.0'
 
-    # Optimized JSON.
-    s.add_dependency 'oj',                  '3.11.5'
-    s.add_dependency 'oj_mimic_json',       '1.0.1'
-
     # Web server
     s.add_dependency 'puma',                '5.5.2'
 

data/lib/cuboid/application.rb

@@ -103,15 +103,15 @@ class Application
             @max_memory ||= 0
         end
 
-        def provision_disk( ram )
-            @max_disk = ram
+        def provision_disk( disk )
+            @max_disk = disk
         end
 
         def max_disk
-            @max_memory ||= 0
+            @max_disk ||= 0
         end
 
-        def instance_service_for(name, service )
+        def instance_service_for( name, service )
             instance_services[name] = service
         end
 
@@ -127,12 +127,12 @@ class Application
             @rest_services ||= {}
         end
 
-        def dispatcher_service_for( name, service )
-            dispatcher_services[name] = service
+        def agent_service_for( name, service )
+            agent_services[name] = service
         end
 
-        def dispatcher_services
-            @dispatcher_services ||= {}
+        def agent_services
+            @agent_services ||= {}
         end
 
         def handler_for( signal, handler )
@@ -183,8 +183,8 @@ class Application
             case type
             when :instance
                 const = :Instances
-            when :dispatcher
-                const = :Dispatchers
+            when :agent
+                const = :Agents
             when :scheduler
                 const = :Schedulers
             when :rest

data/lib/cuboid/option_groups/agent.rb

@@ -0,0 +1,54 @@
+module Cuboid::OptionGroups
+
+# Holds options for {RPC::Server::Agent} servers.
+#
+# @author Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
+class Agent < Cuboid::OptionGroup
+
+    STRATEGIES = Set.new([:horizontal, :vertical])
+
+    # @return   [String]
+    #   URL of a {RPC::Server::Agent}.
+    attr_accessor :url
+
+    # @return   [Array<Integer>]
+    #   Range of ports to use when spawning instances, first entry should be
+    #   the lowest port number, last the max port number.
+    attr_accessor :instance_port_range
+
+    # @return   [String]
+    #   The URL of a peering {RPC::Server::Agent}, applicable when
+    #   {RPC::Server::Agent} are connected to each other to form a Grid.
+    #
+    # @see RPC::Server::Agent::Node
+    attr_accessor :peer
+
+    # @return   [Float]
+    #   How regularly to check for peer statuses.
+    attr_accessor :ping_interval
+
+    # @return   [String]
+    #   Agent name.
+    attr_accessor :name
+
+    attr_accessor :strategy
+
+    set_defaults(
+        strategy:            :horizontal,
+        ping_interval:       5.0,
+        instance_port_range: [1025, 65535]
+    )
+
+    def strategy=( type )
+        return @strategy = defaults[:strategy] if !type
+
+        type = type.to_sym
+        if !STRATEGIES.include? type
+            fail ArgumentError, "Unknown strategy type: #{type}"
+        end
+
+        @strategy = type
+    end
+
+end
+end

data/lib/cuboid/option_groups/paths.rb

@@ -70,20 +70,25 @@ class Paths < Cuboid::OptionGroup
 
     # @!attribute executables
     #   @return [String]
-    #       System processes (instance, dispatcher, browser, etc.).
+    #       System processes (instance, agent, browser, etc.).
 
     def initialize
         @root = self.root_path
+        FileUtils.mkdir_p home_path
 
-        @snapshots = self.config['snapshots'] || @root + 'snapshots/'
-        @reports   = self.config['reports']   || @root + 'reports/'
+        @snapshots = self.config['snapshots'] || home_path + 'snapshots/'
+        FileUtils.mkdir_p @snapshots
+
+        @reports = self.config['reports'] || home_path + 'reports/'
+        FileUtils.mkdir_p @reports
 
         if ENV['CUBOID_LOGDIR'].to_s != ''
             @logs = "#{ENV['CUBOID_LOGDIR']}/"
         elsif self.config['logs']
             @logs = self.config['logs']
         else
-            @logs = "#{@root}logs/"
+            @logs = "#{home_path}logs/"
+            FileUtils.mkdir_p @logs
         end
 
         @lib         = @root    + 'lib/cuboid/'
@@ -98,6 +103,10 @@ class Paths < Cuboid::OptionGroup
         tmpdir
     end
 
+    def home_path
+        @home_path ||= "#{ENV['HOME']}/.cuboid/"
+    end
+
     def root_path
         self.class.root_path
     end

data/lib/cuboid/options.rb

@@ -76,7 +76,7 @@ class Options
     require_relative 'option_groups'
 
     TO_RPC_IGNORE = Set.new([
-        :instance, :rpc, :dispatcher, :queue, :paths,
+        :instance, :rpc, :agent, :queue, :paths,
         :snapshot, :report, :output, :system
     ])
 

data/lib/cuboid/processes/{dispatchers.rb → agents.rb}

@@ -2,28 +2,28 @@ module Cuboid
 module Processes
 
 #
-# Helper for managing {RPC::Server::Dispatcher} processes.
+# Helper for managing {RPC::Server::Agent} processes.
 #
 # @author Tasos "Zapotek" Laskos <tasos.laskos@gmail.com>
 #
-class Dispatchers
+class Agents
     include Singleton
     include Utilities
 
-    # @return   [Array<String>] URLs of all running Dispatchers.
+    # @return   [Array<String>] URLs of all running Agents.
     attr_reader :list
 
     def initialize
         @list = []
-        @dispatcher_connections = {}
+        @agent_connections = {}
     end
 
-    # Connects to a Dispatcher by URL.
+    # Connects to a Agent by URL.
     #
-    # @param    [String]    url URL of the Dispatcher.
+    # @param    [String]    url URL of the Agent.
     # @param    [Hash]    options Options for the RPC client.
     #
-    # @return   [RPC::Client::Dispatcher]
+    # @return   [RPC::Client::Agent]
     def connect( url, options = nil )
         Arachni::Reactor.global.run_in_thread if !Arachni::Reactor.global.running?
 
@@ -33,37 +33,51 @@ class Dispatchers
         end
 
         if fresh
-            @dispatcher_connections[url] = RPC::Client::Dispatcher.new( url, options )
+            @agent_connections[url] = RPC::Client::Agent.new( url, options )
         else
-            @dispatcher_connections[url] ||= RPC::Client::Dispatcher.new( url, options )
+            @agent_connections[url] ||= RPC::Client::Agent.new( url, options )
         end
     end
 
-    # @param    [Block] block   Block to pass an RPC client for each Dispatcher.
+    # @param    [Block] block   Block to pass an RPC client for each Agent.
     def each( &block )
         @list.each do |url|
             block.call connect( url )
         end
     end
 
-    # Spawns a {RPC::Server::Dispatcher} process.
+    # Spawns a {RPC::Server::Agent} process.
     #
     # @param    [Hash]  options
     #   To be passed to {Cuboid::Options#set}. Allows `address` instead of
     #   `rpc_server_address` and `port` instead of `rpc_port`.
     #
-    # @return   [RPC::Client::Dispatcher]
+    # @return   [RPC::Client::Agent]
     def spawn( options = {} )
+        options = options.dup
         fork = options.delete(:fork)
 
+        options[:ssl] ||= {
+          server: {},
+          client: {}
+        }
+
         options = {
-            dispatcher: {
-                neighbour: options[:neighbour],
+            agent: {
+                name:      options[:name],
+                peer: options[:peer],
+                strategy:  options[:strategy],
             },
             rpc:        {
                 server_port:             options[:port]    || Utilities.available_port,
                 server_address:          options[:address] || '127.0.0.1',
-                server_external_address: options[:external_address]
+                server_external_address: options[:external_address],
+
+                ssl_ca:                 options[:ssl][:ca],
+                server_ssl_private_key: options[:ssl][:server][:private_key],
+                server_ssl_certificate: options[:ssl][:server][:certificate],
+                client_ssl_private_key: options[:ssl][:client][:private_key],
+                client_ssl_certificate: options[:ssl][:client][:certificate],
             },
             paths: {
                 application: options[:application] || Options.paths.application
@@ -74,11 +88,11 @@ class Dispatchers
             options[:rpc].delete :server_external_address
         end
 
-        if options[:dispatcher][:neighbour].nil?
-            options[:dispatcher].delete :neighbour
+        if options[:agent][:peer].nil?
+            options[:agent].delete :peer
         end
 
-        pid = Manager.spawn( :dispatcher, options: options, fork: fork )
+        pid = Manager.spawn( :agent, options: options, fork: fork )
 
         url = "#{options[:rpc][:server_address]}:#{options[:rpc][:server_port]}"
         while sleep( 0.1 )
@@ -96,26 +110,26 @@ class Dispatchers
 
     def grid_spawn( options = {} )
         d = spawn( options )
-        spawn( options.merge neighbour: d.url )
+        spawn( options.merge peer: d.url )
     end
 
-    # @note Will also kill all Instances started by the Dispatcher.
+    # @note Will also kill all Instances started by the Agent.
     #
-    # @param    [String]    url URL of the Dispatcher to kill.
+    # @param    [String]    url URL of the Agent to kill.
     def kill( url )
-        dispatcher = connect( url )
-        Manager.kill_many dispatcher.statistics['consumed_pids']
-        Manager.kill dispatcher.pid
+        agent = connect( url )
+        Manager.kill_many agent.statistics['consumed_pids']
+        Manager.kill agent.pid
     rescue => e
         #ap e
         #ap e.backtrace
         nil
     ensure
         @list.delete( url )
-        @dispatcher_connections.delete( url )
+        @agent_connections.delete( url )
     end
 
-    # Kills all {Dispatchers #list}.
+    # Kills all {Agents #list}.
     def killall
         @list.dup.each do |url|
             kill url

data/lib/cuboid/processes/executables/agent.rb

@@ -0,0 +1,5 @@
+require Options.paths.lib  + 'rpc/server/agent'
+
+Arachni::Reactor.global.run do
+    RPC::Server::Agent.new
+end

data/lib/cuboid/processes/helpers/agents.rb

@@ -0,0 +1,23 @@
+# @param (see Cuboid::Processes::Agents#spawn)
+# @return (see Cuboid::Processes::Agents#spawn)
+def agent_spawn( *args )
+    Cuboid::Processes::Agents.spawn( *args )
+end
+
+# @param (see Cuboid::Processes::Agents#kill)
+# @return (see Cuboid::Processes::Agents#kill)
+def agent_kill( *args )
+    Cuboid::Processes::Agents.kill( *args )
+end
+
+# @param (see Cuboid::Processes::Agents#killall)
+# @return (see Cuboid::Processes::Agents#killall)
+def agent_killall
+    Cuboid::Processes::Agents.killall
+end
+
+# @param (see Cuboid::Processes::Agents#connect)
+# @return (see Cuboid::Processes::Agents#connect)
+def agent_connect( *args )
+    Cuboid::Processes::Agents.connect( *args )
+end

data/lib/cuboid/processes/helpers/instances.rb

@@ -10,10 +10,10 @@ def instance_grid_spawn( *args )
     Cuboid::Processes::Instances.grid_spawn( *args )
 end
 
-# @param (see Cuboid::Processes::Instances#dispatcher_spawn)
-# @return (see Cuboid::Processes::Instances#dispatcher_spawn)
-def instance_dispatcher_spawn( *args )
-    Cuboid::Processes::Instances.dispatcher.spawn( *args )
+# @param (see Cuboid::Processes::Instances#agent_spawn)
+# @return (see Cuboid::Processes::Instances#agent_spawn)
+def instance_agent_spawn( *args )
+    Cuboid::Processes::Instances.agent.spawn( *args )
 end
 
 def instance_kill( url )

data/lib/cuboid/processes/helpers.rb

@@ -1,4 +1,4 @@
 require_relative 'helpers/processes'
-require_relative 'helpers/dispatchers'
+require_relative 'helpers/agents'
 require_relative 'helpers/instances'
 require_relative 'helpers/schedulers'

data/lib/cuboid/processes/instances.rb

@@ -21,7 +21,7 @@ class Instances
     #
     # Connects to a Instance by URL.
     #
-    # @param    [String]    url URL of the Dispatcher.
+    # @param    [String]    url URL of the Agent.
     # @param    [String]    token
     #   Authentication token -- only need be provided once.
     #
@@ -61,16 +61,28 @@ class Instances
     # @return   [RPC::Client::Instance, Integer]
     #   RPC client and PID.
     def spawn( options = {}, &block )
+        options = options.dup
         token = options.delete(:token) || Utilities.generate_token
         fork  = options.delete(:fork)
 
         port_range = options.delete( :port_range )
 
+        options[:ssl] ||= {
+          server: {},
+          client: {}
+        }
+
         options = {
             rpc:    {
                 server_socket:  options[:socket],
                 server_port:    options[:port]    || Utilities.available_port( port_range ),
-                server_address: options[:address] || '127.0.0.1'
+                server_address: options[:address] || '127.0.0.1',
+
+                ssl_ca:                 options[:ssl][:ca],
+                server_ssl_private_key: options[:ssl][:server][:private_key],
+                server_ssl_certificate: options[:ssl][:server][:certificate],
+                client_ssl_private_key: options[:ssl][:client][:private_key],
+                client_ssl_certificate: options[:ssl][:client][:certificate],
             },
             paths: {
                 application: options[:application]  || Options.paths.application
@@ -113,10 +125,10 @@ class Instances
         end
     end
 
-    # Starts {RPC::Server::Dispatcher} grid and returns a high-performance Instance.
+    # Starts {RPC::Server::Agent} grid and returns a high-performance Instance.
     #
     # @param    [Hash]  options
-    # @option options [Integer] :grid_size (3)  Amount of Dispatchers to spawn.
+    # @option options [Integer] :grid_size (3)  Amount of Agents to spawn.
     #
     # @return   [RPC::Client::Instance]
     def grid_spawn(options = {} )
@@ -124,23 +136,23 @@ class Instances
 
         last_member = nil
         options[:grid_size].times do |i|
-            last_member = Dispatchers.spawn(
-                neighbour: last_member ? last_member.url : last_member,
+            last_member = Agents.spawn(
+                peer: last_member ? last_member.url : last_member,
                 pipe_id:   Utilities.available_port.to_s + Utilities.available_port.to_s
             )
         end
 
         info = nil
-        info = last_member.dispatch while !info && sleep( 0.1 )
+        info = last_member.spawn while !info && sleep( 0.1 )
 
         connect( info['url'], info['token'] )
     end
 
-    # Starts {RPC::Server::Dispatcher} and returns an Instance.
+    # Starts {RPC::Server::Agent} and returns an Instance.
     #
     # @return   [RPC::Client::Instance]
-    def dispatcher_spawn
-        info = Dispatchers.spawn.dispatch
+    def agent_spawn
+        info = Agents.spawn.spawn
         connect( info['url'], info['token'] )
     end
 

data/lib/cuboid/processes/schedulers.rb

@@ -52,16 +52,29 @@ class Schedulers
     #
     # @return   [RPC::Client::Queue]
     def spawn( options = {} )
+        options = options.dup
         fork = options.delete(:fork)
 
+        options[:ssl] ||= {
+          server: {},
+          client: {}
+        }
+
         options = {
-            dispatcher: {
-                url: options[:dispatcher],
+            agent: {
+                url:      options[:agent],
+                strategy: options[:strategy]
             },
             rpc:        {
                 server_port:             options[:port]    || Utilities.available_port,
                 server_address:          options[:address] || '127.0.0.1',
-                server_external_address: options[:external_address]
+                server_external_address: options[:external_address],
+
+                ssl_ca:                 options[:ssl][:ca],
+                server_ssl_private_key: options[:ssl][:server][:private_key],
+                server_ssl_certificate: options[:ssl][:server][:certificate],
+                client_ssl_private_key: options[:ssl][:client][:private_key],
+                client_ssl_certificate: options[:ssl][:client][:certificate],
             },
             paths: {
               application: options[:application]  || Options.paths.application

data/lib/cuboid/processes.rb

@@ -3,11 +3,11 @@ require 'ostruct'
 
 lib = Cuboid::Options.paths.lib
 require lib + 'rpc/client/instance'
-require lib + 'rpc/client/dispatcher'
+require lib + 'rpc/client/agent'
 require lib + 'rpc/client/scheduler'
 
 lib = Cuboid::Options.paths.lib + 'processes/'
 require lib + 'manager'
-require lib + 'dispatchers'
+require lib + 'agents'
 require lib + 'instances'
 require lib + 'schedulers'

data/lib/cuboid/report.rb

@@ -1,4 +1,5 @@
 require_relative 'rpc/serializer'
+require 'time'
 
 module Cuboid
 

data/lib/cuboid/rest/server/instance_helpers.rb

@@ -5,10 +5,10 @@ class Server
 module InstanceHelpers
 
     @@instances   = {}
-    @@dispatchers = {}
+    @@agents = {}
 
     def get_instance
-        if dispatcher
+        if agent
             options = {
               owner:   self.class.to_s,
               helpers: {
@@ -18,7 +18,7 @@ module InstanceHelpers
                 }
             }
 
-            if (info = dispatcher.dispatch( options ))
+            if (info = agent.spawn( options ))
                 connect_to_instance( info['url'], info['token'] )
             end
         else
@@ -26,24 +26,24 @@ module InstanceHelpers
         end
     end
 
-    def dispatchers
-        @@dispatchers.keys
+    def agents
+        @@agents.keys
     end
 
-    def dispatcher
-        return if !Options.dispatcher.url
-        @dispatcher ||= connect_to_dispatcher( Options.dispatcher.url )
+    def agent
+        return if !Options.agent.url
+        @agent ||= connect_to_agent( Options.agent.url )
     end
 
-    def unplug_dispatcher( url )
-        connect_to_dispatcher( url ).node.unplug
+    def unplug_agent( url )
+        connect_to_agent( url ).node.unplug
 
-        c = @@dispatchers.delete( url )
+        c = @@agents.delete( url )
         c.close if c
     end
 
-    def connect_to_dispatcher( url )
-        @@dispatchers[url] ||= RPC::Client::Dispatcher.new( url )
+    def connect_to_agent( url )
+        @@agents[url] ||= RPC::Client::Agent.new( url )
     end
 
     def connect_to_instance( url, token )