cloud_powers 0.2.4 → 0.2.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 536b4abf5027aa70d284971a54fa2ec662cc8983
-  data.tar.gz: 8bd347c9a99d78f4dce7dec27b837dfa6bd21e22
+  metadata.gz: 56d3873e3042a66d3fd0b284b3b5fe88f49803fb
+  data.tar.gz: dd5d52611cc92cd66c52a08ecb905e0bc0d28788
 SHA512:
-  metadata.gz: 78dd9784282f4d316dce04003efb37cf579b3a52f9c2bd5187f92c6c3afde853cfdeb2038e079a489bf1fcd9fa360121c920ba74829575e0a31fddd02ff8dc9d
-  data.tar.gz: b59c25e12a62757894c46cc1fdf00be0a3b4ff79b06001f18cd9118af2eaeea4c961e2a646b57841d9007c45918ee91bfeecaaeb6d0b0f40dd3d57a9c00953f2
+  metadata.gz: f72cd7f864047a6b1c5060b31fb2d2f75a17df0b22ca72bdc6c545c39091a0e622d94d5db42e2f65840e29c0efd86b5bf2ac98501c7bafa5db4c4466db00f87a
+  data.tar.gz: b0403c5ae413ecbf7b94e0c4296384427b38e96381bf2d6c471cf1e8ae8190225e0e6439de7a66ab5b7a20ce83094e369cf5dc593af2355e875f02c2a0e141f9

data/.gitignore

@@ -11,3 +11,4 @@
 .byebug_history
 lib/cloud_powers/synapse/.DS_Store
 .DS_Store
+./**/.DS_Store

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    cloud_powers (0.2.4)
+    cloud_powers (0.2.5)
       activesupport-core-ext (~> 4)
       aws-sdk (~> 2)
       dotenv (~> 2.1)

data/README.md

@@ -2,7 +2,13 @@
 
 ## Description
 
-CloudPowers is a wrapper around certain AWS services.  It was developed with the _need_ instead of the _resource_ in mind so even though AWS is the only service provider included now, it shouldn't be the only one forever.  There are several pieces of the Cloud Power module to talk about: SelfAwareness, Synapse, Delegator and more.  Each module in Cloud Powers is in charge of a specific type of task that helps bring projects together and communicate with the outside world.
+CloudPowers is a wrapper around certain AWS services.  It was developed with the _need_ instead of the _resource_ in mind so even though AWS is the only service provider included now, it shouldn't be the only one forever.  There are several pieces of the Cloud Power module to talk about:
+* SelfAwareness: Gathers data about "self"
+* Synapse: Handles communication needs
+* Delegator: dynamically include Ruby executables from S3
+and more...
+
+* Each module in Cloud Powers is in charge of a specific type of task that helps bring projects together and communicate with the outside world.
 
 Below is a breakdown of the installation, common services provided and an example or 2 of popular methods.
 _Better docs are on the way_
@@ -11,7 +17,7 @@ _Better docs are on the way_
 
 Add this line to your application's Gemfile:
 
-```ruby
+```Ruby
 gem 'cloud_powers'
 ```
 
@@ -26,17 +32,21 @@ Or install it yourself as:
 then either:
     * set environment variables that matches the below group
     * fill out a .env file and load it from the class that is using CloudPowers, like this
-```ruby
+    Notes:
+      * The code does its best in many cases to make a good guess at some of the configuration
+        for you but if you set them in your system environment variables or in the .env file,
+        it'll have a much better
+```Ruby
 require 'dotenv'
 Dotenv.load('path to your .env file')
 ```
 _things you need for pre-v1_:
-```
+```Ruby
 # Aws access:
 AWS_ACCESS_KEY_ID=""
 AWS_SECRET_ACCESS_KEY=""
 
-# Aws areas and auth-related locations: 
+# Aws areas and auth-related locations:
 AWS_REGION=""
 
 # Aws Build info:
@@ -72,11 +82,11 @@ FINISHED_QUEUE_ADDRESS=""
 
 ### SelfAwareness
   * gets and sets info about the instance -> Neuron/Cerebrum/etc)
-```ruby
+```Ruby
     get_awareness!
 ```
   * retrieves and sets all metadata from the EC2 instance and a few other things like the instance hostname (can find the instance IP from here).
-  * See EC2 metadata for details on the instance/node metadata that is set.  
+  * See EC2 metadata for details on the instance/node metadata that is set.
   * Additionally, the instance public hostname, id and a few others are set using other-than-EC2-metadata methods.
 
 
@@ -97,7 +107,7 @@ FINISHED_QUEUE_ADDRESS=""
     * like a task list or a message board for asynchronous communication
     * Board <Struct>:
       * interface with Queue config, data, name, etc.
-      ```ruby
+      ```Ruby
         default_workflow = Workflow.new
         board = Board.new(default_workflow)
         board.name
@@ -110,15 +120,15 @@ FINISHED_QUEUE_ADDRESS=""
     Example usage:
       1. Give the entire stream name or a symbol or string that is found in the .env for the name of the Queue
       2. provide a block that will create the record that gets sent to the board
-    ```ruby
+    ```Ruby
       poll(board_name <string|symbol>, opts <optional config Hash>) { block }
     ```
     or
-    ```ruby
+    ```Ruby
       poll(:backlog) { |msg, stats| Task.new(instance_id, msg) if stats.success? }
     ```
     or
-    ```ruby
+    ```Ruby
       poll(:backlog, wait_time: 30, delete: false) do
         edited_message = sitrep.merge(foo: 'bar')
         update = some_method(edited_message)
@@ -126,26 +136,26 @@ FINISHED_QUEUE_ADDRESS=""
     ```
   #### Pipe
     * Good for real time information passing, like status updates, results reporting, operations, etc.
-    * The Pipe module is for communicating via streams or webhooks.  Piping is meant to be a way to communicate status, problems and  other general info.  There can be very high traffic through the pipe or none at all.  Very soon, Cerebrums will be data consumers, to the Java KCL and MultiLangDaemon level, so keeping messages straight is done via partition ID.  The partition ID of any message is to identify which node the message is about, and so the instance ID is used in nodes like the Neuron and Cerebrum and other identifiers that are deemed best are used in other projects.
+    * The Pipe module is for communicating via streams.  Piping is meant to be a way to communicate status, problems and  other general info or huge result sets and things of that nature.  There can be very high traffic through the pipe or none at all.  Very soon (probably V-0.2.6 or 7), Cerebrums will be data consumers, to the Java KCL and MultiLangDaemon level, so keeping messages straight is done via partition ID.  The partition ID of any message is to identify which node the message is about or the "batch" or "group" or any other concept like that. So the instance ID is used in nodes like the Neuron and Cerebrum and other identifiers that are deemed best are used in other projects as a batch ID.
 
     Example usage:
-    ruby```
+    ```Ruby
       pipe_to(stream_name <string/symbol>) { &block }
     ```
-    ruby```
+    ```Ruby
       pipe_to(:status_queue) { sitrep(content: 'workflowComplete') }
     ```
     and for multiple records (KCL)
-    ```ruby
+    ```Ruby
       flow_to(stream_name <string/symbol>) { &block }
     ```
-    ```ruby
+    ```Ruby
       flow_to(:status_queue) do
         interesting_instances = neurons.map do |neuron|
           return neuron if neuron.workflow.done?
         end
         find_efficient_neurons(interesting_instances) # this gets sent through the Pipe
-      end 
+      end
     ```
 
 
@@ -153,10 +163,6 @@ FINISHED_QUEUE_ADDRESS=""
     * Allows the nodes to have a collective awareness of each other, the environment they work in and other Jobs (Coming soon...)
 
 
-### Workflow
-  * The workflow is one of the integral ideas behind the Brain architecture.  The workflow describes to us humans what state the Job/Task/etc is currently in and it let's the App(s) know what things it can do from any point in the process.
-
-  
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/cloud_powers.gemspec

@@ -4,30 +4,31 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require 'cloud_powers/version'
 
 Gem::Specification.new do |spec|
-  spec.required_ruby_version  =       '~> 2.3.0'
-  spec.name                   =       'cloud_powers'
-  spec.version                =       CloudPowers::VERSION
-  spec.author                 =       'Adam Phillipps'
-  spec.email                  =       'adam.phillipps@gmail.com'
-  spec.summary                =       %q{Cloud providers wrapper.  Currently only AWS is supported.}
-  spec.description            =       <<-EOF
+  spec.required_ruby_version  =     '~> 2.3.0'
+  spec.name                   =     'cloud_powers'
+  spec.version                =     CloudPowers::VERSION
+  spec.author                 =     'Adam Phillipps'
+  spec.email                  =     'adam.phillipps@gmail.com'
+  spec.summary                =     %q{Cloud providers wrapper.  Currently only AWS is supported.}
+  spec.description            =     <<-EOF
     CloudPowers is a wrapper around AWS and in the future, other cloud service Providers.
     It was developed specifically for the Brain project but hopefully can be used
     in any other ruby project that needs to use cloud service providers' resources.
-    Version 0.2.3 has a little EC2, S3, SQS, SNS and Kinesis.  V 0.2.4 includes
-    a Context a Workflow revamp.  Websockets will be next.
+    Version 0.2.5 has a little EC2, S3, SQS, SNS and Kinesis and some a few other
+    features you can find in the docs.
+    The next versions will have websockets, IoT, more Kinesis and Workflow integration.
     This project is actively being developed, so more additions, specs and docs
-    will be updated frequently with new funcionality but the gem will follow good
-    practices for versioning.  I always welcome input.
+    will be added and updated frequently with new funcionality but the gem will
+    follow good practices for versioning.  I always welcome input.
     Enjoy!
   EOF
-  spec.homepage               =       'https://github.com/adam-phillipps/cloud_powers'
-  spec.license                =       'MIT'
+  spec.homepage               =     'https://github.com/adam-phillipps/cloud_powers'
+  spec.license                =     'MIT'
 
-  spec.files                  =      `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
-  spec.bindir                 =      'exe'
-  spec.executables            =      spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
-  spec.require_paths          =      ['lib']
+  spec.files                  =     `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir                 =     'exe'
+  spec.executables            =     spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths          =     ['lib']
 
   spec.add_runtime_dependency       'activesupport-core-ext', '~> 4'
   spec.add_runtime_dependency       'aws-sdk', '~> 2'

data/lib/cloud_powers/aws_resources.rb

@@ -13,43 +13,63 @@ module Smash
         zfind(:aws_region) || 'us-west-2'
       end
 
-      def ec2
-        @ec2 ||= Aws::EC2::Client.new(
+      def ec2(opts = {})
+        config = {
+          stub_responses: false,
           region: region,
           credentials: Auth.creds
-        )
+        }
+        config = config.merge(opts.select { |k| config.key?(k) })
+
+        @ec2 ||= Aws::EC2::Client.new(config)
       end
 
-      def image(name)
-        ec2.describe_images(
+      def image(name, opts = {})
+        config = {
           filters: [{ name: 'tag:aminame', values: [name.to_s] }]
-        ).images.first
+        }
+        config = config.merge(opts.select { |k| config.key?(k) })
+
+        ec2(opts).describe_images(config).images.first
       end
 
-      def kinesis
-        @kinesis ||= Aws::Kinesis::Client.new(
+      def kinesis(opts = {})
+        config = {
+          stub_responses: false,
           region: region,
-          credentials: Auth.creds,
-        )
+          credentials: Auth.creds
+        }
+        config = config.merge(opts.select { |k| config.key?(k) })
+
+        @kinesis ||= Aws::Kinesis::Client.new(config)
       end
 
-      def s3
-        @s3 ||= Aws::S3::Client.new(
+      def s3(opts = {})
+        config = {
+          stub_responses: false,
           region: region,
           credentials: Auth.creds
-        )
+        }
+        config = config.merge(opts.select { |k| config.key?(k) })
+
+        @s3 ||= Aws::S3::Client.new(config)
       end
 
-      def sns
-        @sns ||= Aws::SNS::Client.new(
+      def sns(opts = {})
+        config = {
+          stub_responses: false,
           region: region,
           credentials: Auth.creds
-        )
+        }
+        config = config.merge(opts.select { |k| config.key?(k) })
+
+        @sns ||= Aws::SNS::Client.new(config)
       end
 
-      def sqs
-        @sqs ||= Aws::SQS::Client.new(
-          credentials: Auth.creds
+      def sqs(opts = {})
+        @sqs ||= Aws::SQS::Client.new({
+            credentials: Auth.creds
+          }.merge(opts)
         )
       end
     end

data/lib/cloud_powers/context.rb

@@ -28,7 +28,6 @@ module Smash
       # === @returns
       #   `Smash::Context`
       def initialize(args)
-        byebug unless valid_args?(args)
         unless valid_args?(args)
           raise ArgumentError.new 'Can be either a Hash, JSON, or an Enumerable ' +
             "arguments: #{args}"
@@ -63,9 +62,10 @@ module Smash
         @structure.to_a
       end
 
-      #
+      # A Hash that represents the resources and some configuration for them
+      # === @returns Hash
       def structure
-        symbolize_keys(@structure)
+        modify_keys_with(@structure) { |key| key.to_sym }
       end
 
       # Valid scheme for @structure is assured by running the arguments through

data/lib/cloud_powers/helper.rb

@@ -8,36 +8,112 @@ module Smash
   module CloudPowers
     module Helper
 
+      # Sets an Array of instance variables, individually to a value that a
+      # user given block returns.
+      # === @params Array
+      #   * each object will be used as the name for the instance variable that
+      #     your block returns
+      # === @&block (optional)
+      #   * this is called for each object in the Array and is used as the value
+      #   for the instance variable that is being named and created for each key
+      # === @return Array
+      #   * each object will either be the result of `#instance_variable_set(key, value)`
+      #     or instance_variable_get(key)
+      # === Sampel use:
+      #   keys = ['foo', 'bar', 'yo']
+      #
+      #   attr_map!(keys) { |key| sleep 1; "#{key}:#{Time.now.to_i}" }
+      #   # => ['foo:1475434058', 'bar:1475434059', 'yo:1475434060']
+      #
+      #   puts @bar
+      #   # => 'bar:1475434059'
       def attr_map!(keys)
-        keys.map do |attr|
-          key = to_i_var(attr)
+        keys.map do |key|
+          new_i_var = to_i_var(key)
           value = yield key if block_given?
-          instance_variable_set(key, value) unless instance_variable_get(to_i_var(key))
+          instance_variable_set(new_i_var, value) unless instance_variable_get(new_i_var)
         end
       end
 
       # This is a way to find out if you are trying to work with a resource
       # available to CloudPowers
       # === @returns <Array>
+      #   * Use `.constants` to find all the modules and classes available.
+      # Notes:
+      #   TODO: make this smartly pick up all the objects, within reason and
+      #   considering need, that we have access to
       def available_resources
         [:Task].concat(Smash::CloudPowers.constants)
       end
 
+      # Does its best job at guessing where this method was called from, in terms
+      # of where it is located on the file system.  It helps track down where a
+      # project root is etc.
       def called_from
         File.expand_path(File.dirname($0))
       end
 
+      # creates a default logger
+      # Notes:
+      #   * TODO: at least make this have overridable defaults
       def create_logger
         logger = Logger.new(STDOUT)
         logger.datetime_format = '%Y-%m-%d %H:%M:%S'
         logger
       end
 
+      # Allows you to modify all keys, including nested, with a block that you pass.
+      # If no block is passed, a copy is returned.
+      # === @params
+      #   * params Hash|Array
+      # === @&block (optional)
+      #   * should modify the key and return that value so it can be used in the copy
+      # === @returns
+      #   * a copy of the given Array or Hash, with all Hash keys modified
+      # === Sample use:
+      #   hash = { 'foo' => 'v1', 'bar' => { fleep: { 'florp' => 'yo' } } }
+      #   modify_keys_with(hash) { |key| key.to_sym }
+      #   # => { foo: 'v1', bar: { fleep: { florp: 'yo' } } }
+      # === Notes:
+      #   * see `#modify_keys_with()` for handling first-level keys
+      #   * see `#pass_the_buck()` for the way nested structures are handled
+      #   * case for different types taken from _MultiXML_ (multi_xml.rb)
+      #   * TODO: look at optimization
+      def deep_modify_keys_with(params)
+        case params
+        when Hash
+          params.inject({}) do |carry, (k, v)|
+            carry.tap do |h|
+              if block_given?
+                key = yield k
+
+                value = if v.kind_of?(Hash)
+                    deep_modify_keys_with(v) do |new_key|
+                      Proc.new.call(new_key)
+                    end
+                  else
+                    v
+                  end
+
+                h[key] = value
+              else
+                h[k] = v
+              end
+            end
+          end
+        when Array
+          params.map{ |value| symbolize_keys(value) }
+        else
+          params
+        end
+      end
+
       def errors
-        # TODO: needs work
+        # TODO: wow...needs work
         $errors ||= SmashError.instance
       end
 
+      # Join the message and backtrace into a String with line breaks
       def format_error_message(error)
         begin
           [error.message, error.backtrace.join("\n")].join("\n")
@@ -58,11 +134,35 @@ module Smash
         @logger ||= create_logger
       end
 
+      # Allows you to modify all first-level keys with a block that you pass.
+      # If no block is passed, a copy is returned.
+      # === @params
+      #   * params Hash|Array
+      # === @&block (optional)
+      #   * should modify the key and return that value so it can be used in the copy
+      # === @returns
+      #   * a copy of the given Array or Hash, with all Hash keys modified
+      # === Sample use:
+      #   hash = { 'foo' => 'v1', 'bar' => { fleep: { 'florp' => 'yo' } } }
+      #   modify_keys_with(hash) { |k| k.to_sym }
+      #   # => { :foo => 'v1', :bar => { fleep: { 'florp' => 'yo' } } }
+      # === Notes:
+      #   * see `#deep_modify_keys_with()` for handling nested keys
+      #   * case for different types taken from _MultiXML_ (multi_xml.rb)
+      def modify_keys_with(params)
+        params.inject({}) do |carry, (k, v)|
+          carry.tap do |h|
+            key = block_given? ? (yield k) : k
+            h[key] = v
+          end
+        end
+      end
+
       # Lets you retry a piece of logic with 1 second sleep in between attempts
       # until another bit of logic does what it's supposed to, kind of like
       # continuing to poll something and doing something when a package is ready
       # to be taken and processed.
-      # @params:
+      # === @params:
       #   * [allowed_attempts] or Infinity(default) <Number>: The number of times
       #       the loop should be allowed to...well, loop, before a failed retry occurs.
       #   * &test <Block>: A predicate method or block of code that is callable
@@ -81,14 +181,10 @@ module Smash
         end
       end
 
-      def symbolize_keys(hash)
-        hash.inject({}) { |carry, (k, v)| carry.tap { |h| h[k.to_sym] = v } }
-      end
-
       # Gives the path from the project root to lib/tasks[/#{file}.rb]
-      # @params:
+      # === @params:
       #   * [file] <String>: name of a file
-      # @returns:
+      # === @returns:
       #   * path[/file] <String>
       #   * If a `file` is given, it will have a '.rb' file extension
       #   * If no `file` is given, it will return the `#task_require_path`
@@ -98,16 +194,20 @@ module Smash
       end
 
       # Gives the path from the project root to lib/tasks[/file]
-      # @params:
-      #   * [file] <String>: name of a file
-      # @returns:
-      #   * path[/file] <String>
+      # === @params String (optional)
+      #   * file_name name of a file
+      # ===  @returns:
+      #   * path[/file] String
       #   * Neither path nor file will have a file extension
       def task_require_path(file_name = '')
         file = File.basename(file_name, File.extname(file_name))
         Pathname(__FILE__).parent.dirname + 'tasks' + file
       end
 
+      # Change strings into camelCase
+      # === @params var String
+      # === @returns String
+      #     * givenString
       def to_camel(var)
         var = var.to_s unless var.kind_of? String
         step_one = to_snake(var)
@@ -115,20 +215,56 @@ module Smash
         step_two[0, 1].downcase + step_two[1..-1]
       end
 
+      # Change strings hyphen-delimited-string
+      # === @params var String
+      # === @returns String
+      #     * given-string
+      def to_hyph(var)
+        var = var.to_s unless var.kind_of? String
+
+        var.gsub(/:{2}|\//, '-').
+          gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
+          gsub(/([a-z\d])([A-Z])/,'\1_\2').
+          gsub(/\s+/, '-').
+          tr("_", "-").
+          gsub(/^\W/, '').
+          downcase
+      end
+
+      # Change strings into snake_case and add '@' at the front
+      # === @params var String
+      # === @returns String
+      #     * @given_string
       def to_i_var(var)
         var = var.to_s unless var.kind_of? String
         /^\W*@\w+/ =~ var ? to_snake(var) : "@#{to_snake(var)}"
       end
 
+      # Change strings into PascalCase
+      # === @params var String
+      # === @returns String
+      #     * givenString
       def to_pascal(var)
         var = var.to_s unless var.kind_of? String
         var.gsub(/^(.{1})|\W.{1}|\_.{1}/) { |s| s.gsub(/[^a-z0-9]+/i, '').capitalize }
       end
 
+      # Change strings into a ruby_file_name with extension
+      # === @params var String
+      # === @returns String
+      #     * given_string.rb
+      #     * includes ruby file extension
+      #     * see #to_snake()
       def to_ruby_file_name(name)
         name[/\.rb$/].nil? ? "#{to_snake(name)}.rb" : "#{to_snake(name)}"
       end
 
+      # Change strings into snake_case
+      # === @params var String
+      # === @returns String
+      #     * given_string
+      #     * will not have file extensions
+      #     * see #to_ruby_file_name()
       def to_snake(var)
         var = var.to_s unless var.kind_of? String
 
@@ -140,13 +276,17 @@ module Smash
           downcase
       end
 
+      # This method provides a default overrideable message body for things like
+      # basic status updates.
+      # === @params Hash
+      #   - instanceId:
+      # === Notes:
+      #   - camel casing is used on the keys because most other languages prefer
+      #   that and it's not a huge problem in ruby.  Besides, there's some other
+      #   handy methods in this module to get you through those issues, like
+      #   `#to_snake()` and or `#symbolize_keys_with()`
       def update_message_body(opts = {})
-        # TODO: find better implementation of merging nested hashes
-        # this should be fixed with Job #sitrep_message
-        # TODO: find a way to trim this method down and get rid
-        # of a lof of the repitition with these messages
-        # IDEA: throw events and have a separate thread listening. the separate
-        # thread could be a communication or status update thread
+        # TODO: Better implementation of merging message bodies and config needed
         unless opts.kind_of? Hash
           update = opts.to_s
           opts[:extraInfo] = { message: update }

data/lib/cloud_powers/node.rb

@@ -0,0 +1,60 @@
+require_relative 'auth'
+require_relative 'helper'
+require_relative 'zenv'
+
+module Smash
+  module CloudPowers
+    module Node
+      include Smash::CloudPowers::Auth
+      include Smash::CloudPowers::Helper
+      include Smash::CloudPowers::Zenv
+      # These are sensible defaults that can be overriden by providing a Hash as a param.
+      # @params [opts <Hash>]
+      #   the opts Hash should have values that should be used instead of the given
+      #   configuration.
+      def node_config(opts = {})
+        {
+          dry_run:                  zfind(:testing) || false,
+          image_id:                 image('crawlbotprod').image_id, # image(:neuron).image_id
+          instance_type:            't2.nano',
+          min_count:                opts[:max_count] || 0,
+          max_count:                0,
+          key_name:                 'crawlBot',
+          security_groups:          ['webCrawler'],
+          security_group_ids:       ['sg-940edcf2'],
+          placement:                { availability_zone: 'us-west-2c' },
+          disable_api_termination:  'false',
+          instance_initiated_shutdown_behavior:   'terminate'
+        }.merge(opts)
+      end
+
+      # Uses `Aws::EC2#run_instances` to create nodes (Neurons or Cerebrums), at
+      # a rate of 0..(n <= 100) at a time, until the required number of instances
+      # has been started.  The #instance_config() method is used to create instance
+      # configuration for the #run_instances method by using the opts hash that was
+      # provided as a parameter.
+      #
+      # === @params opts Hash (optional)
+      #   an optional instance configuration hash can be passed, which will override
+      #   the values in the default configuration returned by #instance_config()
+      def spin_up_neurons(opts = {})
+        ids = nil
+        begin
+          response = ec2.run_instances(node_config(opts))
+          ids = response.instances.map(&:instance_id)
+
+          ec2.wait_until(:instance_running, instance_ids: ids) do
+            logger.info "waiting for #{ids.count} Neurons to start..."
+          end
+
+          # tag(ids, { key: 'task', value: to_camel(self.class.to_s) })
+        rescue Aws::EC2::Errors::DryRunOperation => e
+          ids = (1..(opts[:max_count] || 0)).to_a.map { |n| n.to_s }
+          logger.info "waiting for #{ids.count} Neurons to start..."
+        end
+
+        ids
+      end
+    end
+  end
+end

data/lib/cloud_powers/self_awareness.rb

@@ -17,14 +17,14 @@ module Smash
 
       # Gets the instance time or the time it was called and as seconds from
       # epoch
+      # === @returns Integer
       # TODO: use time codes
       def boot_time
         begin
           @boot_time ||=
-            ec2.describe_instances(dry_run: zfind('testing'), instance_ids:[@instance_id]).
+            ec2.describe_instances(dry_run: zfind(:testing), instance_ids:[instance_id]).
               reservations[0].instances[0].launch_time.to_i
-        # rescue Aws::EC2::Errors::DryRunOperation => e
-        rescue Exception => e
+        rescue Aws::EC2::Errors::DryRunOperation => e
           logger.info "dry run for testing: #{e}"
           @boot_time ||= Time.now.to_i # comment the code below for development mode
         end
@@ -67,28 +67,18 @@ module Smash
         instance_url # gets and sets @instance_url
       end
 
-      # Check self-tags for 'task' and act as an attr_accessor.
-      # A different node's tag's can be checked for a task by passing
-      # the id param
-      # see also: SelfAwareness#task_names
-      def task_name(id = @instance_id)
-        # get @task_name
-        return @task_name unless @task_name.nil?
-        # set @task_name
-        # TODO: get all tasks instead of just the first
-        resp = ec2.describe_instances(instance_ids: [id].flatten).reservations.first
-        return @task_name = nil if resp.nil?
-        @task_name = resp.instances[0].tags.select do |t|
-          t.value if t.key == 'taskType'
-        end.first
+      # Assures there is always a valid instance id because many other Aws calls require it
+      # === @returns: String
+      def instance_id
+        @instance_id ||= metadata_request('instance_id')
       end
 
       # Gets and sets the public hostname of the instance
       def instance_url
         @instance_url ||= if zfind('TESTING')
-          'https://test-url.com'
+          'abc-1234'
         else
-          hostname_uri = 'http://169.254.169.254/latest/meta-data/public-hostname'
+          hostname_uri = 'http://169.254.169.254/latest/meta-data/instance-id'
           HTTParty.get(hostname_uri).parsed_response
         end
       end
@@ -96,28 +86,35 @@ module Smash
       # Makes the http request to self/meta-data to get all the metadata keys or,
       # if a key is given, the method makes the http request to get that
       # particular key from the metadata
-      # @param: [key <String>]
+      # === @param: key String (optional)
+      # === @returns:
       def metadata_request(key = '')
-        unless zfind('TESTING')
-          metadata_uri = "http://169.254.169.254/latest/meta-data/#{key}"
-          HTTParty.get(metadata_uri).parsed_response.split("\n")
-        else
-
-          @z ||= ['i-9254d106', 'ami-id', 'ami-launch-index', 'ami-manifest-path', 'network/thing']
-          if key == ''
-            @boogs = ['instance-id', 'ami-id', 'ami-launch-index', 'ami-manifest-path', 'network/interfaces/macs/mac/device-number']
+        key = to_hyph(key)
+        begin
+          unless zfind('TESTING')
+            metadata_uri = "http://169.254.169.254/latest/meta-data/#{key}"
+            HTTParty.get(metadata_uri).parsed_response.split("\n")
           else
-            @z.shift
+            require_relative '../../spec/stubs/aws_stubs'
+            stubbed_metadata = Smash::CloudPowers::AwsStubs::INSTANCE_METADATA_STUB
+
+            key.empty? ? stubbed_metadata.keys : stubbed_metadata[to_hyph(key)]
           end
+        rescue Exception => e
+          logger.fatal format_error_message e
         end
       end
 
+      # Return the time since boot_time
+      # === @returns: Integer
+      # Notes:
+      # * TODO: refactor to use valid time stamps for better tracking.
+      # * reason -> separate regions or OSs etc.
       def run_time
-        # TODO: refactor to use valid time stamps for better tracking.
-        # reason -> separate regions or OSs etc.
         Time.now.to_i - boot_time
       end
 
+      # Send a message on a Pipe at an interval
       def send_frequent_status_updates(opts = {})
         sleep_time = opts.delete(:interval) || 10
         stream = opts.delete(:stream_name)
@@ -129,6 +126,10 @@ module Smash
         end
       end
 
+      # Get the instance status.
+      # === @params: id String (optional)
+      #     * if no id is given, self-instance ID is returned
+      # === @returns: String
       def status(id = @instance_id)
         begin
           ec2.describe_instances(dry_run: zfind('TESTING'), instance_ids: [id]).
@@ -139,6 +140,22 @@ module Smash
         end
       end
 
+      # Check self-tags for 'task' and act as an attr_accessor.
+      # A different node's tag's can be checked for a task by passing
+      # the id param
+      # see also: SelfAwareness#task_names
+      def task_name(id = @instance_id)
+        # get @task_name
+        return @task_name unless @task_name.nil?
+        # set @task_name
+        # TODO: get all tasks instead of just the first
+        resp = ec2.describe_instances(instance_ids: [id].flatten).reservations.first
+        return @task_name = nil if resp.nil?
+        @task_name = resp.instances[0].tags.select do |t|
+          t.value if t.key == 'taskType'
+        end.first
+      end
+
       # This method will return true if:
       # * The run time is more than 5 minutes
       # and

data/lib/cloud_powers/synapse/queue/board.rb

@@ -66,8 +66,13 @@ module Smash
           # Creates an actual Queue in SQS using the standard format for a queue name (camel case)
           # @returns: Queue::Board
           def create_queue!
-            sqs.create_queue(queue_name: to_camel(@name))
-            self
+            begin
+              sqs.create_queue(queue_name: to_camel(@name))
+              self
+            rescue Aws::SQS::Errors::QueueDeletedRecently => e
+              sleep 5
+              retry
+            end
           end
 
           # Deletes an actual Queue from SQS

data/lib/cloud_powers/version.rb

@@ -1,3 +1,3 @@
 module CloudPowers
-  VERSION = "0.2.4"
+  VERSION = "0.2.5"
 end

data/lib/cloud_powers.rb

@@ -3,6 +3,7 @@ require 'cloud_powers/aws_resources'
 require 'cloud_powers/context'
 require 'cloud_powers/delegator'
 require 'cloud_powers/helper'
+require 'cloud_powers/node'
 require 'cloud_powers/self_awareness'
 require 'cloud_powers/storage'
 require 'cloud_powers/version'
@@ -17,5 +18,6 @@ module Smash
     include Smash::CloudPowers::SelfAwareness
     include Smash::CloudPowers::Storage
     include Smash::CloudPowers::Synapse
+    include Smash::CloudPowers::Node
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cloud_powers
 version: !ruby/object:Gem::Version
-  version: 0.2.4
+  version: 0.2.5
 platform: ruby
 authors:
 - Adam Phillipps
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-10-02 00:00:00.000000000 Z
+date: 2016-10-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport-core-ext
@@ -154,11 +154,12 @@ description: |2
       CloudPowers is a wrapper around AWS and in the future, other cloud service Providers.
       It was developed specifically for the Brain project but hopefully can be used
       in any other ruby project that needs to use cloud service providers' resources.
-      Version 0.2.3 has a little EC2, S3, SQS, SNS and Kinesis.  V 0.2.4 includes
-      a Context a Workflow revamp.  Websockets will be next.
+      Version 0.2.5 has a little EC2, S3, SQS, SNS and Kinesis and some a few other
+      features you can find in the docs.
+      The next versions will have websockets, IoT, more Kinesis and Workflow integration.
       This project is actively being developed, so more additions, specs and docs
-      will be updated frequently with new funcionality but the gem will follow good
-      practices for versioning.  I always welcome input.
+      will be added and updated frequently with new funcionality but the gem will
+      follow good practices for versioning.  I always welcome input.
       Enjoy!
 email: adam.phillipps@gmail.com
 executables: []
@@ -179,12 +180,12 @@ files:
 - bin/setup
 - cloud_powers.gemspec
 - lib/cloud_powers.rb
-- lib/cloud_powers/.DS_Store
 - lib/cloud_powers/auth.rb
 - lib/cloud_powers/aws_resources.rb
 - lib/cloud_powers/context.rb
 - lib/cloud_powers/delegator.rb
 - lib/cloud_powers/helper.rb
+- lib/cloud_powers/node.rb
 - lib/cloud_powers/self_awareness.rb
 - lib/cloud_powers/smash_error.rb
 - lib/cloud_powers/storage.rb