cloud_powers 0.2.7.23 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 97dc22c7470b4936967a4e45da7b3bf962dd2697
-  data.tar.gz: 0b0a347337a1f910bb10c46f8da3e785b3e6af21
+  metadata.gz: 403667ef1b83f93c4534a4273849b4cd471385e4
+  data.tar.gz: c233cc41bd999aaa9b0bc6cc58d1a08c1947af73
 SHA512:
-  metadata.gz: f93ff5427fba0268c10899d3103c220167d4461eb5606c89f5d175cf460f1975f7a9e30dd8ffcff7e4899eee3c1b08c3d53f71e6d39f3bc0fa60c027438539fb
-  data.tar.gz: a3cdab518b3ad93891227d615fab39ec9ddcb32d1825e7ed37d4ec3f106cd647809153053fbe09d76868b103e0ed568e8cb6f9dd15bfa18dc4d1429a5818de9b
+  metadata.gz: bcad8e1544017abf5b498b546271cadb8071d1c498fd599983e9d4e5d7ed391d3033e82589ddd0aec49a171cb098804dfad9d0439d19cb5fb7f10a23d80d3828
+  data.tar.gz: a95f21bad47ff61cf1175716dff63322dc20908b25d92201a59fb61d785f86121e9589b5c4ee939fa0fef02a64ad633fd247c8a0a490f7f2ba26732d27010bab

data/.gitignore

@@ -1,3 +1,4 @@
+**/*old.rb
 **/.bundle
 **/.yardoc
 **/Gemfile.lock

data/.test.env.example

@@ -2,21 +2,21 @@
 PROJECT_ROOT=""
 
 # Aws access:
-AWS_ACCESS_KEY_ID=""
-AWS_SECRET_ACCESS_KEY=""
-ACCOUNT_NUMBER=""
+AWS_ACCESS_KEY_ID="AAAAAAAAAAAAAAAAAAAA"
+AWS_SECRET_ACCESS_KEY="xxddccffvvggbbhhnnjjmmkknhhd+2344322341"
+ACCOUNT_NUMBER="0112233445566"
 
-# Aws areas and auth-related locations: 
+# Aws areas and auth-related locations:
 AWS_REGION=""
 
 # Aws Build info:
-AMI_NAME="for Cerebrums to create a node"
+NODE_IMAGE="for Cerebrums to create a node"
 
 # Aws kinesis stream names
 STATUS_STREAM="e.g. kinesis stream name"
 
 # Aws s3 buckets etc
-TASK_STORAGE="e.g. s3 object name"
+JOB_STORAGE="e.g. s3 object name"
 
 # Aws sqs queue addresses
 BACKLOG_QUEUE_ADDRESS=""

data/.travis.yml

@@ -1,4 +1,4 @@
 sudo: false
 language: ruby
 
-before_install: gem install bundler -v 1.12.5
+before_install: gem install bundler -v 1.13.5

data/README

@@ -0,0 +1,190 @@
+[![Gem Version](https://badge.fury.io/rb/cloud_powers.svg)](https://badge.fury.io/rb/cloud_powers)CloudPowers
+
+## Description
+
+CloudPowers is a wrapper around AWS and in the future, other cloud service Providers.
+It was developed specifically for the Brain project but hopefully can be used
+in any other ruby project that needs to use cloud service providers' resources.
+
+Version 0.2.8 has a some EC2, S3, SQS, SNS, Kinesis, websockets and a few other
+features you can find in the docs.  There is also limitted support for stubbing
+AWS RESTful API calls.  That can come in handy for local testing and extra setup on
+AWS resource clients.
+
+The next versions will have more Kinesis, Workflow integration and IoT.
+
+This project is actively being developed, so more additions, specs and docs
+will be added and updated frequently with new funcionality but the gem will
+follow good practices for versioning.  Input is always welcome.
+Enjoy!
+
+* Each module in Cloud Powers is in charge of a specific type of task that helps bring projects together and communicate with the outside world.
+
+Below is a breakdown of the installation, common services provided and an example or 2 of popular methods.
+_Better docs are on the way_
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```Ruby
+gem 'cloud_powers'
+```
+
+then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install cloud_powers
+
+then either:
+    * set environment variables that matches the below group
+    * fill out a .env file and load it from the class that is using CloudPowers, like this
+    Notes:
+      * The code does its best in many cases to make a good guess at some of the configuration
+        for you but if you set them in your system environment variables or in the .env file,
+        it'll have a much better
+```Ruby
+require 'dotenv'
+Dotenv.load('path to your .env file')
+```
+_things you need for pre-v1_:
+```Ruby
+# Aws access:
+AWS_ACCESS_KEY_ID=""
+AWS_SECRET_ACCESS_KEY=""
+
+# Aws areas and auth-related locations:
+AWS_REGION=""
+
+# Aws Build info:
+AMI_NAME="for Cerebrums to create a node"
+
+# Aws kinesis stream names
+STATUS_STREAM="e.g. kinesis stream name"
+
+# Aws s3 buckets etc
+TASK_STORAGE="e.g. s3 object name"
+
+# Aws sqs queue addresses
+BACKLOG_QUEUE_ADDRESS=""
+WIP_QUEUE_ADDRESS=""
+FINISHED_QUEUE_ADDRESS=""
+```
+
+## Usage
+
+### AwsResource
+    * AWS resources that should be used by many services, e.g. Delegator _on EC2_ uses S3, EC2 and SQS to gain knowledge about the
+    context it will be working in.
+
+
+### Delegator
+  * Helps a node figure out what task it should be running, where its executables are, gathers it/them and loads them.
+  * Lives in the Job and Task instantiation chain
+
+
+### Helper
+  * useful shared methods, like one that turns a string into snake_case
+
+
+### SelfAwareness
+  * gets and sets info about the instance -> Neuron/Cerebrum/etc)
+```Ruby
+    get_awareness!
+```
+  * retrieves and sets all metadata from the EC2 instance and a few other things like the instance hostname (can find the instance IP from here).
+  * See EC2 metadata for details on the instance/node metadata that is set.
+  * Additionally, the instance public hostname, id and a few others are set using other-than-EC2-metadata methods.
+
+
+### SmashError
+
+
+### Storage
+  * S3
+
+
+### Synapse
+  * A Synapse is used for communicating, usually between nodes and an external source for status updates but the Synapse can handle any kind of information that needs to be passed.
+  * Architecture
+    * The Synapse is a communications module that is broken up into separate types of communication, for separate needs.
+    * There are 2 modules, soon to be 3, inside the Synapse module:
+
+  #### Queue
+    * like a task list or a message board for asynchronous communication
+    * Board <Struct>:
+      * interface with Queue config, data, name, etc.
+      ```Ruby
+        default_workflow = Workflow.new
+        board = Board.new(default_workflow)
+        board.name
+        => 'backlog'
+        board.address
+        => 'http://aws-url/backlog-board-url'
+        board.next_board # useful because this is a simple state-machine implementation
+        => 'wip'
+      ```
+    Example usage:
+      1. Give the entire stream name or a symbol or string that is found in the .env for the name of the Queue
+      2. provide a block that will create the record that gets sent to the board
+    ```Ruby
+      poll(board_name <string|symbol>, opts <optional config Hash>) { block }
+    ```
+    or
+    ```Ruby
+      poll(:backlog) { |msg, stats| Task.new(instance_id, msg) if stats.success? }
+    ```
+    or
+    ```Ruby
+      poll(:backlog, wait_time: 30, delete: false) do
+        edited_message = sitrep.merge(foo: 'bar')
+        update = some_method(edited_message)
+      end
+    ```
+  #### Pipe
+    * Good for real time information passing, like status updates, results reporting, operations, etc.
+    * The Pipe module is for communicating via streams.  Piping is meant to be a way to communicate status, problems and  other general info or huge result sets and things of that nature.  There can be very high traffic through the pipe or none at all.  Very soon (probably V-0.2.6 or 7), Cerebrums will be data consumers, to the Java KCL and MultiLangDaemon level, so keeping messages straight is done via partition ID.  The partition ID of any message is to identify which node the message is about or the "batch" or "group" or any other concept like that. So the instance ID is used in nodes like the Neuron and Cerebrum and other identifiers that are deemed best are used in other projects as a batch ID.
+
+    Example usage:
+    ```Ruby
+      pipe_to(stream_name <string/symbol>) { &block }
+    ```
+    ```Ruby
+      pipe_to(:status_queue) { sitrep(content: 'workflowComplete') }
+    ```
+    and for multiple records (KCL)
+    ```Ruby
+      flow_to(stream_name <string/symbol>) { &block }
+    ```
+    ```Ruby
+      flow_to(:status_queue) do
+        interesting_instances = neurons.map do |neuron|
+          return neuron if neuron.workflow.done?
+        end
+        find_efficient_neurons(interesting_instances) # this gets sent through the Pipe
+      end
+    ```
+
+
+  #### Memory
+    * Allows the nodes to have a collective awareness of each other, the environment they work in and other Jobs (Coming soon...)
+
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/adam-phillipps/cloud_powers.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/cloud_powers.gemspec

@@ -15,16 +15,16 @@ Gem::Specification.new do |spec|
     It was developed specifically for the Brain project but hopefully can be used
     in any other ruby project that needs to use cloud service providers' resources.
 
-    Version 0.2.8 has a some EC2, S3, SQS, SNS, Kinesis, websockets and a few other
+    Version 1.0 has a some EC2, S3, SQS, SNS, Kinesis, websockets and a few other
     features you can find in the docs.  There is also limitted support for stubbing
     AWS RESTful API calls.  That can come in handy for local testing and extra setup on
     AWS resource clients.
 
-    The next versions will have more Kinesis, Workflow integration and IoT.
-
     This project is actively being developed, so more additions, specs and docs
     will be added and updated frequently with new funcionality but the gem will
-    follow good practices for versioning.  Input is always welcome.
+    follow good practices for versioning and so the behavior won't change on
+    existing features.
+    Input is always welcome. :thumbsup:
     Enjoy!
   EOF
   spec.homepage               =     'https://github.com/adam-phillipps/cloud_powers'

data/lib/cloud_powers.rb

@@ -1,13 +1,11 @@
 require 'cloud_powers/auth'
 require 'cloud_powers/aws_resources'
-require 'cloud_powers/context'
-require 'cloud_powers/delegator'
-require 'cloud_powers/helper'
+require 'cloud_powers/helpers'
 require 'cloud_powers/node'
-require 'cloud_powers/self_awareness'
+require 'cloud_powers/resource'
 require 'cloud_powers/storage'
 require 'cloud_powers/version'
-require 'cloud_powers/workflow_factory'
+require 'cloud_powers/synapse/synapse'
 
 # The Smash module allows us to use CloudPowers under a shared name space with other projects.
 module Smash
@@ -15,21 +13,13 @@ module Smash
   module CloudPowers
     # Authentication mixin
     extend Smash::CloudPowers::Auth
-    # Dynamic Resource creation and delegation
-    extend Smash::CloudPowers::Delegator
     # Aws clients, like EC2 and S3
     include Smash::CloudPowers::AwsResources
-    # Various helper methods
-    include Smash::CloudPowers::Helper
-    # Gathers data about an instance, itself
-    include Smash::CloudPowers::SelfAwareness
     # Store files
     include Smash::CloudPowers::Storage
     # Communication modules
     include Smash::CloudPowers::Synapse
     # CRUD on Nodes, which are individual instances
     include Smash::CloudPowers::Node
-    # Dynamically Builds and loads a Workflow into a class at runtime
-    include Smash::CloudPowers::WorkflowFactory
   end
 end

data/lib/cloud_powers/aws_resources.rb

@@ -1,12 +1,12 @@
-require_relative 'auth'
-require_relative 'helper'
-require_relative 'zenv'
+require 'cloud_powers/auth'
+require 'cloud_powers/helpers'
+require 'cloud_powers/zenv'
 
 module Smash
   module CloudPowers
     module AwsResources
       include Smash::CloudPowers::Auth
-      include Smash::CloudPowers::Helper
+      include Smash::CloudPowers::Helpers
       include Smash::CloudPowers::Zenv
 
       # Get the region from the environment/context or use a default region for AWS API calls.
@@ -89,6 +89,23 @@ module Smash
         @kinesis ||= Aws::Kinesis::Client.new(config)
       end
 
+      # Create a QueuePoller for an already created SQS Queue
+      # (CloudPowers::Synapse::Board)
+      #
+      # Parameters
+      # * :url +String+ - the url for the Queue/Board
+      # * :client <tt>Aws::SQS::Client</tt>
+      #
+      # Returns
+      # <tt>Aws::SQS::QueuePoller</tt>
+      #
+      # Notes
+      # * this is different from the other methods in here, in that it doesn't
+      # create an i-var on your class.
+      def queue_poller(url:, client: sqs)
+        Aws::SQS::QueuePoller.new(url, client: client)
+      end
+
       # Get or create an S3 client and cache that client so that a Context is more well tied together
       #
       # Parameters

data/lib/cloud_powers/creatable.rb

@@ -0,0 +1,122 @@
+require 'cloud_powers/helpers'
+
+module Smash
+  module CloudPowers
+    # Give an object an
+    # Active Record-like interface, with a twist.  Instead of a database, we're
+    # using cloud resources. This module includes handy methods that can act
+    # like before and after hooks, for object instantiation and cloud-resource
+    # creation.  To use the module, you have to participate in the interface,
+    # which means you have to follow <i><b>a few rules</b></i>:
+    #   1. your +initialize+ method can take a list of keyword arguments or a +Hash+
+    #   for configuration
+    #   2. your resource class has a <tt>create_resource</tt> method that can
+    #   deal with making the http request for creating the resource in the cloud
+    #   3. your class extends this module
+    module Creatable
+      include Smash::CloudPowers::Helpers
+
+      def self.included base
+        base.send :include, AfterHooks
+        base.extend BeforeHooks
+      end
+
+      module BeforeHooks
+        # +Boolean+ - tells if this resource is mapped to a resource in the cloud
+        attr_accessor :saved
+
+        # A +before_hook+ style method that allows you to do things after you
+        # instantiate the resource's object but <b>before</b> you create the
+        # resource, in the cloud.  Because we're likely to be in the middle of
+        # requesting a resource be requested, in the cloud, this method gives us
+        # a means to do some checking, linking and/or setup before we make a request.
+        #
+        # Parameters
+        # * +Hash+ or keyword argument(s) - configuration
+        #
+        # * +Block+ (optional) - your optional before_hook that runs before you've
+        # instantiated your object, yielding the new instance to you, to run in your block.
+        #
+        # Returns
+        # +Object+ - a new instance of the class that extended this module
+        def build(name:, **config)
+          new_resource = self.new(name: name, **config)
+          yield new_resource if block_given?
+          new_resource
+        end
+
+        # An +after_hook+ style method that allows you to do things after you
+        # instantiate the resource's object, but before you create the resource,
+        # in the cloud.  Because we're likely to be in the middle of requesting to
+        # create a resource, in the cloud, this method
+        # gives us a means to do some checking or setup after we make a request.
+        #
+        # Parameters
+        # * +Hash+ or keyword argument(s) - configuration
+        #
+        # * +Block+ (optional) - your optional after_hook that runs after you've
+        # created your resource(s) in the cloud, yielding the new instance to you,
+        # to run in your block.
+        #
+        # Returns
+        # +Object+ - a new instance of the class that extended this module
+        #
+        # Notes
+        # * See <tt>#build</tt>
+        # * See <tt>#save!</tt>
+        # * See <tt>CloudPowers::Helpers::LogicHelp#instance_attr_accessor</tt>
+        # * See <tt>CloudPowers::Helpers::LogicHelp#attr_map</tt>
+        def create!(name:, **config)
+          new_resource = self.build(name: name, **config)
+
+          new_resource.attr_map(config) do |config_key, config_value|
+            new_resource.instance_attr_accessor new_resource.to_snake(config_key)
+            config_value
+          end
+
+          new_resource.save!
+
+          yield new_resource if block_given?
+          new_resource
+        end
+      end
+
+      module AfterHooks
+        # Alternative to <tt>save!()</tt>.  This predicate method is based off
+        # the <tt>@linked</tt> i-var and is set to true after it has been confirmed
+        # that this resource is a good map to the resource in the cloud.
+        #
+        # Returns
+        # * +Boolean+
+        def linked?
+          !!@linked
+        end
+
+        # An +after_hook+ style method that sends a reqeust to your custom implementation
+        # of the <tt>create_resource</tt> methodallows you to do things after you
+        # create the resource, in the cloud.  This method relies on you having
+        # a <tt>create_resource</tt> method that can handle every aspect of creating
+        # your resource.  If this piece of the contract isn't obeyed, you will
+        # receive a <tt>NoMethodError</tt>.
+        #
+        # Returns
+        # +Boolean+ - +true+ if the resource
+        def save!
+          resp = create_resource if self.respond_to? :create_resource
+          @saved = !resp.nil?
+        end
+
+        # Find out if the resource was created
+        #
+        # Returns
+        # +Boolean+
+        def saved?
+          !!@saved
+        end
+      end
+    end
+  end
+end
+
+
+