cloud_powers 0.2.7 → 0.2.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 87b0d2d0b422a079bb6f136b7629e583ebaeb88e
-  data.tar.gz: 5f4313c4ca1aab96ef5550e1f5048a43900f6b4c
+  metadata.gz: f93298a530fa939a15a4e6631b7cda3bc64e558d
+  data.tar.gz: 6f0498a0f808e3aa52cfefe4a58878a6393ae929
 SHA512:
-  metadata.gz: aa4816ebb01b5457e2dea8ea60914c08eb34cb0be5b88df46c301b6c146e6ef2954f676746d19600f05558e439a97f6cb00509fc31072e031b522017cf2d6fd5
-  data.tar.gz: fee80d3368a5dcc9acacb7ebf2924e1be611c07daa6204b0984208c480e4881ca32b6fc71998c150032807bd9fab39e96b0e05c7982ad8fb2377fcb0f36b0d2f
+  metadata.gz: 1516b53957f5229ef3ea64ac37f639262060a0337dbcdbae46ff584c5f03f23018cc284097b34f7813186d5fa0f3cbd76b0a5d1100662eeacc65532789e340dc
+  data.tar.gz: 0ab5d2c4213506152e93c21cbcc76153a10c967df3283531615e43898898f8ec8f66816c4be115675856fe67ec2f78f98cf4fb454b02046677c280d570d2bf6b

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    cloud_powers (0.2.7)
+    cloud_powers (0.2.7.1)
       activesupport-core-ext (~> 4)
       aws-sdk (~> 2)
       dotenv (~> 2.1)

data/README.rdoc.mdown

@@ -0,0 +1,180 @@
+# CloudPowers
+
+## Description
+
+CloudPowers is a wrapper around certain AWS services.  It was developed with the _need_ instead of the _resource_ in mind so even though AWS is the only service provider included now, it shouldn't be the only one forever.  There are several pieces of the Cloud Power module to talk about:
+* SelfAwareness: Gathers data about "self"
+* Synapse: Handles communication needs
+* Delegator: dynamically include Ruby executables from S3
+and more...
+
+* Each module in Cloud Powers is in charge of a specific type of task that helps bring projects together and communicate with the outside world.
+
+Below is a breakdown of the installation, common services provided and an example or 2 of popular methods.
+_Better docs are on the way_
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```Ruby
+gem 'cloud_powers'
+```
+
+then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install cloud_powers
+
+then either:
+    * set environment variables that matches the below group
+    * fill out a .env file and load it from the class that is using CloudPowers, like this
+    Notes:
+      * The code does its best in many cases to make a good guess at some of the configuration
+        for you but if you set them in your system environment variables or in the .env file,
+        it'll have a much better
+```Ruby
+require 'dotenv'
+Dotenv.load('path to your .env file')
+```
+_things you need for pre-v1_:
+```Ruby
+# Aws access:
+AWS_ACCESS_KEY_ID=""
+AWS_SECRET_ACCESS_KEY=""
+
+# Aws areas and auth-related locations:
+AWS_REGION=""
+
+# Aws Build info:
+AMI_NAME="for Cerebrums to create a node"
+
+# Aws kinesis stream names
+STATUS_STREAM="e.g. kinesis stream name"
+
+# Aws s3 buckets etc
+TASK_STORAGE="e.g. s3 object name"
+
+# Aws sqs queue addresses
+BACKLOG_QUEUE_ADDRESS=""
+WIP_QUEUE_ADDRESS=""
+FINISHED_QUEUE_ADDRESS=""
+```
+
+## Usage
+
+### AwsResource
+    * AWS resources that should be used by many services, e.g. Delegator _on EC2_ uses S3, EC2 and SQS to gain knowledge about the
+    context it will be working in.
+
+
+### Delegator
+  * Helps a node figure out what task it should be running, where its executables are, gathers it/them and loads them.
+  * Lives in the Job and Task instantiation chain
+
+
+### Helper
+  * useful shared methods, like one that turns a string into snake_case
+
+
+### SelfAwareness
+  * gets and sets info about the instance -> Neuron/Cerebrum/etc)
+```Ruby
+    get_awareness!
+```
+  * retrieves and sets all metadata from the EC2 instance and a few other things like the instance hostname (can find the instance IP from here).
+  * See EC2 metadata for details on the instance/node metadata that is set.
+  * Additionally, the instance public hostname, id and a few others are set using other-than-EC2-metadata methods.
+
+
+### SmashError
+
+
+### Storage
+  * S3
+
+
+### Synapse
+  * A Synapse is used for communicating, usually between nodes and an external source for status updates but the Synapse can handle any kind of information that needs to be passed.
+  * Architecture
+    * The Synapse is a communications module that is broken up into separate types of communication, for separate needs.
+    * There are 2 modules, soon to be 3, inside the Synapse module:
+
+  #### Queue
+    * like a task list or a message board for asynchronous communication
+    * Board <Struct>:
+      * interface with Queue config, data, name, etc.
+      ```Ruby
+        default_workflow = Workflow.new
+        board = Board.new(default_workflow)
+        board.name
+        => 'backlog'
+        board.address
+        => 'http://aws-url/backlog-board-url'
+        board.next_board # useful because this is a simple state-machine implementation
+        => 'wip'
+      ```
+    Example usage:
+      1. Give the entire stream name or a symbol or string that is found in the .env for the name of the Queue
+      2. provide a block that will create the record that gets sent to the board
+    ```Ruby
+      poll(board_name <string|symbol>, opts <optional config Hash>) { block }
+    ```
+    or
+    ```Ruby
+      poll(:backlog) { |msg, stats| Task.new(instance_id, msg) if stats.success? }
+    ```
+    or
+    ```Ruby
+      poll(:backlog, wait_time: 30, delete: false) do
+        edited_message = sitrep.merge(foo: 'bar')
+        update = some_method(edited_message)
+      end
+    ```
+  #### Pipe
+    * Good for real time information passing, like status updates, results reporting, operations, etc.
+    * The Pipe module is for communicating via streams.  Piping is meant to be a way to communicate status, problems and  other general info or huge result sets and things of that nature.  There can be very high traffic through the pipe or none at all.  Very soon (probably V-0.2.6 or 7), Cerebrums will be data consumers, to the Java KCL and MultiLangDaemon level, so keeping messages straight is done via partition ID.  The partition ID of any message is to identify which node the message is about or the "batch" or "group" or any other concept like that. So the instance ID is used in nodes like the Neuron and Cerebrum and other identifiers that are deemed best are used in other projects as a batch ID.
+
+    Example usage:
+    ```Ruby
+      pipe_to(stream_name <string/symbol>) { &block }
+    ```
+    ```Ruby
+      pipe_to(:status_queue) { sitrep(content: 'workflowComplete') }
+    ```
+    and for multiple records (KCL)
+    ```Ruby
+      flow_to(stream_name <string/symbol>) { &block }
+    ```
+    ```Ruby
+      flow_to(:status_queue) do
+        interesting_instances = neurons.map do |neuron|
+          return neuron if neuron.workflow.done?
+        end
+        find_efficient_neurons(interesting_instances) # this gets sent through the Pipe
+      end
+    ```
+
+
+  #### Memory
+    * Allows the nodes to have a collective awareness of each other, the environment they work in and other Jobs (Coming soon...)
+
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/adam-phillipps/cloud_powers.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -8,3 +8,8 @@ task :default => :spec
 task :console do
   exec "irb -r cloud_powers -I ./lib"
 end
+
+# Rake::RDocTask.new do |rd|
+#   rd.main = "README.rdoc"
+#   rd.rdoc_files.include("README.rdoc", "lib/**/*.rb")
+# end

data/cloud_powers.gemspec

@@ -15,7 +15,7 @@ Gem::Specification.new do |spec|
     It was developed specifically for the Brain project but hopefully can be used
     in any other ruby project that needs to use cloud service providers' resources.
 
-    Version 0.2.7 has a little EC2, S3, SQS, SNS, Kinesis, websockets and a few other
+    Version 0.2.8 has a little EC2, S3, SQS, SNS, Kinesis, websockets and a few other
     features you can find in the docs.  There is also limitted support for stubbing
     AWS RESTful API calls.  That can come in handy for local testing and extra setup on
     AWS resource clients.

data/lib/cloud_powers/auth.rb

@@ -4,26 +4,40 @@ require_relative 'zenv'
 
 module Smash
   module CloudPowers
+    # Provides authentication to cloud resources, e.g. AWS
     module Auth
       extend Smash::CloudPowers::Zenv
 
-      # This method is able to be called before an object is instantiated in order
-      # to provide a region in AWS-land.
-      # === @returns: The region set in configuration or a 'us-west-2' default String
-      def self.region
-        zfind(:aws_region) || 'us-west-2'
-      end
-
-      # This method is able to be called before an object is instantiated in order
-      # to provide an Aws::Credentials object that will allow access to all the
-      # resources in the account that zfind searches for, using the "ACCOUNT_NUMBER"
+      # This method is usable before an object is instantiated in order
+      # to provide an <tt>Aws::Credentials</tt> object that will allow access to all the
+      # resources in the account that zfind searches for, using the <tt>ACCOUNT_NUMBER</tt>
       # key.
+      #
+      # === Returns
+      # <tt>Aws::Credentials</tt>
+      #
+      # === Example
+      #   Auth.creds
+      #   => Aws::Credentials
       def self.creds
         @creds ||= Aws::Credentials.new(
           zfind(:aws_access_key_id),
           zfind(:aws_secret_access_key)
         )
       end
+
+      # This method is able to be called before an object is instantiated in order
+      # to provide a region in AWS-landia.
+      #
+      # === Returns
+      # The region set in configuration or a <tt>'us-west-2'</tt> default <tt>String</tt>
+      #
+      # === Example
+      #   Auth.region
+      #   => 'us-east-1'
+      def self.region
+        zfind(:aws_region) || 'us-west-2'
+      end
     end
   end
 end

data/lib/cloud_powers/aws_resources.rb

@@ -10,41 +10,28 @@ module Smash
       include Smash::CloudPowers::Zenv
 
       # Get the region from the environment/context or use a default region for AWS API calls.
-      # === @returns String
+      #
+      # === Returns
+      # +String+
       def region
         zfind(:aws_region) || 'us-west-2'
       end
 
       # Get or create an EC2 client and cache that client so that a Context is more well tied together
-      # === @params: opts [Hash]
-      #      * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
-      #      * region: defaulted to use the `#region()` method
-      #      * AWS::Credentials object, which will also scour the context and environment for your keys
-      # === @returns: AWS::EC2 client
+      #
+      # Parameters
+      # * opts +Hash+ (optional)
+      # * * stub_responses - defaulted to +false+ but it can be overriden with the desired responses for local testing
+      # * * region - defaulted to use the <tt>#region()</tt> method
+      # * * AWS::Credentials object, which will also scour the context and environment for your keys
+      #
+      # === Returns
+      # <tt>AWS::EC2::Client</tt>
+      #
       # === Sample Usage
-      #     ```
-      #       config = stub_responses: {
-      #         run_instances: {
-      #           instances: [
-      #             { instance_id: 'asd-1234', launch_time: Time.now, state: { name: 'running' }
-      #         ] }
-      #         describe_instances: {
-      #           reservations: [
-      #             { instances: [
-      #               { instance_id: 'asd-1234', state: { code: 200, name: 'running' } },
-      #           ] }] },
-      #         describe_images: {
-      #           images: [
-      #             { image_id: 'asdf', state: 'available' }
-      #         ] }
-      #       }
-      #
-      #       ec2(config) # sets and gets an EC2 client
-      #
-      #       images = ec2.describe_images
-      #       images.first[:image_id]
-      #       # => 'asdf'
-      #     ```
+      #   images = ec2.describe_images
+      #   images.first[:image_id]
+      #   # => 'asdf'
       def ec2(opts = {})
         config = {
           stub_responses: false,
@@ -59,11 +46,15 @@ module Smash
       # Get an image using a name and filters functionality from EC2.  The name is required but the filter defaults
       # to search for the tag with the key `aminame` because this is the key that most Nodes will search for, when
       # they gather an AMI to start with.
-      # === @params: opts [Hash]
-      #      * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
-      #      * region: defaulted to use the `#region()` method
-      #      * AWS::Credentials object, which will also scour the context and environment for your keys
-      # === @returns: AMI
+      #
+      # Parameters
+      # * opts [Hash]
+      # * * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
+      # * * region: defaulted to use the `#region()` method
+      # * * AWS::Credentials object, which will also scour the context and environment for your keys
+      #
+      # === Returns
+      # Aws::EC2::Image
       def image(name, opts = {})
         config = {
           filters: [{ name: 'tag:aminame', values: [name.to_s] }]
@@ -74,36 +65,18 @@ module Smash
       end
 
       # Get or create an Kinesis client and cache that client so that a Context is more well tied together
-      # === @params: opts [Hash]
-      #      * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
-      #      * region: defaulted to use the `#region()` method
-      #      * AWS::Credentials object, which will also scour the context and environment for your keys
-      # === @returns: AWS::Kinesis client
-      # === Sample Usage
-      #     ```
-      #       config = {
-      #         stub_responses: {
-      #           create_stream: {},
-      #             put_record: {
-      #               shard_id: opts[:shard_id] || 'idididididididid',
-      #               sequence_number: opts[:sequence_number] || Time.now.to_i.to_s
-      #             },
-      #             describe_stream: {
-      #               stream_description: {
-      #                 stream_name: opts[:name] || 'somePipe',
-      #                 stream_arn:  'arnarnarnarnar',
-      #                 stream_status: 'ACTIVE',
-      #               }
-      #             }
-      #           }
-      #         }
-      #       }
-      #
-      #       kinesis(config) # sets and gets an Kinesis client
-      #
-      #       pipe_to('somePipe') { update_body(status: 'waHoo') }
-      #       # => sequence_number: '1676151970'
-      #     ```
+      # Parameters
+      # * opts <tt>Hash</tt>
+      # * * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
+      # * * region: defaulted to use the `#region()` method
+      # * * AWS::Credentials object, which will also scour the context and environment for your keys
+      #
+      # === Returns
+      # AWS::Kinesis client
+      #
+      # === Example
+      #   pipe_to('somePipe') { update_body(status: 'waHoo') } # uses Aws::Kinesis::Client.put_recor()
+      #   # => sequence_number: '1676151970'
       def kinesis(opts = {})
         config = {
           stub_responses: false,
@@ -116,21 +89,19 @@ module Smash
       end
 
       # Get or create an S3 client and cache that client so that a Context is more well tied together
-      # === @params: opts [Hash]
-      #      * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
-      #      * region: defaulted to use the `#region()` method
-      #      * AWS::Credentials object, which will also scour the context and environment for your keys
-      # === @returns: AWS::S3 client
-      # === Sample Usage
-      #     ```
-      #       config = {
-      #         stub_responses: {
       #
-      #         }
-      #       }
+      # Parameters
+      # * opts <tt>Hash</tt>
+      # * * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
+      # * * region: defaulted to use the `#region()` method
+      # * * AWS::Credentials object, which will also scour the context and environment for your keys
       #
-      #     ```
-
+      # === Returns
+      # AWS::S3 client
+      #
+      # === Example
+      #   expect(s3.head_bucket).to be_empty
+      #   # passing expectation
       def s3(opts = {})
         config = {
           stub_responses: false,
@@ -143,28 +114,18 @@ module Smash
       end
 
       # Get or create an SNS client and cache that client so that a Context is more well tied together
-      # === @params: opts [Hash]
-      #      * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
-      #      * region: defaulted to use the `#region()` method
-      #      * AWS::Credentials object, which will also scour the context and environment for your keys
-      # === @returns: AWS::SNS client
-      # === Sample Usage
-      #     ```
-      #       config = {
-      #         stub_responses: {
-      #           create_topic: {},
-      #           delete_topic: {},
-      #           list_topics: [],
-      #           publish: {},
-      #           subscribe: {}
-      #         }
-      #       }
-      #
-      #       sns(config) # sets and gets an Kinesis client
-      #
-      #       create_channel!('testBroadcast')
-      #       # => true
-      #     ```
+      # Parameters
+      # * opts +Hash+
+      # * * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
+      # * * region: defaulted to use the `#region()` method
+      # * * AWS::Credentials object, which will also scour the context and environment for your keys
+      #
+      # === Returns
+      # AWS::SNS client
+      #
+      # === Example
+      #   create_channel!('testBroadcast') # uses Aws::SNS::Client
+      #   # => true
       def sns(opts = {})
         config = {
           stub_responses: false,
@@ -177,23 +138,18 @@ module Smash
       end
 
       # Get or create an SQS client and cache that client so that a Context is more well tied together
-      # === @params: opts [Hash]
-      #      * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
-      #      * region: defaulted to use the `#region()` method
-      #      * AWS::Credentials object, which will also scour the context and environment for your keys
-      # === @returns: AWS::SQS client
-      # === Sample Usage
-      #     ```
-      #       config = stub_responses: {
-      #         create_queue: {
-      #           queue_url: "https://sqs.us-west-2.amazonaws.com/12345678/#{opts[:name] || 'testQueue'}"
-      #         }
-      #       }
       #
-      #       sqs(config) # sets and gets an Kinesis client
+      # Parameters
+      # * opts <tt>Hash</tt>
+      # * * stub_responses: defaulted to false but it can be overriden with the desired responses for local testing
+      # * * region: defaulted to use the `#region()` method
+      # * * AWS::Credentials object, which will also scour the context and environment for your keys
+      #
+      # === Returns
+      # AWS::SQS client
       #
-      #       create_queue('someQueue')
-      #     ```
+      # === Example
+      #   create_queue('someQueue') # Uses Aws::SQS::Client
       def sqs(opts = {})
         config = {
           stub_responses: false,