cloud_powers 0.1.4 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 391860e2ec54145cbb41486fd29c2915caaaf7b5
-  data.tar.gz: 1bccfe4b9cbb9f3293c2c3a740567edc0e9eb9d5
+  metadata.gz: 5276c7479f1dfcdc4d59aaffa8841d5ab4839180
+  data.tar.gz: 9dcab0fc49c1c0bd9e13fd259719ef697f209429
 SHA512:
-  metadata.gz: 83e75d3065430d6e2123a4b7e7104818c2fbcf9f0a091022c6e85aec929bc7d335bb0abdda6393e8435eb0e9ee533fd0a4eae8d17cfac3be93f7a505de410ae1
-  data.tar.gz: f1c12786a596e76cd4d87aefff57a8d8d2abf4924a80c8653624c3749c4aec290dbe2cc9e81c4696c813fd7930b6c2faac31f0d50eea5e285b9f76923ff25f47
+  metadata.gz: b5c3f428b50af4dd03c40521d461ed4bc7c23c28b598557b3495bfe85ee63a1571d6f863c0d1203ba9a309214344222b073a232cd340e67c6841ed30863ae65c
+  data.tar.gz: 6bbac08f878d73345f023fde38ffbde8ab92c53a134c828fe4721d176363126ad99e028672e4c3383837afe935da5b32b2f4bffa73a06506710f6e58fa22f29c

data/.gitignore

@@ -8,3 +8,4 @@
 /spec/reports/
 /tmp/
 *.env
+.byebug_history

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    cloud_powers (0.1.4)
+    cloud_powers (0.1.5)
       activesupport-core-ext (~> 4)
       aws-sdk (~> 2)
       dotenv (~> 2.1)

data/README.md

@@ -1,8 +1,11 @@
 # CloudPowers
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/cloud_powers`. To experiment with that code, run `bin/console` for an interactive prompt.
+## Description
 
-TODO: Delete this and the text above, and describe your gem
+CloudPowers is a wrapper around certain AWS services.  It was developed with the _need_ instead of the _resource_ in mind so even though AWS is the only service provider included now, it shouldn't be the only one forever.  There are several pieces of the Cloud Power module to talk about: SelfAwareness, Synapse, Delegator and more.  Each module in Cloud Powers is in charge of a specific type of task that helps bring projects together and communicate with the outside world.
+
+Below is a breakdown of the installation, common services provided and an example or 2 of popular methods.
+_Better docs are on the way_
 
 ## Installation
 
@@ -12,7 +15,7 @@ Add this line to your application's Gemfile:
 gem 'cloud_powers'
 ```
 
-And then execute:
+then execute:
 
     $ bundle
 
@@ -20,10 +23,140 @@ Or install it yourself as:
 
     $ gem install cloud_powers
 
+then either:
+    * set environment variables that matches the below group
+    * fill out a .env file and load it from the class that is using CloudPowers, like this
+```ruby
+require 'dotenv'
+Dotenv.load('path to your .env file')
+```
+_things you need for pre-v1_:
+```
+# Aws access:
+AWS_ACCESS_KEY_ID=""
+AWS_SECRET_ACCESS_KEY=""
+
+# Aws areas and auth-related locations: 
+AWS_REGION=""
+
+# Aws Build info:
+AMI_NAME="for Cerebrums to create a node"
+
+# Aws kinesis stream names
+STATUS_STREAM="e.g. kinesis stream name"
+
+# Aws s3 buckets etc
+TASK_STORAGE="e.g. s3 object name"
+
+# Aws sqs queue addresses
+BACKLOG_QUEUE_ADDRESS=""
+WIP_QUEUE_ADDRESS=""
+FINISHED_QUEUE_ADDRESS=""
+```
+
 ## Usage
 
-TODO: Write usage instructions here
+### AwsResource
+    * AWS resources that should be used by many services, e.g. Delegator _on EC2_ uses S3, EC2 and SQS to gain knowledge about the
+    context it will be working in.
+
+
+### Delegator
+  * Helps a node figure out what task it should be running, where its executables are, gathers it/them and loads them.
+  * Lives in the Job and Task instantiation chain
+
 
+### Helper
+  * useful shared methods, like one that turns a string into snake_case
+
+
+### SelfAwareness
+  * gets and sets info about the instance -> Neuron/Cerebrum/etc)
+```ruby
+    get_awareness!
+```
+  * retrieves and sets all metadata from the EC2 instance and a few other things like the instance hostname (can find the instance IP from here).
+  * See EC2 metadata for details on the instance/node metadata that is set.  
+  * Additionally, the instance public hostname, id and a few others are set using other-than-EC2-metadata methods.
+
+
+### SmashError
+
+
+### Storage
+  * S3
+
+
+### Synapse
+  * A Synapse is used for communicating, usually between nodes and an external source for status updates but the Synapse can handle any kind of information that needs to be passed.
+  * Architecture
+    * The Synapse is a communications module that is broken up into separate types of communication, for separate needs.
+    * There are 2 modules, soon to be 3, inside the Synapse module:
+
+  #### Queue
+    * like a task list or a message board for asynchronous communication
+    * Board <Struct>:
+      * interface with Queue config, data, name, etc.
+      ```ruby
+        default_workflow = Workflow.new
+        board = Board.new(default_workflow)
+        board.name
+        => 'backlog'
+        board.address
+        => 'http://aws-url/backlog-board-url'
+        board.next_board # useful because this is a simple state-machine implementation
+        => 'wip'
+      ```
+    Example usage:
+      1. Give the entire stream name or a symbol or string that is found in the .env for the name of the Queue
+      2. provide a block that will create the record that gets sent to the board
+    ```ruby
+      poll(board_name <string|symbol>, opts <optional config Hash>) { block }
+    ```
+    or
+    ```ruby
+      poll(:backlog) { |msg, stats| Task.new(instance_id, msg) if stats.success? }
+    ```
+    or
+    ```ruby
+      poll(:backlog, wait_time: 30, delete: false) do
+        edited_message = sitrep.merge(foo: 'bar')
+        update = some_method(edited_message)
+      end
+    ```
+  #### Pipe
+    * Good for real time information passing, like status updates, results reporting, operations, etc.
+    * The Pipe module is for communicating via streams or webhooks.  Piping is meant to be a way to communicate status, problems and  other general info.  There can be very high traffic through the pipe or none at all.  Very soon, Cerebrums will be data consumers, to the Java KCL and MultiLangDaemon level, so keeping messages straight is done via partition ID.  The partition ID of any message is to identify which node the message is about, and so the instance ID is used in nodes like the Neuron and Cerebrum and other identifiers that are deemed best are used in other projects.
+
+    Example usage:
+    ruby```
+      pipe_to(stream_name <string/symbol>) { &block }
+    ```
+    ruby```
+      pipe_to(:status_queue) { sitrep(content: 'workflowComplete') }
+    ```
+    and for multiple records (KCL)
+    ```ruby
+      flow_to(stream_name <string/symbol>) { &block }
+    ```
+    ```ruby
+      flow_to(:status_queue) do
+        interesting_instances = neurons.map do |neuron|
+          return neuron if neuron.workflow.done?
+        end
+        find_efficient_neurons(interesting_instances) # this gets sent through the Pipe
+      end 
+    ```
+
+
+  #### Memory
+    * Allows the nodes to have a collective awareness of each other, the environment they work in and other Jobs (Coming soon...)
+
+
+### Workflow
+  * The workflow is one of the integral ideas behind the Brain architecture.  The workflow describes to us humans what state the Job/Task/etc is currently in and it let's the App(s) know what things it can do from any point in the process.
+
+  
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -32,7 +165,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/cloud_powers.
+Bug reports and pull requests are welcome on GitHub at https://github.com/adam-phillipps/cloud_powers.
 
 
 ## License

data/bin/setup

@@ -5,4 +5,16 @@ set -vx
 
 bundle install
 
-# Do any other automated setup that you need to do here
+find_or_create_env_file() {
+    pushd "../spec"
+        cp .test.env.example .test.env
+        printf "Testing:\n- Fill out the .test.env file inside the spec/ directory"
+    popd
+}
+
+printf "Better docs are on the way..."
+printf "In order to use this project, until I get it to v1, you'll need to fill out a .env file and reference it in your project\n \
+    After v1, you should just need to have the account at AWS and your creds either in your path or in the .env file because\n \
+    the code will interrogate your account to find SQS queue urls, Kinesis stream names etc."
+printf "In order to test this project, until I get it to v1, you'll need to fill out a .env file and reference it in the\n \
+    spec/ directory.  There should be one with values that might be used by whatever spec you're running."

data/cloud_powers.gemspec

@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require 'cloud_powers/version'
 
 Gem::Specification.new do |spec|
-  # spec.required_ruby_version  =      '~> 2.3.0'
+  spec.required_ruby_version  =       '~> 2.3.0'
   spec.name                   =       'cloud_powers'
   spec.version                =       CloudPowers::VERSION
   spec.author                 =       'Adam Phillipps'
@@ -15,7 +15,7 @@ Gem::Specification.new do |spec|
     This wrapper was developed specifically for the Brain project
     but can be used in any other ruby project.
   EOF
-  spec.homepage               =       'https://smashanalytics.atlassian.net/wiki/display/SB/Cloud+Powers'
+  spec.homepage               =       'https://github.com/adam-phillipps/cloud_powers'
   spec.license                =       'MIT'
 
   spec.files                  =      `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }

data/lib/cloud_powers/helper.rb

@@ -42,36 +42,59 @@ module Smash
         end
       end
 
+      # Gets the path from the environment and sets @log_file using the path
+      # @returns @log_file path <String>
       def log_file
         @log_file ||= env('LOG_FILE')
       end
 
+      # @returns: An instance of Logger, cached as @logger
       def logger
         @logger ||= create_logger
       end
 
+      # Lets you retry a piece of logic with 1 second sleep in between attempts
+      # until another bit of logic does what it's supposed to, kind of like
+      # continuing to poll something and doing something when a package is ready
+      # to be taken and processed.
+      # @params:
+      #   * [allowed_attempts] or Infinity(default) <Number>: The number of times
+      #       the loop should be allowed to...well, loop, before a failed retry occurs.
+      #   * &test <Block>: A predicate method or block of code that is callable
+      #       is used to test if the block being retried is successful yet.
+      #   * []
       # Sample usage:
       # check_stuff = lambda { |params| return true }
-      # retry(3, check_stuff(params)) { do_stuff_that_needs_to_be_checked }
-      def retry(allowed_attempts = Float::Infinity, &test)
+      # smart_retry(3, check_stuff(params)) { do_stuff_that_needs_to_be_checked }
+      def smart_retry(test, allowed_attempts = Float::INFINITY)
         result = yield if block_given?
         tries = 1
         until test.call(result) || tries >= allowed_attempts
           result = yield if block_given?
+          tries += 1
           sleep 1
         end
       end
 
+      # Gives the path from the project root to lib/tasks[/#{file}.rb]
+      # @params:
+      #   * [file] <String>: name of a file
+      # @returns:
+      #   * path[/file] <String>
+      #   * If a `file` is given, it will have a '.rb' file extension
+      #   * If no `file` is given, it will return the `#task_require_path`
       def task_path(file = '')
-        # t_p = Pathname(__FILE__).parent.dirname + 'tasks'
-        if file.empty?
-          Pathname(__FILE__).parent.dirname + 'tasks'
-        else
-          Pathname(__FILE__).parent.dirname + "tasks/#{to_snake(file)}"
-        end
+        return task_require_path if file.empty?
+        Pathname(__FILE__).parent.dirname + 'tasks' + to_ruby_file_name(file)
       end
 
-      def task_require_path(file_name)
+      # Gives the path from the project root to lib/tasks[/file]
+      # @params:
+      #   * [file] <String>: name of a file
+      # @returns:
+      #   * path[/file] <String>
+      #   * Neither path nor file will have a file extension
+      def task_require_path(file_name = '')
         file = File.basename(file_name, File.extname(file_name))
         Pathname(__FILE__).parent.dirname + 'tasks' + file
       end
@@ -93,12 +116,19 @@ module Smash
       end
 
       def to_ruby_file_name(name)
-        "#{to_snake(name)}.rb"
+        name[/\.rb$/].nil? ? "#{to_snake(name)}.rb" : "#{to_snake(name)}"
       end
 
       def to_snake(var)
-        file_ext = var.to_s[/\.{1}[a-z]+$/] || ''
-        var.to_s.gsub(/\.\w+$/, '').gsub(/\W/, '_').downcase + file_ext
+        var = var.to_s unless var.kind_of? String
+
+        # var.gsub(/\W/, '_').downcase
+        var.gsub(/:{2}|\//, '_').
+          gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
+          gsub(/([a-z\d])([A-Z])/,'\1_\2').
+          gsub(/\s+/, '_').
+          tr("-", "_").
+          downcase
       end
 
       def update_message_body(opts = {})

data/lib/cloud_powers/self_awareness.rb

@@ -13,6 +13,9 @@ module Smash
       extend Smash::CloudPowers::Synapse::Queue
       include Smash::CloudPowers::AwsResources
 
+      # gets the instance time or the time it was called and as seconds from
+      # epoch
+      # TODO: use time codes
       def boot_time
         begin
           @boot_time ||=
@@ -24,6 +27,7 @@ module Smash
         end
       end
 
+      # Send a status message on the status Pipe then terminates the instance.
       def die!
         Thread.kill(@status_thread) unless @status_thread.nil?
         # blame = errors.sort_by(&:reverse).last.first
@@ -74,6 +78,7 @@ module Smash
         end.first
       end
 
+      # Gets and sets the public hostname of the instance
       def instance_url
         @instance_url ||= if env('TESTING')
           'https://test-url.com'
@@ -83,6 +88,10 @@ module Smash
         end
       end
 
+      # Makes the http request to self/meta-data to get all the metadata keys or,
+      # if a key is given, the method makes the http request to get that
+      # particular key from the metadata
+      # @param: [key <String>]
       def metadata_request(key = '')
         unless env('TESTING')
           metadata_uri = "http://169.254.169.254/latest/meta-data/#{key}"

data/lib/cloud_powers/version.rb

@@ -1,3 +1,3 @@
 module CloudPowers
-  VERSION = "0.1.4"
+  VERSION = "0.1.5"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: cloud_powers
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.1.5
 platform: ruby
 authors:
 - Adam Phillipps
@@ -159,7 +159,6 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- ".byebug_history"
 - ".gitignore"
 - ".rspec"
 - ".ruby-version"
@@ -185,7 +184,7 @@ files:
 - lib/cloud_powers/synapse/synapse.rb
 - lib/cloud_powers/version.rb
 - lib/cloud_powers/workflow.rb
-homepage: https://smashanalytics.atlassian.net/wiki/display/SB/Cloud+Powers
+homepage: https://github.com/adam-phillipps/cloud_powers
 licenses:
 - MIT
 metadata: {}
@@ -195,9 +194,9 @@ require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - "~>"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 2.3.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="

data/.byebug_history

@@ -1,256 +0,0 @@
-exit
-Smash::CloudPowers.included_modules
-Smash::CloudPowers
-Pipe
-Queue
-CloudPowers.public_methods(false)
-CloudPowers.public_methods
-CloudPowers.included_modules
-CloudPowers.methods.sort
-CloudPowers
-self.methods.sort
-self.modules
-c
-exit
-Smash.const_get(to_pascal(task)).new
-Smash.const_get(to_pascal(task))
-to_pascal(task)
-c
-exit
-n
-var.to_s[/\.{1}[a-z]+$/]
-l
-l-
-u = basename(Pathname(var))
-u
-v
-var.to_s.gsub(/\.\w+$/, '').gsub(/\W/, '_').downcase + (file_ext || '')
-var.to_s.gsub(/\.\w+$/, '').gsub(/\W/, '_').downcase + file_ext
-file_ext = var.to_s[/\.{1}[a-z]+$/]
-var.to_s[/\.{1}[a-z]+$/]
-var.to_s
-eval 'var'
-var
-s
-name
-to_snake(name)
-s
-task
-n
-body
-n
-s
-c
-n
-s
-c
-n
-exit
-FileUtils::touch(task_path('testinz.rb'))
-FileUtils::mkdir_p task_path
-task_path
-FileUtils::touch(task_path('testinz.rb'))
-task_path('testinz.rb')
-exit
-Pathname(__FILE__).parent.dirname + "tasks/#{to_snake(file)}"
-file.empty?
-s
-exit
-Pathname(__FILE__).dirname.parent + 'tasks' + ''
-to_snake(t_p)
-t_p
-file.empty? ? t_p : Pathname(to_snake(t_p))
-file
-t_p
-n
-s
-c
-b = ok['.rb']
-ok
-ok['.rb']
-ok = 'bla.rb'
-s3
-s3.stub_responses(:list_buckets, [])
-c
-exit
-@uuu
-n
-exit
-File.open '/Users/adam/code/cloud_powers/lib/tasks/testinz.rb'
-task_path 'testinz'
-task_path
-task_path_name
-exit
-ext
-exit
-FileUtils.ls(task_path '')
-FileUtils.ls(task_path)
-task_path('testinz') + '.rb'
-c
-exit
-puts ENV.keys.sort
-ENV
-key
-s
-region
-s
-s3.list_objects(bucket: bucket).contents
-bucket = 'jobRequests'
-bucket
-n
-task_path(file).exist?
-n
-s
-n
-s
-c
-s
-n
-task
-n
-msg.body
-msg
-s
-Smash::Testinz.new
-Smash::Testinz
-c
-name
-s
-c
-b = OpenStruct.new(name: 'yo', body: {})
-require 'ostruct'
-k = Struct.new(name: 'ok')
-JSON.parse(msg.body)
-s
-exit
-exception
-n
-{ body: {} }.to_json
-c
-exit
-self.class
-build
-Delegator.build
-c
-exit
-Delegator.build
-exit
-build
-l-
-Smash::CloudPowers::Delegator.methods.sort
-puts Delegator.class
-puts Delegator.methods.sort
-Delegator.class
-Delegator
-c
-exit
-@instance_url =~ URI::regexp
-@instance_url
-exit
-get_instance_url
-instance_url
-@instance_url
-c
-n
-c
-expect(now).to be_less_than(boot_time)
-boot_time
-task_name
-@boot_time
-@task_name
-n
-s
-exit
-l
-l-
-ec2.describe_instances(instance_ids: [ids].flatten).reservations[0].instances[0].tags.select { |t| t.value if t.key == 'taskType' }
-env '*'
-env *
-env
-ec2.describe_instances(instance_ids: [ids].flatten).reservations[0].instances[0].tags.select { |t| t.value if t.key == 'taskType' }
-ec2.stub_responses
-ec2.stub_response
-ec2.describe_instances(instance_ids: [ids].flatten)
-ids
-s
-@task_name
-s
-n
-@boot_time
-n
-c
-exit
-e
-format_error_message e
-e
-s
-format_error_message e
-format_error_message
-n
-ec2.describe_instances(dry_run: env('testing'), instance_ids:[@instance_id]).reservations[0].instances[0].launch_time.to_i
-@instance_id
-n
-s
-n
-exit
-n
-value
-key
-n
-key
-n
-s
-block_given?
-key
-n
-s
-keys
-n
-key
-n
-env('TESTING')
-s
-exit
-get_awareness!
-methods.sort
-get!
-@instance_id
-@boot_time
-n
-s
-exit
-region
-boot_time
-exit
-f = File.new('.test.env')
-`pwd`
-pwd
-Dotenv.load('.test.env')
-Dotenv.load(.test.env)
-require 'dotenv'
-c
-exit
-ENV[to_snake(key).upcase]
-require 'dotenv'
-s
-n
-s
-l-
-l
-puts ENV.keys.sort
-ENV['TESTING']
-ENV['testing']
-region
-n
-s
-c
-exit
-q = Boogs.new
-exit
-build
-self.extend Smash::Delegator
-exit
-build
-self.extend Smash::Delegator
-exit
-self