aws_one_click_staging 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d779231c0e0dc99fe5f3c7ed558c953517b52eb0
-  data.tar.gz: 6bad4ad285f0cc504c48b32efcecb0ed00660692
+  metadata.gz: eed057b20d3a62ae312c0c390fb47ede95b1eb03
+  data.tar.gz: 36281e22f2ea58d70d5e7674dc6d502972103a21
 SHA512:
-  metadata.gz: 20456486a2357d797831905748a5352a8ad32aed06e11682c2071c3957066ed1341b744f3b5831076f374b1cc3f15085dd5517e0d7f4f2e443573efb0e5c7a01
-  data.tar.gz: 2c3e1a8eb4829d92be2edab97e39fff8edc018a6f51e3c5f6712ad7fe003e1db97f3e0ee4b487e2a36466c30d10041650337e07b68a18781a1a1f1726c01778b
+  metadata.gz: 20af3681b0de1ad656917b8b5f56da342dec76467e17c0fdd7d190134c21137ca793e439d2b14fe77785988daeec644d8c71fd25f2ad6eac3f556d440cee6436
+  data.tar.gz: 87ffdce901f9e9881be8b5df95ed27de21923f3c5b2ae7772d503a28475c193e23c7160a353f392fc8cd4d0f0abab7e46b9686de70e928633fecc4b2903a8691

data/README.md

@@ -2,7 +2,7 @@
 
 AwsOneClickStaging is a CLI app that will allow you to create a clone of amazon's S3 bucket (named app_name-staging) and the RDS database (named app_name-staging) so your staging server will be 100% up to date with your production server and allow you to perform some serious testing without the fear of losing important files stored on your S3 bucket.  
 
-If you didn't already know, S3 is a file storage solution offered by Amazon, and it's not user friendly so you pretty much need to write/ use a script like this in order to setup a staging server.  RDS is Amazon's database storage.  As with all amazon services, RDS databases are in the clown so you can rely on it working well with your app and working hard to provide your users with hillarious service.  
+If you didn't already know, S3 is a file storage solution offered by Amazon, and it's not user friendly so you pretty much need to write/ use a script like this in order to setup a staging server.  RDS is Amazon's database storage.  As with all amazon services, RDS databases are in the clown so you can rely on it working well with your app and working hard to provide your users with hilarious service.  
 
 
 ## Installation
@@ -20,7 +20,6 @@ Next you'll need to setup the config file:
 aws_access_key_id: ""
 aws_secret_access_key: ""
 aws_region: 'us-west-1'
-aws_master_username: ""
 aws_master_user_password: ""
 aws_production_bucket: "" # this bucket is read from
 aws_staging_bucket: ""    # this bucket is DELETED and written to!
@@ -45,7 +44,7 @@ After a while, the operation will complete and it will say 'congrats' or somethi
 ## AWS Permissions
 
 Because you're a professional, you want to grant only the permissions absolutely necessary to the 'staging-bot' user.  
-That's commendable.  Use the below scripts and replace `PRODUCTIONDB` with the name of your production database/ s3 bucket (hopefully you used the same name for both).  These rules are set via the `Identity & Access Management` subconsole on amazon.  
+That's commendable.  Use the below rules, replacing `PRODUCTIONDB` with the name of your production database/ s3 bucket (hopefully you used the same name for both).  These rules are set via the `Identity & Access Management` subconsole on amazon.  
 
 (staging-bot-rds-can-do-anything-to-staging-db)
 ```
@@ -166,7 +165,7 @@ That's commendable.  Use the below scripts and replace `PRODUCTIONDB` with the n
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release` to create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-There's a couple unit tests with commented out method calls.  This was how I tested against amazon, simply uncomment a line, drop in a `binding.pry;exit!` and test/ debug what ever methods you think are messing up.  
+  There's a couple unit tests with commented out method calls.  This was how I tested against amazon, simply uncomment a line, drop in a `binding.pry;exit!` and test/ debug what ever methods you think are messing up.  Also, if you create the file config/aws_actual_fffing_secrets.yml, you can set the config variables that will be used throughout testing...  naturally, ensure such credentials never see the light of github.  
 
 
 ## Contributing
@@ -176,3 +175,8 @@ There's a couple unit tests with commented out method calls.  This was how I tes
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create a new Pull Request
+
+
+## Credits
+Cory Forsyth [Bantic](github.com/bantic) wrote the S3 synchronization code used by this gem.  
+rob@eff did the other stuff.  

data/changelog.md

@@ -0,0 +1,14 @@
+## Future
+
+* Should have multi-app support for config file
+
+
+## 0.1.1
+
+- Added multi-threaded support (up to 16 requests at a time!)
+
+## 0.1.0
+
+- released the gem kinda
+
+

data/config/aws_one_click_staging.yml

@@ -7,7 +7,6 @@ aws_access_key_id: ""
 aws_secret_access_key: ""
 
 aws_region: 'us-west-1'
-aws_master_username: ""
 aws_master_user_password: ""
 aws_production_bucket: "" # this bucket is read from
 aws_staging_bucket: ""    # this bucket is DELETED and written to!

data/lib/aws_one_click_staging/aws_warrior.rb

@@ -44,7 +44,7 @@ module AwsOneClickStaging
 
     def get_fancy_string_of_staging_db_uri
       setup_aws_credentials_and_configs
-      
+
       l = 66
       msg = ""
       msg += "*" * l + "\n"
@@ -61,7 +61,6 @@ module AwsOneClickStaging
       aws_region = @config["aws_region"]
       @access_key_id = @config["aws_access_key_id"]
       @secret_access_key = @config["aws_secret_access_key"]
-      @master_username = @config["aws_master_username"]
       @master_user_password = @config["aws_master_user_password"]
       @aws_production_bucket = @config["aws_production_bucket"]
       @aws_staging_bucket = @config["aws_staging_bucket"]
@@ -110,14 +109,25 @@ module AwsOneClickStaging
 
     def spawn_new_staging_db_instance!
       puts "Spawning a new fully clony RDS db instance for staging purposes"
-      response = @c.create_db_instance(db_instance_identifier: @db_instance_id_staging,
-        db_instance_class: "db.t1.micro",
-        engine: "postgres",
-        master_username: @master_username,
-        master_user_password: @master_user_password,
-        allocated_storage: "10")
+
+      @c.describe_db_snapshots(db_snapshot_identifier: @db_snapshot_id).db_snapshots.first
+
+      response = @c.restore_db_instance_from_db_snapshot(
+        db_instance_identifier: @db_instance_id_staging,
+        db_snapshot_identifier: @db_snapshot_id,
+        db_instance_class: "db.t1.micro"
+      )
+
 
       sleep 10 while get_fresh_db_instance_state(@db_instance_id_staging).db_instance_status != "available"
+
+      # sets password for staging db and disables automatic backups
+      response = @c.modify_db_instance(
+        db_instance_identifier: @db_instance_id_staging,
+        backup_retention_period: 0,
+        master_user_password: @master_user_password
+      )
+      sleep 2 while get_fresh_db_instance_state(@db_instance_id_staging).db_instance_status != "available"
     end
 
 

data/lib/aws_one_click_staging/bucket_sync_service.rb

@@ -7,13 +7,18 @@ class BucketSyncService
   attr_reader :from_bucket, :to_bucket, :logger
   attr_accessor :debug
 
+  # DEFAULT_ACL = :public_read
+  # PRIVATE_ACL = :private
+
+
+
   # from_credentials and to_credentials are both hashes with these keys:
   #  * :aws_access_key_id
   #  * :aws_secret_access_key
   #  * :bucket
   def initialize(from_credentials, to_credentials)
-    @from_bucket = get_bucket_reference(from_credentials)
-    @to_bucket   = get_bucket_reference(to_credentials)
+    @from_bucket = bucket_from_credentials(from_credentials)
+    @to_bucket   = bucket_from_credentials(to_credentials)
   end
 
   def perform(output=STDOUT)
@@ -35,7 +40,7 @@ class BucketSyncService
           object_counts[:skip] += 1
         end
       }
-      sleep 0.01 while threads.select {|t| t.alive?}.count > 16
+      sleep 0.01 while threads.select {|t| t.alive?}.count > 16 # throttling
     end
 
 
@@ -55,7 +60,17 @@ class BucketSyncService
 
   def sync(object)
     logger.debug "Syncing #{pp object}"
-    object.copy_to(to_bucket.objects[object.key])
+    acl_setting = file_is_public?(object) ? :public_read : :private
+    object.copy_to(to_bucket.objects[object.key], acl: acl_setting)
+  end
+
+  # Crude, but ala aws I think :)
+  def file_is_public?(object)
+    grants = object.acl.grants
+    grants.each do |g|
+      return true if g.permission.name == :read && g.grantee.uri == "http://acs.amazonaws.com/groups/global/AllUsers"
+    end
+    return false
   end
 
   def pp(object)
@@ -71,16 +86,15 @@ class BucketSyncService
     return to_object.etag != object.etag # does the etag on the dst object differ from src?
   end
 
-  def get_bucket_reference(credentials)
+
+  def bucket_from_credentials(credentials)
     s3 = AWS::S3.new(access_key_id:      credentials[:aws_access_key_id],
                      secret_access_key:  credentials[:aws_secret_access_key])
 
     bucket = s3.buckets[ credentials[:bucket] ]
-    create_bucket_if_needed!(bucket)
-  end
-
-  def create_bucket_if_needed!(bucket)
-    return s3.buckets.create( credentials[:bucket] ) if !bucket.exists?
+    if !bucket.exists?
+      bucket = s3.buckets.create( credentials[:bucket] )
+    end
     bucket
   end
 end

data/lib/aws_one_click_staging/version.rb

@@ -1,3 +1,3 @@
 module AwsOneClickStaging
-  VERSION = "0.1.1"
+  VERSION = "0.1.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: aws_one_click_staging
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
 platform: ruby
 authors:
 - TheNotary
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-11-27 00:00:00.000000000 Z
+date: 2015-12-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: aws-sdk
@@ -126,6 +126,7 @@ files:
 - aws_one_click_staging.gemspec
 - bin/aws_one_click_staging
 - changelog
+- changelog.md
 - config/aws_one_click_staging.yml
 - lib/aws_one_click_staging.rb
 - lib/aws_one_click_staging/aws_warrior.rb
@@ -151,7 +152,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.4.4
+rubygems_version: 2.4.8
 signing_key: 
 specification_version: 4
 summary: When setup with the proper credentials, this gem sets up a staging instance