r53z 0.2.0

data/LICENSE.txt

@@ -0,0 +1,16 @@
+Name: r53z
+Copyright (c) 2016 Joe Cooper
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+

data/README.md

@@ -0,0 +1,141 @@
+# R53z
+
+A simple CLI, REPL, and library for managing Route 53. It's primary purpose is to back up and restore Route 53 zones. It can write zones to files in JSON format. It also provides a simple API (not a whole lot easier than the Amazon official API, but for backups and restores, it's much easier to script with and removes tons of boilerplate).
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'r53z'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install r53z
+
+## Usage
+
+Configure a credentials file in `~/.aws/credentials` (this should be an INI file; same as several other AWS utilities expect). It'll look something like this:
+
+```ini
+[default]
+aws_access_key_id = ACCESS_KEY_ID
+aws_secret_access_key = SECRET_ACCESS_KEY
+region=us-east-1
+```
+
+You can use the `--section` option to choose which section to use from the credentials file, and the credentials file can be specified with the `--credentials` option. Region is irrelevant with Route 53, but the aws-sdk weirdly still requires it be present.
+
+```
+Usage: r53z [options] [args...]
+
+Simple CLI to manage, backup, and restore, Route 53 zones
+
+v0.2.0
+
+Options:
+    -h, --help                       Show command line help
+    -x, --export                     Export zones to files in specified directory, optionally specify one or more zones.
+    -r, --restore                    Restore zone from directory, optionally specify one or more zones.
+    -l, --list                       List name and ID of one or all zones.
+    -s, --record-sets                List record sets for the given zone.
+    -c, --create NAME                Create a zone with the given name and optional --comment and --delegation-set.
+    -n, --comment COMMENT            Optional comment when creating a zone.
+    -d, --delete                     Delete one or more zone(s) by name (WARNING: No confirmation!)
+    -C, --credentials                File containing credentials information.
+    -u, --section                    Section (user) in the credentials file to use.
+    -g, --delegation-set ID          Delegation set ID to use for various operations.
+    -t, --list-delegation-sets       List delegation set for named zone, or all sets if no zone specified.
+    -D, --delete-delegation-sets     Delete one or more delegation sets by ID (WARNING: No confirmation!
+    -N, --name-servers ID            List name servers for delegation set.
+        --version                    Show help/version info
+        --log-level LEVEL            Set the logging level
+                                     (debug|info|warn|error|fatal)
+                                     (Default: info)
+
+```
+
+### Command Line Options
+
+#### --export|-x {path} [zones]
+
+Export one or more zones to the named directory.
+
+Requires a directory path (e.g. /tmp/zonedumps), and optionally one or more zone names. All zones will be exported if none are specified. If a delegation set ID is given, only zones that share the given delegation set will be exported.
+
+Two files will be generated in the directory specified, one for the zone metadata (with a zoneinfo.json extension) and the zone records information (with a .json extension). The zone metadata file will contain all of the information needed to recreate the zone, incluing the delegation set ID. And, the record set file will contain all of the record sets needed to repopulate the zone; SOA and NS records are not restored as they are defined by the delegation set, rather than by records within the zone.
+
+##### Example
+
+```
+$ r53z --export /home/joe/zones swelljoe.com
+```
+
+#### --restore|-r {path} [zones]
+
+Restore one or more zones from files in the named directory.
+
+Requires a directory path, and optionally one or more zone names. If no names are given, all files in the directory will be restored. If a delegation set is specified, all zones will be added to the delegation set specified. (The zone info and the record sets don't contain delegation set information, making delegation set selection on restore a little difficult to automate.)
+
+If `--delegation-set` is specified on the command line, it will override the delegation-set information provided in the zoneinfo file. This can be used if the delegation set specified in the file is no longer available, for some reason (deletion, migrating to a new account, etc.).
+
+##### Example
+
+```
+$ r53z --restore /home/joe/zones swelljoe.com
+```
+
+#### --list|-l [--delegation-set ID] [zones]
+
+List hosted zones. List can be restricted to just the listed zones, or to a given delegation set. Output is JSON encoded, and will contain the name and ID of each zone. If no zones are specified, all zones will be listed.
+
+#### --create|-c NAME [--comment COMMENT] [--delegation-set ID]
+
+Create zone of the NAME provided. An optional command an delegation set ID may be provided.
+
+##### Example
+
+```
+$ r53z --create swelljoe.com --comment "My domain"
+```
+
+#### --delete|-d {zone}
+
+Delete one or more zones. Argument is the name of the zone, or zones, to delete. This command deletes the record sets for the zone first, and then deletes the zone itself (a zone with records cannot be deleted). There is no confirmation step for this option.
+
+##### Example
+
+```
+$ r53z --delete swelljoe.com virtualmin.com
+```
+
+#### --credentials|-c {path/filename}
+
+Specify the credentials configuration file on the command line. The file must be an INI file. By default, it will look for a file in ~/.aws/credentials (which is common across several AWS management tools). You can use the `--section` option to choose what section of the file to use.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`.
+
+#### Running Tests
+
+To run the full test suite, you need a read/write capable account. There must be a configuration file in `test/data/secret-credentials` containing a `[default]` section with your keys. The format of this file is, as above, a .ini file.
+
+To run all tests:
+
+```
+$ rake test
+```
+
+This will create a few test zones in your account, but unless something goes wrong during the test, they will be removed immediately after, never triggering billing from Amazon. The zones will have somewhat randomly generated names, so they should never clash with existing names (but you may wish to create a non-production account just for testing).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/swelljoe/r53z.

data/README.rdoc

@@ -0,0 +1 @@
+See README.md

data/Rakefile

@@ -0,0 +1,62 @@
+def dump_load_path
+  puts $LOAD_PATH.join("\n")
+  found = nil
+  $LOAD_PATH.each do |path|
+    if File.exists?(File.join(path,"rspec"))
+      puts "Found rspec in #{path}"
+      if File.exists?(File.join(path,"rspec","core"))
+        puts "Found core"
+        if File.exists?(File.join(path,"rspec","core","rake_task"))
+          puts "Found rake_task"
+          found = path
+        else
+          puts "!! no rake_task"
+        end
+      else
+        puts "!!! no core"
+      end
+    end
+  end
+  if found.nil?
+    puts "Didn't find rspec/core/rake_task anywhere"
+  else
+    puts "Found in #{path}"
+  end
+end
+require 'bundler'
+require 'rake/clean'
+
+require 'rake/testtask'
+
+require 'cucumber'
+require 'cucumber/rake/task'
+gem 'rdoc' # we need the installed RDoc gem, not the system one
+require 'rdoc/task'
+
+include Rake::DSL
+
+Bundler::GemHelper.install_tasks
+
+
+Rake::TestTask.new do |t|
+  t.pattern = 'test/tc_*.rb'
+  t.warning = false # Get rid of purious warnings from the AWS and Methadone libs?
+end
+
+
+CUKE_RESULTS = 'results.html'
+CLEAN << CUKE_RESULTS
+Cucumber::Rake::Task.new(:features) do |t|
+  t.cucumber_opts = "features --format html -o #{CUKE_RESULTS} --format pretty --no-source -x"
+  t.fork = false
+end
+
+Rake::RDocTask.new do |rd|
+  
+  rd.main = "README.rdoc"
+  
+  rd.rdoc_files.include("README.rdoc","lib/**/*.rb","bin/**/*")
+end
+
+task :default => [:test,:features]
+

data/bin/console

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "r53z"
+require "pry"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+creds = R53z::Config.new()
+client = R53z::Client.new('default', creds)
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+binding.pry
+
+#require "irb"
+#IRB.start

data/bin/r53z

@@ -0,0 +1,39 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'methadone'
+require 'r53z.rb'
+
+class App
+  include Methadone::Main
+  include Methadone::CLILogging
+
+  main do |*args|
+    help_now! "No options provided." if options.empty? 
+    R53z::Cli.new(:options => options, :args => args)
+  end
+
+  # Command line interface specification
+  description "Simple CLI to manage, backup, and restore, Route 53 zones"
+  #
+  # Accept flags via:
+  on("-x", "--export", "Export zones to files in specified directory, optionally specify one or more zones")
+  on("-r", "--restore", "Restore zone from directory, optionally specify one or more zones")
+  on("-l", "--list", "List name and ID of one or all zones")
+  on("-s", "--record-sets", "List record sets for the given zone")
+  on("-d", "--delete", "Delete one or more zone(s) by name (WARNING: No confirmation!)")
+  on("-c", "--credentials", "File containing credentials information")
+  on("-u", "--section", "Section (user) in the credentials file to use")
+  on("-g ID", "--delegation-set", "Delegation set ID to use for various operations")
+  on("-t", "--list-delegation-sets", "List all delegation sets")
+  on("-n ID", "--name-servers", "List name servers for delegation set")
+  # args greedily grabs any files or zones listed after the options
+  arg(:args, :any)
+
+  version R53z::VERSION
+
+  use_log_level_option :toggle_debug_on_signal => 'USR1'
+
+  go!
+end
+

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/r53z.rb

@@ -0,0 +1,10 @@
+require_relative "r53z/version"
+require "methadone"
+require "aws-sdk"
+require_relative "r53z/config"
+require_relative "r53z/client"
+require_relative "r53z/file"
+require_relative "r53z/cli"
+
+module R53z
+end

data/lib/r53z/cli.rb

@@ -0,0 +1,117 @@
+require 'json'
+
+module R53z
+  class Cli
+    include Methadone::Main
+    include Methadone::CLILogging
+
+    def initialize(options:, args:)
+      section = options[:section] || 'default'
+      config_file = options[:credentials]
+      creds = R53z::Config.new(config_file)
+      @client = R53z::Client.new(section, creds)
+
+      # XXX Dispath table seems smarter...can't figure out how to calls methods based
+      # directly on hash keys at the moment.
+      if options[:export]
+        help_now! "Export requires a directory path for zone files" if args.length < 1
+        export(:options => options, :args => args)
+      end
+
+      if options[:restore]
+        if args.empty?
+          help_now! "Restore requires a directory containing zone files and optionally one or more zones to restore"
+        end
+        restore(:options => options, :args => args)
+      end
+
+      if options[:list]
+        list(options: options, args: args)
+      end
+
+      if options[:delete]
+        if args.empty?
+          help_now! "Delete requires one or more zone names"
+        end
+        args.each do |name|
+          @client.delete(name)
+        end
+      end
+
+      if options['list-delegation-sets']
+        sets = @client.list_delegation_sets
+        sets.each do |set|
+          puts JSON.pretty_generate(set.to_h)
+        end
+      end
+
+      if options['record-sets']
+        if args.empty?
+          help_now! "List record sets requires one or more zone names"
+        end
+        args.each do |name|
+          sets = @client.record_list(@client.get_zone_id(name))
+          puts JSON.pretty_generate(sets)
+        end
+      end
+
+      if options['name-servers']
+        dset = @client.get_delegation_set(id: options['name-servers'])
+        puts JSON.pretty_generate(dset.delegation_set[:name_servers])
+      end
+    end
+
+    def export(options:, args:)
+      path = args.shift
+      # If no zones, dump all zones
+      zones = []
+      # One zone, multiple, or all?
+      if args.empty?
+        @client.list.each do |zone|
+          zones.push(zone[:name])
+        end
+      else
+        zones = args
+      end
+
+      zones.each do |name|
+        @client.dump(path, name)
+      end
+    end
+
+    def restore(options:, args:)
+      path = args.shift
+      # If no zones, restore all zones in directory
+      zones = []
+      if args.empty?
+        # derive list of zones from files in path
+        zones = Dir[File.join(path, "*.json")].reject {|n| n.match("zoneinfo")}
+      else
+        # restore the ones specified
+        args.each do |zone|
+          zones.push(zone)
+        end
+      end
+
+      zones.each do |zone|
+        @client.restore(path, zone)
+      end
+    end
+
+    def list(options:, args:)
+      if args.any?
+        args.each do |name|
+          puts JSON.pretty_generate(
+            @client.list(
+              :name => name,
+              :delegation_set_id => options['delegation-set']))
+        end
+      else
+        puts JSON.pretty_generate(
+          @client.list(:delegation_set_id => options['delegation-set'])
+        )
+      end
+    end
+  end
+end
+

data/lib/r53z/client.rb

@@ -0,0 +1,220 @@
+module R53z
+  class Client 
+    include Methadone::Main
+    include Methadone::CLILogging
+    attr_accessor :client
+
+    def initialize(section, creds)
+      @client = Aws::Route53::Client.new(
+        access_key_id: creds[section]['aws_access_key_id'],
+        secret_access_key: creds[section]['aws_secret_access_key'],
+        region: creds[section]['region']
+      )
+    end
+
+    # list one or all zones by name and ID
+    def list(name: nil, delegation_set_id: nil)
+      begin
+        zones = self.client.list_hosted_zones(
+          delegation_set_id: delegation_set_id
+          )['hosted_zones']
+      rescue Aws::Route53::Errors::ServiceError
+        error "Failed to list zones" # XXX How do we get AWS error message out of it?
+      end
+
+      rv = []
+      zones.each do |zone|
+        if name
+          unless name[-1] == '.'
+            name = name + '.'
+          end
+          unless name == zone[:name]
+            next
+          end
+        end
+        rv.push({:name => zone[:name], :id => zone[:id]})
+      end
+      rv
+    end
+
+    # Create zone with record(s) from an info and records hash
+    def create(info:, records:)
+      rv = {} # pile up the responses in a single hash
+      #if self.list(info[:name]).any?
+      #  error(info[:name] + "exists")
+      #end
+      # XXX: AWS sends out a data structure with config:, but expects
+      # hosted_zone_config on create/restore. argh.
+      # XXX: also, private_zone is not accepted here for some reason
+      zone = info[:hosted_zone]
+      zone_data = {
+        :name => zone[:name],
+        :caller_reference => 'r53z-create-' + self.random_string,
+        :hosted_zone_config => {
+          :comment => zone[:config][:comment]
+        }
+      }
+      # command line overrides everything else
+      if options['delegation-set']
+        zone_data[:delegation_set_id] = options['delegation-set']
+      elsif info[:delegation_set] and info[:delegation_set][:id]
+        zone_data[:delegation_set_id] = info[:delegation_set][:id]
+      end
+      zone_resp = self.client.create_hosted_zone(zone_data)
+      rv[:hosted_zone_resp] = zone_resp
+      rv[:record_set_resp] = []
+      records.each do |record|
+        # skip these, as they are handled separately (delegation set?)
+        unless (record[:type] == "NS" || record[:type] == "SOA")
+          record_resp = self.client.change_resource_record_sets({
+            :hosted_zone_id => zone_resp[:hosted_zone][:id],
+            :change_batch => {
+              :changes => [
+                {
+                  :action => "CREATE",
+                  :resource_record_set => record
+                }
+              ]
+            }
+          })
+          rv[:record_set_resp].push(record_resp)
+        end
+      end
+      return rv
+    end
+
+    # delete a zone by name
+    def delete(name)
+      # get the ID
+      zone_id = self.list(:name => name).first[:id]
+      self.delete_all_rr_sets(zone_id)
+      client.delete_hosted_zone(:id => zone_id)
+    end
+
+    # delete all of the resource record sets in a zone (this is required to delete
+    # a zone
+    def delete_all_rr_sets(zone_id)
+      self.record_list(zone_id).reject do |rs|
+        (rs[:type] == "NS" || rs[:type] == "SOA")
+      end.each do |record_set|
+        self.client.change_resource_record_sets({
+          :hosted_zone_id => zone_id,
+          :change_batch => {
+            :changes => [{
+              :action=> "DELETE",
+              :resource_record_set => record_set
+            }]
+          }
+        })
+      end
+    end
+
+    # dump a zone to a direcory. Will generate two files; a zoneinfo file and a 
+    # records file.
+    def dump(dirpath, name)
+      # Get the ID
+      zone_id = self.list(:name => name).first[:id]
+
+      # normalize name
+      unless name[-1] == '.'
+        name = name + '.'
+      end
+
+      # dump the record sets
+      R53z::JsonFile.write_json(
+        path: File.join(dirpath, name),
+        data: self.record_list(zone_id))
+
+      # Dump the zone metadata, plus the delegated set info
+      out = { :hosted_zone => 
+                self.client.get_hosted_zone({ :id => zone_id}).hosted_zone.to_h,
+              :delegation_set => 
+                self.client.get_hosted_zone({:id => zone_id}).delegation_set.to_h
+            }
+
+      R53z::JsonFile.write_json(
+        path: File.join(dirpath, name + "zoneinfo"),
+        data: out)
+        #data: self.client.get_hosted_zone({
+        #  :id => zone_id}).hosted_zone.to_h)
+    end
+
+    # Restore a zone from the given path. It expects files named
+    # zone.zoneinfo.json and zone.json
+    def restore(path, domain)
+      # normalize domain
+      unless domain[-1] == '.'
+        domain = domain + '.'
+      end
+      # Load up the zone info file
+      file = File.join(path, domain)
+      info = R53z::JsonFile.read_json(path: file + "zoneinfo")
+      records = R53z::JsonFile.read_json(path: file)
+      # create the zone and the record sets
+      self.create(:info => info, :records => records)
+    end
+
+    def record_list(zone_id)
+      records = self.client.list_resource_record_sets(hosted_zone_id: zone_id)
+      rv = []
+      records[:resource_record_sets].each do |record|
+        rv.push(record.to_h)
+      end
+      rv
+    end
+
+    # create a new delegation set, optionally associated with an existing zone
+    def create_delegation_set(zone_id = nil)
+      self.client.create_reusable_delegation_set({
+        caller_reference: 'r53z-create-del-set-' + self.random_string,
+        hosted_zone_id: zone_id
+      })
+    end
+
+    # list all delegation sets
+    def list_delegation_sets
+      resp = self.client.list_reusable_delegation_sets({})
+      return resp.delegation_sets
+    end
+    
+    # get details of a delegation set specified by ID, incuding name servers
+    def get_delegation_set(id)
+      self.client.get_reusable_delegation_set({
+        id: id
+      })
+    end
+
+    # delete a delegation set by ID or name
+    def delete_delegation_set(id: nil, name: nil)
+      if name and not id
+        id = get_delegation_set_id(name)
+      end
+      self.client.delete_reusable_delegation_set({
+        id: id
+      })
+    end
+
+    # Get delegation set ID for the given zone
+    def get_delegation_set_id(name)
+      begin
+        zone_id = self.list(:name => name).first[:id]
+      rescue
+        return nil
+      end
+      return self.client.get_hosted_zone({
+        id: zone_id 
+      }).delegation_set[:id]
+    end
+
+    # Get the zone id from name
+    def get_zone_id(name)
+      return self.list(:name => name).first[:id]
+    end
+
+    # random string generator helper function
+    def random_string(len=16)
+      rand(36**len).to_s(36)
+    end
+  end
+end
+