s3_website_revived 4.0.0

data/additional-docs/debugging.md

@@ -0,0 +1,21 @@
+# Tips for debugging
+
+## Debugging with source code
+
+First, clone the git repository:
+
+    git clone https://github.com/ivoanjo/s3_website_revived.git /tmp/s3_website
+
+Next, edit a source file.
+
+For example, you can change the AWS logging level in the
+[src/main/resources/log4j.properties](https://github.com/laurilehmijoki/s3_website/blob/master/src/main/resources/log4j.properties) file. See [AWS SDK for Java docs](http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-logging.html).
+
+Another example: modify a `.scala` file by adding a `print()` statement into a
+relevant location.
+
+Then push your website with the cloned code:
+
+    cd YOUR_WEBSITE_DIR
+    /tmp/s3_website/bin/s3_website push
+

data/additional-docs/development.md

@@ -0,0 +1,29 @@
+## Coding
+
+Install a Scala editor. Intellij IDEA has great Scala support.
+
+If you use IDEA, install the [Grep
+Console](http://plugins.jetbrains.com/plugin/?idea&pluginId=7125) plugin. It
+shows the ANSI colors in your IDEA console.
+
+### Test runs with IDEA
+
+1. Create a run profile: *Run* –> *Edit Configurations...*.
+2. Add *Application*
+3. Set *Main class* to `s3.website.Push`
+4. Set *Program arguments* to `--site=/Users/you/yourtestsite/_site --config-dir=/Users/you/yourtestsite --verbose`
+
+## Automated tests
+
+    ./sbt test
+
+## Test Linux distributions
+
+Use Vagrant for testing the installation procedure on Linux.
+
+Here's howto:
+
+1. Install <https://www.vagrantup.com/downloads.html>
+2. `cd vagrant && vagrant status`
+3. launch with `vagrant up <name>` and ssh into with `vagrant ssh <name>`
+4. test the latest release with `gem install s3_website && s3_website push`

data/additional-docs/example-configurations.md

@@ -0,0 +1,113 @@
+# Example `s3_website` configurations
+
+This document shows examples of complete `s3_website.yml` configurations.
+
+## Minimal
+
+````yaml
+s3_bucket: your.domain.net
+````
+
+This configuration will use AWS access credentials from the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. If those are not set, it will fall back to the credentials saved by `aws configure`.
+
+If you run `s3_website` on an EC2 instance with IAM roles, this configuration will use the instance's role instead.
+
+## Minimal with explicit credentials
+
+````yaml
+s3_id: abcd
+s3_secret: 2s+x92
+s3_bucket: your.domain.net
+````
+
+Use caution when embedding AWS credentials directly in `s3_website.yml`. Do not commit the file to a public Git repository or share it publicly.
+
+## Minimal for temporary security credentials
+
+````yaml
+s3_id: abcd
+s3_secret: 2s+x92
+session_token: hex!xeh
+s3_bucket: your.domain.net
+````
+
+## Optimised for speed
+
+Use CloudFront, gzip, cache headers and greater concurrency:
+
+````yaml
+s3_bucket: your.domain.net
+cloudfront_distribution_id: <%= ENV['your_domain_net_cloudfront_distribution_id'] %>
+cloudfront_distribution_config:
+  default_cache_behavior:
+    min_ttl: <%= 60 * 60 * 24 %>
+  aliases:
+    quantity: 1
+    items:
+      - your.domain.net
+max_age: 120
+gzip: true
+````
+
+In this example, we keep the CloudFront distribution ID in an environment variable.
+This is convenient, since you can keep the `s3_website.yml` in a public Git repo, and
+thus have your deployment configurations version-controlled.
+
+## Setup for HTTP2 and Custom SNI SSL Certificate
+
+To fully utilize HTTP2 you'll need to setup SSL for your distribution. While HTTP/2 does
+not mandate the use of encryption, it turns out that [all of the common web browsers 
+require the use of HTTPS connections in conjunction with HTTP/2](http://caniuse.com/#feat=http2).
+Therefore, you may need to make some changes to your site or application in order 
+to take full advantage of HTTP/2. While you can test the site by using the Default
+CloudFront Certificate you will likely want to use a custom SSL Certificate. 
+This isn't yet automated by s3_website, [but is a few manual steps](https://medium.com/@richardkall/setup-lets-encrypt-ssl-certificate-on-amazon-cloudfront-b217669987b2#.7jyust8os), 
+which is now free thanks to Let's Encrypt. 
+
+````yaml
+s3_bucket: your.domain.net
+cloudfront_distribution_id: <%= ENV['your_domain_net_cloudfront_distribution_id'] %>
+cloudfront_distribution_config:
+  default_cache_behavior:
+    min_ttl: <%= 60 * 60 * 24 %>
+  http_version: http2
+max_age: 120
+gzip: true
+````
+
+## Multiple CNAMEs
+
+Sometimes you want to use multiple CNAMEs aliases in your CloudFront distribution:
+
+````yaml
+s3_bucket: your.domain.net
+cloudfront_distribution_id: <%= ENV['your_domain_net_cloudfront_distribution_id'] %>
+cloudfront_distribution_config:
+  default_cache_behavior:
+    min_ttl: <%= 60 * 60 * 24 %>
+  aliases:
+    quantity: 3
+    items:
+      - your1.domain.net
+      - your2.domain.net
+      - your3.domain.net
+max_age: 120
+gzip: true
+````
+
+Always remember to set the 'quantity' property to match the number of items you have.
+
+## Using redirects
+
+````yaml
+redirects:
+  index.php: /
+  about.php: about.html
+routing_rules:
+  - condition:
+      key_prefix_equals: code/repositories/git/
+    redirect:
+      host_name: git.johnny.com
+      replace_key_prefix_with: ""
+      http_redirect_code: 301
+````

data/additional-docs/running-from-ec2-with-dropbox.md

@@ -0,0 +1,6 @@
+# Running from an EC2 instance with jekyll files on Dropbox
+
+Based on [this](http://namelesshorror.com/2015/02/26/jekyll-dropbox-aws-and-ifttt-easy-blogging/) article about automating the deployment of jekyll via an EC2 instance, I wrote the following shell script for use with such a setup. It assumes that you have a Linux Amazon EC2 instance, with the Dropbox client running as a daemon, and s3_website installed and configured. The script could be run from the default user's cron every so often, and allow anyone to effectively serve their jekyll source files to AWS from Dropbox.
+
+
+[jekyll-s3-dropbox.sh](https://gist.github.com/RNCTX/359489a5432937578bf5736850917d70)

data/additional-docs/setting-up-aws-credentials.md

@@ -0,0 +1,52 @@
+# Setting up AWS credentials
+
+Before starting to use s3\_website, you need to create AWS credentials.
+
+## Easy setup
+
+* Go to [AWS IAM console](https://console.aws.amazon.com/iam)
+* Create a new user that has full permissions to the S3 and CloudFront services
+* Call `s3_website cfg create` and place the credentials of your new AWS user
+  into the *s3_website.yml* file
+* Read the main documentation for further info
+
+## Limiting the permissions of the credentials
+
+AWS IAM offers multiple ways of limiting the permissions of a user. Below is one
+way of configuring the limitations and yet retaining the capability to use all
+s3\_website features.
+
+If you know the hostname of your public website (say `my.website.com`), perform the
+following steps:
+
+* Create a user that has full permissions to the S3 bucket
+* In addition, let the user have full permissions to CloudFront
+
+Here is the IAM Policy Document of the above setup:
+
+```json
+{
+  "Version":"2012-10-17",
+  "Statement": [
+    {
+      "Action": [
+        "cloudfront:*"
+      ],
+      "Effect": "Allow",
+      "Resource": [
+        "*"
+      ]
+    },
+    {
+      "Action": [
+        "s3:*"
+      ],
+      "Effect": "Allow",
+      "Resource": [
+        "arn:aws:s3:::my.website.com",
+        "arn:aws:s3:::my.website.com/*"
+      ]
+    }
+  ]
+}
+```

data/assembly.sbt

@@ -0,0 +1,3 @@
+import AssemblyKeys._ // put this at the top of the file
+
+assemblySettings

data/bin/s3_website

@@ -0,0 +1,285 @@
+#!/usr/bin/env ruby
+
+require File.expand_path(File.dirname(__FILE__)+ '/../lib/s3_website')
+require 'colored'
+require 'digest/md5'
+
+class Cfg < Thor
+  def self.exit_on_failure?
+    true
+  end
+
+  desc 'create', 'Create a config file with placeholder values'
+  def create
+    config_file_src = File.dirname(__FILE__) + '/../resources/configuration_file_template.yml'
+    config_file = Dir.pwd + '/s3_website.yml'
+    unless File.exists? config_file
+      require 'fileutils'
+      FileUtils.cp config_file_src, config_file
+      puts "Created the config file s3_website.yml. Go fill it with your settings."
+    end
+  rescue Exception => error
+    puts "#{error.message} (#{error.class})"
+    exit 1
+  end
+
+  option(
+    :headless,
+    :type => :boolean,
+    :desc => "Apply the settings headlessly. See --autocreate-cloudfront-distribution for more info."
+  )
+  option(
+    'autocreate-cloudfront-dist',
+    :type => :boolean,
+    :desc => "When used with --headless, automatically create a CloudFront distribution for your S3 website."
+  )
+  option(
+    'config-dir',
+    :type => :string,
+    :desc => "The directory where your config file is. When not defined, s3_website will look in the current working directory.",
+    :default => "."
+  )
+  desc 'apply', 'Apply the configuration on the AWS services'
+  long_desc <<-LONGDESC
+    `s3_website cfg apply` will apply the configuration the S3 bucket.
+
+    In addition, if you CloudFront related settings, this command will apply them on the CloudFront distribution.
+
+    If the S3 bucket does not exist, this command will create it
+    and configure it to function as a website.
+  LONGDESC
+  def apply
+    puts 'Applying the configurations in s3_website.yml on the AWS services ...'
+    require 'configure-s3-website'
+    config_source = ConfigureS3Website::FileConfigSource.new "#{options['config-dir']}/s3_website.yml"
+    ConfigureS3Website::Runner.run({
+      :config_source => config_source,
+      :headless => options[:headless],
+      'autocreate-cloudfront-dist' => options['autocreate-cloudfront-dist']
+    })
+  rescue Exception => error
+    puts "#{error.message} (#{error.class})"
+    exit 1
+  end
+end
+
+class Cli < Thor
+  def self.exit_on_failure?
+    true
+  end
+
+  option(
+    :site,
+    :type => :string,
+    :desc => "The directory where your website files are. When not defined, s3_website will look for the site in either _site or public/output."
+  )
+  option(
+    'config-dir'.to_sym,
+    :type => :string,
+    :desc => "The directory where your config file is. When not defined, s3_website will look in the current working directory."
+  )
+  option(
+    :verbose,
+    :type => :boolean,
+    :default => false,
+    :desc => "Print verbose output"
+  )
+  option(
+    :force,
+    :type => :boolean,
+    :default => false,
+    :desc => "Skip diff calculation and push all the files. This option is useful when you need to update metadata on the S3 objects."
+  )
+  option(
+    :dry_run,
+    :type => :boolean,
+    :default => false,
+    :desc => "Run the operation without actually making the modifications. When this switch is on, changes will not be applied on the S3 website. They will be only printed to the console."
+  )
+  desc 'push', 'Push local files with the S3 website'
+  long_desc <<-LONGDESC
+    `s3_website push` will upload new and changes files to S3. It will
+    also delete from S3 the files that you no longer have locally.
+  LONGDESC
+  def push
+    project_root = File.expand_path(File.dirname(__FILE__)+ '/..')
+    logger = Logger.new(options[:verbose])
+    success =
+      if false && run_with_sbt(project_root) # TODO: Disabled this because current sbt is too old
+        Dir.chdir(project_root) {
+          system './sbt assembly' # Build the jar
+        }
+        system "java -cp #{project_root}/target/scala-2.11/s3_website.jar #{resolve_java_command 'push'}"
+      else
+        # Find the jar
+        jar_file = resolve_jar(project_root, logger)
+        # Then run it
+        run_s3_website_jar(jar_file, logger)
+      end
+
+    if success
+      exit 0
+    else
+      exit 1
+    end
+  end
+
+  desc 'install', 'Download s3_website.jar'
+  long_desc <<-LONGDESC
+    Use `s3_website install` to download the s3_website.jar dependency
+    before you run `s3_website push` (good for CI dependency caching).
+  LONGDESC
+  def install
+    project_root = File.expand_path(File.dirname(__FILE__)+ '/..')
+    logger = Logger.new(options[:verbose])
+    resolve_jar(project_root, logger)
+  end
+
+  desc 'cfg SUBCOMMAND ...ARGS', 'Operate on the config file'
+  subcommand 'cfg', Cfg
+end
+
+def run_with_sbt(project_root)
+  File.exists?(project_root + '/project/sbt-launch.jar')
+end
+
+def resolve_java_command(command_name)
+  args = ARGV.join(' ').sub(command_name, '')
+  "s3.website.#{command_name.capitalize} #{args}"
+end
+
+def run_s3_website_jar(jar_file, logger)
+  java_installed = (resolve_exit_status('which java') or resolve_exit_status('java -version'))
+  unless java_installed
+    logger.info_msg "Cannot find Java. s3_website push is implemented in Scala, and it needs Java to run."
+    autoinstall_java_or_print_help_and_exit(logger)
+  end
+  logger.debug_msg "Using #{jar_file}"
+  system("java -cp #{jar_file} #{resolve_java_command 'push'}")
+end
+
+def resolve_exit_status(cmd)
+  `#{cmd}`
+  cmd_succeeded = $? == 0
+rescue
+  cmd_succeeded = false
+end
+
+def autoinstall_java_or_print_help_and_exit(logger)
+  @logger = logger
+  automatic_methods = [
+    {
+      :package_manager_lookup => 'which apt-get',
+      :install_command => 'sudo apt-get install --assume-yes openjdk-7-jre'
+    },
+    {
+      :package_manager_lookup => 'which yum',
+      :install_command => 'sudo yum install --assumeyes java-1.7.0-openjdk'
+    },
+    {
+      :package_manager_lookup => 'which dnf',
+      :install_command => 'sudo dnf install --assumeyes java-1.7.0-openjdk'
+    }
+  ]
+
+  def print_manual_method_and_exit
+    @logger.info_msg 'Go to http://java.com, install Java and then try again.'
+    @logger.info_msg "(If you cannot or do not want to install Java, you can use latest 1.x version of this gem, which requires only Ruby. For more info, see https://github.com/laurilehmijoki/s3_website/tree/1.x)"
+    exit 1
+  end
+
+  automatic_method = automatic_methods.find { |automatic_method|
+    resolve_exit_status automatic_method.fetch(:package_manager_lookup)
+  }
+  if automatic_method
+    @logger.info_msg "Do you want me to install Java with the command `#{automatic_method.fetch(:install_command)}`? [Y/n]"
+    user_answer = $stdin.gets
+    if user_answer.chomp.downcase == 'y' or user_answer == "\n"
+      automatic_method_succeeded = system automatic_method.fetch(:install_command)
+      unless automatic_method_succeeded
+        @logger.fail_msg "Could not automatically install Java. Try setting it up manually:"
+        print_manual_method_and_exit
+      end
+    else
+      print_manual_method_and_exit
+    end
+  else
+    print_manual_method_and_exit
+  end
+end
+
+def resolve_jar(project_root, logger)
+  jar_lookup_paths = [
+    project_root + "/s3_website-#{S3Website::VERSION}.jar",
+    (ENV['TMPDIR'] || '/tmp') + "/s3_website-#{S3Website::VERSION}.jar"
+  ]
+  found_jar = jar_lookup_paths.
+    select { |jar_path|
+      File.exist? jar_path
+    }.
+    first
+  # def jar_has_valid_checksum(jar_path, logger)
+  #   expected_checksum = File.read(File.dirname(__FILE__) + '/../resources/s3_website.jar.md5')
+  #   found_checksum = Digest::MD5.file(jar_path).hexdigest
+  #   if expected_checksum == found_checksum
+  #     true
+  #   else
+  #     logger.info_msg "The jar file is corrupted. (Expected checksum #{expected_checksum} but got #{found_checksum}.)"
+  #     false
+  #   end
+  # end
+  jar_file =
+    if found_jar #and jar_has_valid_checksum(found_jar, logger)
+      found_jar
+    else
+      download_jar(jar_lookup_paths, logger)
+    end
+end
+
+def download_jar(jar_lookup_paths, logger)
+  tag_name = "v#{S3Website::VERSION}"
+  downloaded_jar = jar_lookup_paths.select { |jar_path|
+    File.writable? File.dirname(jar_path)
+  }.first
+  unless downloaded_jar
+    logger.fail_msg "Neither #{jar_lookup_paths.join ' or '} is writable. Cannot download s3_website.jar."
+    logger.fail_msg "Set either directory as writable to the current user and try again."
+    exit 1
+  end
+  download_url = "https://github.com/laurilehmijoki/s3_website/releases/download/#{tag_name}/s3_website.jar"
+  logger.info_msg "Downloading #{download_url} into #{downloaded_jar}"
+  require 'open-uri'
+  open(downloaded_jar, 'wb') do |file|
+    file << URI.open(download_url).read
+  end
+  downloaded_jar
+end
+
+class Logger
+  attr_reader :verbose
+  def initialize(verbose)
+    @verbose = verbose
+  end
+
+  def debug_msg(msg)
+    if verbose
+      print_msg 'debg'.cyan, msg
+    end
+  end
+
+  def info_msg(msg)
+    print_msg 'info'.blue, msg
+  end
+
+  def fail_msg(msg)
+    print_msg 'fail'.red, msg
+  end
+
+  private
+
+  def print_msg(prefix, msg)
+    puts "[#{prefix}] #{msg}"
+  end
+end
+
+Cli.start(ARGV)

data/build.sbt

@@ -0,0 +1,48 @@
+import AssemblyKeys._
+
+name := "s3_website"
+
+version := "0.0.1"
+
+scalaVersion := "2.11.12"
+
+scalacOptions += "-feature"
+
+scalacOptions += "-language:implicitConversions"
+
+scalacOptions += "-language:postfixOps"
+
+scalacOptions += "-target:jvm-1.6"
+
+libraryDependencies += "org.yaml" % "snakeyaml" % "1.13"
+
+libraryDependencies += "org.jruby" % "jruby" % "9.3.3.0"
+
+libraryDependencies += "com.amazonaws" % "aws-java-sdk-s3" % "1.12.145" excludeAll ExclusionRule(organization = "com.fasterxml")
+libraryDependencies += "com.amazonaws" % "aws-java-sdk-cloudfront" % "1.12.145" excludeAll ExclusionRule(organization = "com.fasterxml")
+libraryDependencies += "com.amazonaws" % "aws-java-sdk-sts" % "1.12.145" excludeAll ExclusionRule(organization = "com.fasterxml")
+libraryDependencies += "javax.xml.bind" % "jaxb-api" % "2.3.1"
+
+libraryDependencies += "log4j" % "log4j" % "1.2.17"
+
+libraryDependencies += "commons-codec" % "commons-codec" % "1.9"
+
+libraryDependencies += "commons-io" % "commons-io" % "2.4"
+
+libraryDependencies += "org.apache.tika" % "tika-core" % "1.4"
+
+libraryDependencies += "com.lexicalscope.jewelcli" % "jewelcli" % "0.8.9"
+
+libraryDependencies += "org.specs2" %% "specs2" % "2.3.11" % "test"
+
+resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
+
+jarName in assembly := "s3_website.jar"
+
+test in assembly := {}
+
+mergeStrategy in assembly := {
+ case PathList("module-info.class", xs @ _*) => MergeStrategy.discard
+ case PathList("META-INF", "MANIFEST.MF", xs @ _*) => MergeStrategy.discard
+ case x => MergeStrategy.first
+}