s3_website_monadic 0.0.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (151) hide show
  1. checksums.yaml +7 -0
  2. data/.gitignore +10 -0
  3. data/.travis.yml +3 -0
  4. data/Gemfile +3 -0
  5. data/LICENSE +42 -0
  6. data/README.md +451 -0
  7. data/Rakefile +24 -0
  8. data/additional-docs/example-configurations.md +62 -0
  9. data/additional-docs/setting-up-aws-credentials.md +51 -0
  10. data/assembly.sbt +3 -0
  11. data/bin/s3_website +80 -0
  12. data/build.sbt +33 -0
  13. data/changelog.md +215 -0
  14. data/features/as-library.feature +29 -0
  15. data/features/cassettes/cucumber_tags/create-redirect.yml +384 -0
  16. data/features/cassettes/cucumber_tags/empty-bucket.yml +89 -0
  17. data/features/cassettes/cucumber_tags/new-and-changed-files.yml +303 -0
  18. data/features/cassettes/cucumber_tags/new-files-for-sydney.yml +211 -0
  19. data/features/cassettes/cucumber_tags/new-files.yml +355 -0
  20. data/features/cassettes/cucumber_tags/no-new-or-changed-files.yml +359 -0
  21. data/features/cassettes/cucumber_tags/one-file-to-delete.yml +390 -0
  22. data/features/cassettes/cucumber_tags/only-changed-files.yml +411 -0
  23. data/features/cassettes/cucumber_tags/s3-and-cloudfront-after-deleting-a-file.yml +434 -0
  24. data/features/cassettes/cucumber_tags/s3-and-cloudfront-when-updating-a-file.yml +435 -0
  25. data/features/cassettes/cucumber_tags/s3-and-cloudfront.yml +290 -0
  26. data/features/cloudfront.feature +54 -0
  27. data/features/command-line-help.feature +54 -0
  28. data/features/delete.feature +19 -0
  29. data/features/error_reporting.feature +24 -0
  30. data/features/instructions-for-new-user.feature +154 -0
  31. data/features/jekyll-support.feature +20 -0
  32. data/features/nanoc-support.feature +20 -0
  33. data/features/push.feature +115 -0
  34. data/features/redirects.feature +14 -0
  35. data/features/security.feature +15 -0
  36. data/features/step_definitions/steps.rb +86 -0
  37. data/features/support/env.rb +26 -0
  38. data/features/support/test_site_dirs/cdn-powered.blog.fi/_site/css/styles.css +3 -0
  39. data/features/support/test_site_dirs/cdn-powered.blog.fi/_site/index.html +5 -0
  40. data/features/support/test_site_dirs/cdn-powered.blog.fi/s3_website.yml +4 -0
  41. data/features/support/test_site_dirs/cdn-powered.when-deleted-a-file.blog.fi/_site/index.html +10 -0
  42. data/features/support/test_site_dirs/cdn-powered.when-deleted-a-file.blog.fi/s3_website.yml +5 -0
  43. data/features/support/test_site_dirs/cdn-powered.with-one-change.blog.fi/_site/css/styles.css +3 -0
  44. data/features/support/test_site_dirs/cdn-powered.with-one-change.blog.fi/_site/index.html +10 -0
  45. data/features/support/test_site_dirs/cdn-powered.with-one-change.blog.fi/s3_website.yml +4 -0
  46. data/features/support/test_site_dirs/create-redirects/_site/.gitkeep +0 -0
  47. data/features/support/test_site_dirs/create-redirects/s3_website.yml +6 -0
  48. data/features/support/test_site_dirs/ignored-files.com/_site/css/styles.css +4 -0
  49. data/features/support/test_site_dirs/ignored-files.com/_site/index.html +8 -0
  50. data/features/support/test_site_dirs/ignored-files.com/s3_website.yml +5 -0
  51. data/features/support/test_site_dirs/index-and-assets.blog.fi/_site/assets/picture.gif +0 -0
  52. data/features/support/test_site_dirs/index-and-assets.blog.fi/_site/css/styles.css +3 -0
  53. data/features/support/test_site_dirs/index-and-assets.blog.fi/_site/index.html +5 -0
  54. data/features/support/test_site_dirs/index-and-assets.blog.fi/s3_website.yml +3 -0
  55. data/features/support/test_site_dirs/jekyllrb.com/_site/css/styles.css +3 -0
  56. data/features/support/test_site_dirs/jekyllrb.com/_site/index.html +5 -0
  57. data/features/support/test_site_dirs/jekyllrb.com/s3_website.yml +3 -0
  58. data/features/support/test_site_dirs/my.blog-with-clean-urls.com/_site/css/styles.css +3 -0
  59. data/features/support/test_site_dirs/my.blog-with-clean-urls.com/_site/index +5 -0
  60. data/features/support/test_site_dirs/my.blog-with-clean-urls.com/s3_website.yml +3 -0
  61. data/features/support/test_site_dirs/my.blog.com/_site/css/styles.css +3 -0
  62. data/features/support/test_site_dirs/my.blog.com/_site/index.html +5 -0
  63. data/features/support/test_site_dirs/my.blog.com/s3_website.yml +3 -0
  64. data/features/support/test_site_dirs/my.sydney.blog.au/_site/css/styles.css +3 -0
  65. data/features/support/test_site_dirs/my.sydney.blog.au/_site/index.html +5 -0
  66. data/features/support/test_site_dirs/my.sydney.blog.au/s3_website.yml +4 -0
  67. data/features/support/test_site_dirs/nanoc.ws/public/output/css/styles.css +3 -0
  68. data/features/support/test_site_dirs/nanoc.ws/public/output/index.html +5 -0
  69. data/features/support/test_site_dirs/nanoc.ws/s3_website.yml +3 -0
  70. data/features/support/test_site_dirs/new-and-changed-files.com/_site/css/styles.css +4 -0
  71. data/features/support/test_site_dirs/new-and-changed-files.com/_site/index.html +8 -0
  72. data/features/support/test_site_dirs/new-and-changed-files.com/s3_website.yml +3 -0
  73. data/features/support/test_site_dirs/no-new-or-changed-files.com/_site/css/styles.css +3 -0
  74. data/features/support/test_site_dirs/no-new-or-changed-files.com/_site/index.html +5 -0
  75. data/features/support/test_site_dirs/no-new-or-changed-files.com/s3_website.yml +3 -0
  76. data/features/support/test_site_dirs/only-changed-files.com/_site/css/styles.css +3 -0
  77. data/features/support/test_site_dirs/only-changed-files.com/_site/index.html +9 -0
  78. data/features/support/test_site_dirs/only-changed-files.com/s3_website.yml +3 -0
  79. data/features/support/test_site_dirs/site-that-contains-s3-website-file.com/_site/s3_website.yml +3 -0
  80. data/features/support/test_site_dirs/site-that-contains-s3-website-file.com/s3_website.yml +3 -0
  81. data/features/support/test_site_dirs/site-with-text-doc.com/_site/file.txt +1 -0
  82. data/features/support/test_site_dirs/site.with.css-maxage.com/_site/css/styles.css +3 -0
  83. data/features/support/test_site_dirs/site.with.css-maxage.com/_site/index.html +5 -0
  84. data/features/support/test_site_dirs/site.with.css-maxage.com/s3_website.yml +5 -0
  85. data/features/support/test_site_dirs/site.with.gzipped-and-max-aged-content.com/_site/css/styles.css +3 -0
  86. data/features/support/test_site_dirs/site.with.gzipped-and-max-aged-content.com/_site/index.html +5 -0
  87. data/features/support/test_site_dirs/site.with.gzipped-and-max-aged-content.com/s3_website.yml +5 -0
  88. data/features/support/test_site_dirs/site.with.gzipped-html.com/_site/css/styles.css +3 -0
  89. data/features/support/test_site_dirs/site.with.gzipped-html.com/_site/index.html +5 -0
  90. data/features/support/test_site_dirs/site.with.gzipped-html.com/s3_website.yml +5 -0
  91. data/features/support/test_site_dirs/site.with.maxage.com/_site/css/styles.css +3 -0
  92. data/features/support/test_site_dirs/site.with.maxage.com/_site/index.html +5 -0
  93. data/features/support/test_site_dirs/site.with.maxage.com/s3_website.yml +4 -0
  94. data/features/support/test_site_dirs/unpublish-a-post.com/_site/css/styles.css +3 -0
  95. data/features/support/test_site_dirs/unpublish-a-post.com/s3_website.yml +3 -0
  96. data/features/support/vcr.rb +20 -0
  97. data/features/website-performance.feature +57 -0
  98. data/lib/cloudfront/invalidator.rb +37 -0
  99. data/lib/s3_website/config_loader.rb +55 -0
  100. data/lib/s3_website/diff_helper.rb +113 -0
  101. data/lib/s3_website/endpoint.rb +37 -0
  102. data/lib/s3_website/errors.rb +42 -0
  103. data/lib/s3_website/jekyll.rb +5 -0
  104. data/lib/s3_website/keyboard.rb +27 -0
  105. data/lib/s3_website/nanoc.rb +5 -0
  106. data/lib/s3_website/parallelism.rb +25 -0
  107. data/lib/s3_website/paths.rb +39 -0
  108. data/lib/s3_website/retry.rb +19 -0
  109. data/lib/s3_website/tasks.rb +36 -0
  110. data/lib/s3_website/upload.rb +137 -0
  111. data/lib/s3_website/uploader.rb +177 -0
  112. data/lib/s3_website.rb +34 -0
  113. data/project/assembly.sbt +1 -0
  114. data/project/build.properties +0 -0
  115. data/project/plugins.sbt +1 -0
  116. data/project/sbt-launch-0.13.2.jar +0 -0
  117. data/resources/configuration_file_template.yml +56 -0
  118. data/s3_website.gemspec +41 -0
  119. data/sbt +4 -0
  120. data/spec/lib/cloudfront/invalidator_spec.rb +60 -0
  121. data/spec/lib/config_loader_spec.rb +20 -0
  122. data/spec/lib/endpoint_spec.rb +31 -0
  123. data/spec/lib/error_spec.rb +21 -0
  124. data/spec/lib/keyboard_spec.rb +62 -0
  125. data/spec/lib/parallelism_spec.rb +81 -0
  126. data/spec/lib/paths_spec.rb +7 -0
  127. data/spec/lib/retry_spec.rb +34 -0
  128. data/spec/lib/upload_spec.rb +303 -0
  129. data/spec/lib/uploader_spec.rb +37 -0
  130. data/spec/sample_files/hyde_site/_site/.vimrc +5 -0
  131. data/spec/sample_files/hyde_site/_site/css/styles.css +3 -0
  132. data/spec/sample_files/hyde_site/_site/index.html +1 -0
  133. data/spec/sample_files/hyde_site/s3_website.yml +3 -0
  134. data/spec/sample_files/tokyo_site/_site/.vimrc +5 -0
  135. data/spec/sample_files/tokyo_site/_site/css/styles.css +3 -0
  136. data/spec/sample_files/tokyo_site/_site/index.html +1 -0
  137. data/spec/sample_files/tokyo_site/s3_website.yml +4 -0
  138. data/spec/spec_helper.rb +1 -0
  139. data/src/main/scala/s3/website/CloudFront.scala +96 -0
  140. data/src/main/scala/s3/website/Diff.scala +42 -0
  141. data/src/main/scala/s3/website/Implicits.scala +7 -0
  142. data/src/main/scala/s3/website/Push.scala +191 -0
  143. data/src/main/scala/s3/website/Ruby.scala +12 -0
  144. data/src/main/scala/s3/website/S3.scala +139 -0
  145. data/src/main/scala/s3/website/model/Config.scala +152 -0
  146. data/src/main/scala/s3/website/model/S3Endpoint.scala +22 -0
  147. data/src/main/scala/s3/website/model/Site.scala +68 -0
  148. data/src/main/scala/s3/website/model/errors.scala +11 -0
  149. data/src/main/scala/s3/website/model/push.scala +192 -0
  150. data/src/test/scala/s3/website/S3WebsiteSpec.scala +445 -0
  151. metadata +508 -0
data/bin/s3_website ADDED
@@ -0,0 +1,80 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require File.expand_path(File.dirname(__FILE__)+ '/../lib/s3_website')
4
+
5
+ class Cfg < Thor
6
+ desc 'create', 'Create a config file with placeholder values'
7
+ def create
8
+ S3Website::Tasks.config_create Dir.pwd
9
+ rescue Exception => e
10
+ puts S3Website.error_report e
11
+ exit 1
12
+ end
13
+
14
+ desc 'apply', 'Apply the configuration on the AWS services'
15
+ long_desc <<-LONGDESC
16
+ `s3_website cfg apply` will apply the configuration the S3 bucket.
17
+
18
+ In addition, if you CloudFront related settings, this command will apply them on the CloudFront distribution.
19
+
20
+ If the S3 bucket does not exist, this command will create it
21
+ and configure it to function as a website.
22
+ LONGDESC
23
+ def apply
24
+ puts 'Applying the configurations in s3_website.yml on the AWS services ...'
25
+ require 'configure-s3-website'
26
+ config_source = ConfigureS3Website::FileConfigSource.new 's3_website.yml'
27
+ ConfigureS3Website::Runner.run({
28
+ :config_source => config_source
29
+ })
30
+ rescue Exception => e
31
+ puts S3Website.error_report e
32
+ exit 1
33
+ end
34
+ end
35
+
36
+ class Cli < Thor
37
+ option(
38
+ :site,
39
+ :type => :string,
40
+ :default => 'infer automatically',
41
+ :desc => "The directory where your website files are. When not defined, s3_website will look for the site in #{S3Website::Paths.site_paths.join(' or ')}."
42
+ )
43
+ option(
44
+ :config_dir,
45
+ :type => :string,
46
+ :default => Dir.pwd,
47
+ :desc => "The directory where your config file is. When not defined, s3_website will look in the current working directory."
48
+ )
49
+ desc 'push', 'Push local files with the S3 website'
50
+ long_desc <<-LONGDESC
51
+ `s3_website push` will upload new and changes files to S3. It will
52
+ also delete from S3 the files that you no longer have locally.
53
+ LONGDESC
54
+ def push
55
+ call_dir = Dir.pwd
56
+ Dir.chdir(File.dirname(__FILE__)+ '/..') {
57
+ # Compile Scala if needed
58
+ jar_file = File.dirname(__FILE__) + '/../target/scala-2.11/s3_website.jar'
59
+ unless File.exists? jar_file
60
+ puts "Compiling the s3_website.jar. This happens only once. Please wait up to a few minutes..."
61
+ assembly_succeeded = system "./sbt assembly"
62
+ unless assembly_succeeded
63
+ puts "Failed to create the s3_website.jar. Please file a bug report."
64
+ exit 1
65
+ end
66
+ end
67
+
68
+ # Then run it
69
+ site_path = File.expand_path(S3Website::Paths.infer_site_path options[:site], call_dir)
70
+ config_dir = File.expand_path options[:config_dir]
71
+ args = "--site=#{site_path} --config-dir=#{config_dir}"
72
+ exit system("java -cp #{jar_file} s3.website.Push #{args}")
73
+ }
74
+ end
75
+
76
+ desc 'cfg SUBCOMMAND ...ARGS', 'Operate on the config file'
77
+ subcommand 'cfg', Cfg
78
+ end
79
+
80
+ Cli.start(ARGV)
data/build.sbt ADDED
@@ -0,0 +1,33 @@
1
+ import AssemblyKeys._
2
+
3
+ name := "s3_website"
4
+
5
+ version := "0.0.1"
6
+
7
+ scalaVersion := "2.11.0"
8
+
9
+ scalacOptions += "-feature"
10
+
11
+ scalacOptions += "-language:implicitConversions"
12
+
13
+ libraryDependencies += "org.yaml" % "snakeyaml" % "1.13"
14
+
15
+ libraryDependencies += "org.jruby" % "jruby" % "1.7.11"
16
+
17
+ libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.7.7"
18
+
19
+ libraryDependencies += "commons-codec" % "commons-codec" % "1.9"
20
+
21
+ libraryDependencies += "commons-io" % "commons-io" % "2.4"
22
+
23
+ libraryDependencies += "org.apache.tika" % "tika-core" % "1.4"
24
+
25
+ libraryDependencies += "com.lexicalscope.jewelcli" % "jewelcli" % "0.8.9"
26
+
27
+ libraryDependencies += "org.specs2" %% "specs2" % "2.3.11" % "test"
28
+
29
+ resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
30
+
31
+ jarName in assembly := "s3_website.jar"
32
+
33
+ test in assembly := {}
data/changelog.md ADDED
@@ -0,0 +1,215 @@
1
+ # Changelog
2
+
3
+ This project uses [Semantic Versioning](http://semver.org).
4
+
5
+ ## 2.0.0
6
+
7
+ This release contains backward breaking changes. Please read the section below
8
+ for more info.
9
+
10
+ ### Java is now required
11
+
12
+ * The `push` command is now written in Scala. This means that you need Java 1.6
13
+ or above to run the command `s3_website push`.
14
+
15
+ ### Removed features
16
+
17
+ * `extensionless_mime_type`
18
+
19
+ s3_website now relies on Apache Tika to infer the mime type.
20
+
21
+ * the `--headless` switch on the command-line
22
+
23
+ s3_website always deletes the files that are on the S3 bucket but not on the local file system.
24
+ Use the settings `ignore_on_server` and `exclude_from_upload` to control the retained files.
25
+
26
+ ## 1.7.4
27
+
28
+ * Fix issue https://github.com/laurilehmijoki/s3_website/issues/83
29
+
30
+ ## 1.7.3
31
+
32
+ * Fix Digest::Digest deprecation warn on Ruby 2.1.0
33
+
34
+ This warning did appear then one used the `cfg create` or `cfg apply`
35
+ commands.
36
+
37
+ ## 1.7.2
38
+
39
+ * This release contains no code changes (the indended change is in the 1.7.3
40
+ release)
41
+
42
+ ## 1.7.1
43
+
44
+ * Do not override ERB when adding CloudFront dist
45
+
46
+ ## 1.7.0
47
+
48
+ * Add zopfli compression support
49
+
50
+ ## 1.6.13
51
+
52
+ * Depend on any 1-series version of the aws-sdk gem
53
+
54
+ ## 1.6.12
55
+
56
+ * Fix bug <https://github.com/laurilehmijoki/s3_website/issues/63>
57
+
58
+ ## 1.6.11
59
+
60
+ * Loosen the dependency spec of mime-types (#70)
61
+
62
+ ## 1.6.10
63
+
64
+ * Fix bug <https://github.com/laurilehmijoki/s3_website/issues/38>
65
+
66
+ ## 1.6.9
67
+
68
+ * Fix Digest::Digest deprecation warn on Ruby 2.1.0
69
+
70
+ ## 1.6.8
71
+
72
+ * Fix content-type problem
73
+ (<https://github.com/laurilehmijoki/s3_website/pull/66>)
74
+
75
+ ## 1.6.7
76
+
77
+ * Support the eu-west-1 location constraint for the commands `cfg apply` and
78
+ `cfg create`
79
+
80
+ ## 1.6.6
81
+
82
+ * Mark all text documents as UTF-8 encoded
83
+
84
+ ## 1.6.5
85
+
86
+ * In case of error, exit with status 1
87
+
88
+ ## 1.6.4
89
+
90
+ * Add systematic error handling
91
+
92
+ Fixes issue https://github.com/laurilehmijoki/s3_website/issues/52.
93
+
94
+ ## 1.6.3
95
+
96
+ * Invalidate a deleted file on CloudFront
97
+
98
+ ## 1.6.2
99
+
100
+ * Fix issue <https://github.com/laurilehmijoki/s3_website/pull/54>
101
+
102
+ ## 1.6.1
103
+
104
+ * Fix issue <https://github.com/laurilehmijoki/s3_website/issues/30>
105
+
106
+ ## 1.6.0
107
+
108
+ * Add support for excluding files from upload
109
+ * s3_website.yml now supports `exclude_from_upload`
110
+ * Support multiple values on the `ignore_on_server` setting
111
+
112
+ ## 1.5.0
113
+
114
+ * Add support for specifying the MIME type for extensionless files
115
+
116
+ ## 1.4.5
117
+
118
+ * If max-age=0, set `Cache-Control: no-cache, max-age=0`
119
+
120
+ ## 1.4.4
121
+
122
+ * Add support for eu-west-1 as a location constraint
123
+
124
+ ## 1.4.3
125
+
126
+ * Decrease the default concurrency level to 3
127
+
128
+ See https://github.com/laurilehmijoki/s3_website/issues/8#issuecomment-24855991
129
+ for discussion.
130
+
131
+ ## 1.4.2
132
+
133
+ * Fix `s3_website cfg apply` for CloudFront setup (#33)
134
+
135
+ ## 1.4.1
136
+
137
+ * Fix diff for Windows users
138
+
139
+ See
140
+ <https://github.com/laurilehmijoki/s3_website/issues/8#issuecomment-23683568>
141
+ for discussion.
142
+
143
+ ## 1.4.0
144
+
145
+ * Add setting `cloudfront_invalidate_root`
146
+
147
+ ## 1.3.2
148
+
149
+ * Move blacklist filtering into a better place
150
+
151
+ ## 1.3.1
152
+
153
+ * Print to stdout the initial state of the diff progress indicator
154
+
155
+ ## 1.3.0
156
+
157
+ * Show a progress indicator when calculating diff
158
+
159
+ ## 1.2.1
160
+
161
+ * Use `print` instead of `puts` when printing to stdout in a concurrent context
162
+
163
+ ## 1.2.0
164
+
165
+ * Use the `--config_dir` CLI option to specify the directory from where to read
166
+ the `s3_website.yml` config file
167
+
168
+ ## 1.1.2
169
+
170
+ * Mention the MIT license in the gemspec file
171
+
172
+ ## 1.1.1
173
+
174
+ * Mention the new `concurrency_level` setting in the sample config file
175
+
176
+ ## 1.1.0
177
+
178
+ * Add possibility to define the concurrency level in *s3_website.yml*
179
+
180
+ ## 1.0.3
181
+
182
+ * Reject blacklisted files in a more appropriate place
183
+
184
+ ## 1.0.2
185
+
186
+ * Never upload the file *s3_website.yml*
187
+
188
+ ## 1.0.1
189
+
190
+ * Set default concurrency level to 100. Related to issue [#6](https://github.com/laurilehmijoki/s3_website/issues/6).
191
+
192
+ ## 1.0.0
193
+
194
+ * Make 0.4.0 the version 1.0.0
195
+
196
+ ## 0.4.0
197
+
198
+ * Include the available configs in the sample s3_website.yml file
199
+
200
+ ## 0.3.0
201
+
202
+ * Add Nanoc support
203
+
204
+ ## 0.2.1
205
+
206
+ * Remove Gemfile.lock
207
+ * Rename gemspec file
208
+
209
+ ## 0.2.0
210
+
211
+ * Add command `s3_website cfg apply`
212
+
213
+ ## 0.1.0
214
+
215
+ * First version
@@ -0,0 +1,29 @@
1
+ Feature: Using s3_website as a library
2
+
3
+ As a developer
4
+ I want to programmatically use the s3_website API
5
+ So that I can extend my software with the capabilities of s3_website
6
+
7
+ @one-file-to-delete
8
+ Scenario: Developer wants feedback on how many files s3_website deleted
9
+ When my S3 website is in "features/support/test_site_dirs/unpublish-a-post.com"
10
+ And I call the push command
11
+ Then s3_website should report that it deleted 1 file from S3
12
+
13
+ @new-and-changed-files
14
+ Scenario: Developer wants feedback on how many files s3_website uploaded
15
+ When my S3 website is in "features/support/test_site_dirs/new-and-changed-files.com"
16
+ And I call the push command
17
+ Then s3_website should report that it uploaded 1 new and 1 changed files into S3
18
+
19
+ @s3-and-cloudfront-when-updating-a-file
20
+ Scenario: Developer wants feedback on how many Cloudfront items s3_website invalidated
21
+ When my S3 website is in "features/support/test_site_dirs/cdn-powered.with-one-change.blog.fi"
22
+ And I call the push command
23
+ Then s3_website should report that it invalidated 2 Cloudfront item
24
+
25
+ @create-redirect
26
+ Scenario: Developer wants feedback on how many redirects s3_website created
27
+ When my S3 website is in "features/support/test_site_dirs/create-redirects"
28
+ And I call the push command
29
+ Then s3_website should report that it created 2 new redirects