s3_website_monadic 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (151) hide show
  1. checksums.yaml +7 -0
  2. data/.gitignore +10 -0
  3. data/.travis.yml +3 -0
  4. data/Gemfile +3 -0
  5. data/LICENSE +42 -0
  6. data/README.md +451 -0
  7. data/Rakefile +24 -0
  8. data/additional-docs/example-configurations.md +62 -0
  9. data/additional-docs/setting-up-aws-credentials.md +51 -0
  10. data/assembly.sbt +3 -0
  11. data/bin/s3_website +80 -0
  12. data/build.sbt +33 -0
  13. data/changelog.md +215 -0
  14. data/features/as-library.feature +29 -0
  15. data/features/cassettes/cucumber_tags/create-redirect.yml +384 -0
  16. data/features/cassettes/cucumber_tags/empty-bucket.yml +89 -0
  17. data/features/cassettes/cucumber_tags/new-and-changed-files.yml +303 -0
  18. data/features/cassettes/cucumber_tags/new-files-for-sydney.yml +211 -0
  19. data/features/cassettes/cucumber_tags/new-files.yml +355 -0
  20. data/features/cassettes/cucumber_tags/no-new-or-changed-files.yml +359 -0
  21. data/features/cassettes/cucumber_tags/one-file-to-delete.yml +390 -0
  22. data/features/cassettes/cucumber_tags/only-changed-files.yml +411 -0
  23. data/features/cassettes/cucumber_tags/s3-and-cloudfront-after-deleting-a-file.yml +434 -0
  24. data/features/cassettes/cucumber_tags/s3-and-cloudfront-when-updating-a-file.yml +435 -0
  25. data/features/cassettes/cucumber_tags/s3-and-cloudfront.yml +290 -0
  26. data/features/cloudfront.feature +54 -0
  27. data/features/command-line-help.feature +54 -0
  28. data/features/delete.feature +19 -0
  29. data/features/error_reporting.feature +24 -0
  30. data/features/instructions-for-new-user.feature +154 -0
  31. data/features/jekyll-support.feature +20 -0
  32. data/features/nanoc-support.feature +20 -0
  33. data/features/push.feature +115 -0
  34. data/features/redirects.feature +14 -0
  35. data/features/security.feature +15 -0
  36. data/features/step_definitions/steps.rb +86 -0
  37. data/features/support/env.rb +26 -0
  38. data/features/support/test_site_dirs/cdn-powered.blog.fi/_site/css/styles.css +3 -0
  39. data/features/support/test_site_dirs/cdn-powered.blog.fi/_site/index.html +5 -0
  40. data/features/support/test_site_dirs/cdn-powered.blog.fi/s3_website.yml +4 -0
  41. data/features/support/test_site_dirs/cdn-powered.when-deleted-a-file.blog.fi/_site/index.html +10 -0
  42. data/features/support/test_site_dirs/cdn-powered.when-deleted-a-file.blog.fi/s3_website.yml +5 -0
  43. data/features/support/test_site_dirs/cdn-powered.with-one-change.blog.fi/_site/css/styles.css +3 -0
  44. data/features/support/test_site_dirs/cdn-powered.with-one-change.blog.fi/_site/index.html +10 -0
  45. data/features/support/test_site_dirs/cdn-powered.with-one-change.blog.fi/s3_website.yml +4 -0
  46. data/features/support/test_site_dirs/create-redirects/_site/.gitkeep +0 -0
  47. data/features/support/test_site_dirs/create-redirects/s3_website.yml +6 -0
  48. data/features/support/test_site_dirs/ignored-files.com/_site/css/styles.css +4 -0
  49. data/features/support/test_site_dirs/ignored-files.com/_site/index.html +8 -0
  50. data/features/support/test_site_dirs/ignored-files.com/s3_website.yml +5 -0
  51. data/features/support/test_site_dirs/index-and-assets.blog.fi/_site/assets/picture.gif +0 -0
  52. data/features/support/test_site_dirs/index-and-assets.blog.fi/_site/css/styles.css +3 -0
  53. data/features/support/test_site_dirs/index-and-assets.blog.fi/_site/index.html +5 -0
  54. data/features/support/test_site_dirs/index-and-assets.blog.fi/s3_website.yml +3 -0
  55. data/features/support/test_site_dirs/jekyllrb.com/_site/css/styles.css +3 -0
  56. data/features/support/test_site_dirs/jekyllrb.com/_site/index.html +5 -0
  57. data/features/support/test_site_dirs/jekyllrb.com/s3_website.yml +3 -0
  58. data/features/support/test_site_dirs/my.blog-with-clean-urls.com/_site/css/styles.css +3 -0
  59. data/features/support/test_site_dirs/my.blog-with-clean-urls.com/_site/index +5 -0
  60. data/features/support/test_site_dirs/my.blog-with-clean-urls.com/s3_website.yml +3 -0
  61. data/features/support/test_site_dirs/my.blog.com/_site/css/styles.css +3 -0
  62. data/features/support/test_site_dirs/my.blog.com/_site/index.html +5 -0
  63. data/features/support/test_site_dirs/my.blog.com/s3_website.yml +3 -0
  64. data/features/support/test_site_dirs/my.sydney.blog.au/_site/css/styles.css +3 -0
  65. data/features/support/test_site_dirs/my.sydney.blog.au/_site/index.html +5 -0
  66. data/features/support/test_site_dirs/my.sydney.blog.au/s3_website.yml +4 -0
  67. data/features/support/test_site_dirs/nanoc.ws/public/output/css/styles.css +3 -0
  68. data/features/support/test_site_dirs/nanoc.ws/public/output/index.html +5 -0
  69. data/features/support/test_site_dirs/nanoc.ws/s3_website.yml +3 -0
  70. data/features/support/test_site_dirs/new-and-changed-files.com/_site/css/styles.css +4 -0
  71. data/features/support/test_site_dirs/new-and-changed-files.com/_site/index.html +8 -0
  72. data/features/support/test_site_dirs/new-and-changed-files.com/s3_website.yml +3 -0
  73. data/features/support/test_site_dirs/no-new-or-changed-files.com/_site/css/styles.css +3 -0
  74. data/features/support/test_site_dirs/no-new-or-changed-files.com/_site/index.html +5 -0
  75. data/features/support/test_site_dirs/no-new-or-changed-files.com/s3_website.yml +3 -0
  76. data/features/support/test_site_dirs/only-changed-files.com/_site/css/styles.css +3 -0
  77. data/features/support/test_site_dirs/only-changed-files.com/_site/index.html +9 -0
  78. data/features/support/test_site_dirs/only-changed-files.com/s3_website.yml +3 -0
  79. data/features/support/test_site_dirs/site-that-contains-s3-website-file.com/_site/s3_website.yml +3 -0
  80. data/features/support/test_site_dirs/site-that-contains-s3-website-file.com/s3_website.yml +3 -0
  81. data/features/support/test_site_dirs/site-with-text-doc.com/_site/file.txt +1 -0
  82. data/features/support/test_site_dirs/site.with.css-maxage.com/_site/css/styles.css +3 -0
  83. data/features/support/test_site_dirs/site.with.css-maxage.com/_site/index.html +5 -0
  84. data/features/support/test_site_dirs/site.with.css-maxage.com/s3_website.yml +5 -0
  85. data/features/support/test_site_dirs/site.with.gzipped-and-max-aged-content.com/_site/css/styles.css +3 -0
  86. data/features/support/test_site_dirs/site.with.gzipped-and-max-aged-content.com/_site/index.html +5 -0
  87. data/features/support/test_site_dirs/site.with.gzipped-and-max-aged-content.com/s3_website.yml +5 -0
  88. data/features/support/test_site_dirs/site.with.gzipped-html.com/_site/css/styles.css +3 -0
  89. data/features/support/test_site_dirs/site.with.gzipped-html.com/_site/index.html +5 -0
  90. data/features/support/test_site_dirs/site.with.gzipped-html.com/s3_website.yml +5 -0
  91. data/features/support/test_site_dirs/site.with.maxage.com/_site/css/styles.css +3 -0
  92. data/features/support/test_site_dirs/site.with.maxage.com/_site/index.html +5 -0
  93. data/features/support/test_site_dirs/site.with.maxage.com/s3_website.yml +4 -0
  94. data/features/support/test_site_dirs/unpublish-a-post.com/_site/css/styles.css +3 -0
  95. data/features/support/test_site_dirs/unpublish-a-post.com/s3_website.yml +3 -0
  96. data/features/support/vcr.rb +20 -0
  97. data/features/website-performance.feature +57 -0
  98. data/lib/cloudfront/invalidator.rb +37 -0
  99. data/lib/s3_website/config_loader.rb +55 -0
  100. data/lib/s3_website/diff_helper.rb +113 -0
  101. data/lib/s3_website/endpoint.rb +37 -0
  102. data/lib/s3_website/errors.rb +42 -0
  103. data/lib/s3_website/jekyll.rb +5 -0
  104. data/lib/s3_website/keyboard.rb +27 -0
  105. data/lib/s3_website/nanoc.rb +5 -0
  106. data/lib/s3_website/parallelism.rb +25 -0
  107. data/lib/s3_website/paths.rb +39 -0
  108. data/lib/s3_website/retry.rb +19 -0
  109. data/lib/s3_website/tasks.rb +36 -0
  110. data/lib/s3_website/upload.rb +137 -0
  111. data/lib/s3_website/uploader.rb +177 -0
  112. data/lib/s3_website.rb +34 -0
  113. data/project/assembly.sbt +1 -0
  114. data/project/build.properties +0 -0
  115. data/project/plugins.sbt +1 -0
  116. data/project/sbt-launch-0.13.2.jar +0 -0
  117. data/resources/configuration_file_template.yml +56 -0
  118. data/s3_website.gemspec +41 -0
  119. data/sbt +4 -0
  120. data/spec/lib/cloudfront/invalidator_spec.rb +60 -0
  121. data/spec/lib/config_loader_spec.rb +20 -0
  122. data/spec/lib/endpoint_spec.rb +31 -0
  123. data/spec/lib/error_spec.rb +21 -0
  124. data/spec/lib/keyboard_spec.rb +62 -0
  125. data/spec/lib/parallelism_spec.rb +81 -0
  126. data/spec/lib/paths_spec.rb +7 -0
  127. data/spec/lib/retry_spec.rb +34 -0
  128. data/spec/lib/upload_spec.rb +303 -0
  129. data/spec/lib/uploader_spec.rb +37 -0
  130. data/spec/sample_files/hyde_site/_site/.vimrc +5 -0
  131. data/spec/sample_files/hyde_site/_site/css/styles.css +3 -0
  132. data/spec/sample_files/hyde_site/_site/index.html +1 -0
  133. data/spec/sample_files/hyde_site/s3_website.yml +3 -0
  134. data/spec/sample_files/tokyo_site/_site/.vimrc +5 -0
  135. data/spec/sample_files/tokyo_site/_site/css/styles.css +3 -0
  136. data/spec/sample_files/tokyo_site/_site/index.html +1 -0
  137. data/spec/sample_files/tokyo_site/s3_website.yml +4 -0
  138. data/spec/spec_helper.rb +1 -0
  139. data/src/main/scala/s3/website/CloudFront.scala +96 -0
  140. data/src/main/scala/s3/website/Diff.scala +42 -0
  141. data/src/main/scala/s3/website/Implicits.scala +7 -0
  142. data/src/main/scala/s3/website/Push.scala +191 -0
  143. data/src/main/scala/s3/website/Ruby.scala +12 -0
  144. data/src/main/scala/s3/website/S3.scala +139 -0
  145. data/src/main/scala/s3/website/model/Config.scala +152 -0
  146. data/src/main/scala/s3/website/model/S3Endpoint.scala +22 -0
  147. data/src/main/scala/s3/website/model/Site.scala +68 -0
  148. data/src/main/scala/s3/website/model/errors.scala +11 -0
  149. data/src/main/scala/s3/website/model/push.scala +192 -0
  150. data/src/test/scala/s3/website/S3WebsiteSpec.scala +445 -0
  151. metadata +508 -0
data/bin/s3_website ADDED
@@ -0,0 +1,80 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require File.expand_path(File.dirname(__FILE__)+ '/../lib/s3_website')
4
+
5
+ class Cfg < Thor
6
+ desc 'create', 'Create a config file with placeholder values'
7
+ def create
8
+ S3Website::Tasks.config_create Dir.pwd
9
+ rescue Exception => e
10
+ puts S3Website.error_report e
11
+ exit 1
12
+ end
13
+
14
+ desc 'apply', 'Apply the configuration on the AWS services'
15
+ long_desc <<-LONGDESC
16
+ `s3_website cfg apply` will apply the configuration the S3 bucket.
17
+
18
+ In addition, if you CloudFront related settings, this command will apply them on the CloudFront distribution.
19
+
20
+ If the S3 bucket does not exist, this command will create it
21
+ and configure it to function as a website.
22
+ LONGDESC
23
+ def apply
24
+ puts 'Applying the configurations in s3_website.yml on the AWS services ...'
25
+ require 'configure-s3-website'
26
+ config_source = ConfigureS3Website::FileConfigSource.new 's3_website.yml'
27
+ ConfigureS3Website::Runner.run({
28
+ :config_source => config_source
29
+ })
30
+ rescue Exception => e
31
+ puts S3Website.error_report e
32
+ exit 1
33
+ end
34
+ end
35
+
36
+ class Cli < Thor
37
+ option(
38
+ :site,
39
+ :type => :string,
40
+ :default => 'infer automatically',
41
+ :desc => "The directory where your website files are. When not defined, s3_website will look for the site in #{S3Website::Paths.site_paths.join(' or ')}."
42
+ )
43
+ option(
44
+ :config_dir,
45
+ :type => :string,
46
+ :default => Dir.pwd,
47
+ :desc => "The directory where your config file is. When not defined, s3_website will look in the current working directory."
48
+ )
49
+ desc 'push', 'Push local files with the S3 website'
50
+ long_desc <<-LONGDESC
51
+ `s3_website push` will upload new and changes files to S3. It will
52
+ also delete from S3 the files that you no longer have locally.
53
+ LONGDESC
54
+ def push
55
+ call_dir = Dir.pwd
56
+ Dir.chdir(File.dirname(__FILE__)+ '/..') {
57
+ # Compile Scala if needed
58
+ jar_file = File.dirname(__FILE__) + '/../target/scala-2.11/s3_website.jar'
59
+ unless File.exists? jar_file
60
+ puts "Compiling the s3_website.jar. This happens only once. Please wait up to a few minutes..."
61
+ assembly_succeeded = system "./sbt assembly"
62
+ unless assembly_succeeded
63
+ puts "Failed to create the s3_website.jar. Please file a bug report."
64
+ exit 1
65
+ end
66
+ end
67
+
68
+ # Then run it
69
+ site_path = File.expand_path(S3Website::Paths.infer_site_path options[:site], call_dir)
70
+ config_dir = File.expand_path options[:config_dir]
71
+ args = "--site=#{site_path} --config-dir=#{config_dir}"
72
+ exit system("java -cp #{jar_file} s3.website.Push #{args}")
73
+ }
74
+ end
75
+
76
+ desc 'cfg SUBCOMMAND ...ARGS', 'Operate on the config file'
77
+ subcommand 'cfg', Cfg
78
+ end
79
+
80
+ Cli.start(ARGV)
data/build.sbt ADDED
@@ -0,0 +1,33 @@
1
+ import AssemblyKeys._
2
+
3
+ name := "s3_website"
4
+
5
+ version := "0.0.1"
6
+
7
+ scalaVersion := "2.11.0"
8
+
9
+ scalacOptions += "-feature"
10
+
11
+ scalacOptions += "-language:implicitConversions"
12
+
13
+ libraryDependencies += "org.yaml" % "snakeyaml" % "1.13"
14
+
15
+ libraryDependencies += "org.jruby" % "jruby" % "1.7.11"
16
+
17
+ libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.7.7"
18
+
19
+ libraryDependencies += "commons-codec" % "commons-codec" % "1.9"
20
+
21
+ libraryDependencies += "commons-io" % "commons-io" % "2.4"
22
+
23
+ libraryDependencies += "org.apache.tika" % "tika-core" % "1.4"
24
+
25
+ libraryDependencies += "com.lexicalscope.jewelcli" % "jewelcli" % "0.8.9"
26
+
27
+ libraryDependencies += "org.specs2" %% "specs2" % "2.3.11" % "test"
28
+
29
+ resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
30
+
31
+ jarName in assembly := "s3_website.jar"
32
+
33
+ test in assembly := {}
data/changelog.md ADDED
@@ -0,0 +1,215 @@
1
+ # Changelog
2
+
3
+ This project uses [Semantic Versioning](http://semver.org).
4
+
5
+ ## 2.0.0
6
+
7
+ This release contains backward breaking changes. Please read the section below
8
+ for more info.
9
+
10
+ ### Java is now required
11
+
12
+ * The `push` command is now written in Scala. This means that you need Java 1.6
13
+ or above to run the command `s3_website push`.
14
+
15
+ ### Removed features
16
+
17
+ * `extensionless_mime_type`
18
+
19
+ s3_website now relies on Apache Tika to infer the mime type.
20
+
21
+ * the `--headless` switch on the command-line
22
+
23
+ s3_website always deletes the files that are on the S3 bucket but not on the local file system.
24
+ Use the settings `ignore_on_server` and `exclude_from_upload` to control the retained files.
25
+
26
+ ## 1.7.4
27
+
28
+ * Fix issue https://github.com/laurilehmijoki/s3_website/issues/83
29
+
30
+ ## 1.7.3
31
+
32
+ * Fix Digest::Digest deprecation warn on Ruby 2.1.0
33
+
34
+ This warning did appear then one used the `cfg create` or `cfg apply`
35
+ commands.
36
+
37
+ ## 1.7.2
38
+
39
+ * This release contains no code changes (the indended change is in the 1.7.3
40
+ release)
41
+
42
+ ## 1.7.1
43
+
44
+ * Do not override ERB when adding CloudFront dist
45
+
46
+ ## 1.7.0
47
+
48
+ * Add zopfli compression support
49
+
50
+ ## 1.6.13
51
+
52
+ * Depend on any 1-series version of the aws-sdk gem
53
+
54
+ ## 1.6.12
55
+
56
+ * Fix bug <https://github.com/laurilehmijoki/s3_website/issues/63>
57
+
58
+ ## 1.6.11
59
+
60
+ * Loosen the dependency spec of mime-types (#70)
61
+
62
+ ## 1.6.10
63
+
64
+ * Fix bug <https://github.com/laurilehmijoki/s3_website/issues/38>
65
+
66
+ ## 1.6.9
67
+
68
+ * Fix Digest::Digest deprecation warn on Ruby 2.1.0
69
+
70
+ ## 1.6.8
71
+
72
+ * Fix content-type problem
73
+ (<https://github.com/laurilehmijoki/s3_website/pull/66>)
74
+
75
+ ## 1.6.7
76
+
77
+ * Support the eu-west-1 location constraint for the commands `cfg apply` and
78
+ `cfg create`
79
+
80
+ ## 1.6.6
81
+
82
+ * Mark all text documents as UTF-8 encoded
83
+
84
+ ## 1.6.5
85
+
86
+ * In case of error, exit with status 1
87
+
88
+ ## 1.6.4
89
+
90
+ * Add systematic error handling
91
+
92
+ Fixes issue https://github.com/laurilehmijoki/s3_website/issues/52.
93
+
94
+ ## 1.6.3
95
+
96
+ * Invalidate a deleted file on CloudFront
97
+
98
+ ## 1.6.2
99
+
100
+ * Fix issue <https://github.com/laurilehmijoki/s3_website/pull/54>
101
+
102
+ ## 1.6.1
103
+
104
+ * Fix issue <https://github.com/laurilehmijoki/s3_website/issues/30>
105
+
106
+ ## 1.6.0
107
+
108
+ * Add support for excluding files from upload
109
+ * s3_website.yml now supports `exclude_from_upload`
110
+ * Support multiple values on the `ignore_on_server` setting
111
+
112
+ ## 1.5.0
113
+
114
+ * Add support for specifying the MIME type for extensionless files
115
+
116
+ ## 1.4.5
117
+
118
+ * If max-age=0, set `Cache-Control: no-cache, max-age=0`
119
+
120
+ ## 1.4.4
121
+
122
+ * Add support for eu-west-1 as a location constraint
123
+
124
+ ## 1.4.3
125
+
126
+ * Decrease the default concurrency level to 3
127
+
128
+ See https://github.com/laurilehmijoki/s3_website/issues/8#issuecomment-24855991
129
+ for discussion.
130
+
131
+ ## 1.4.2
132
+
133
+ * Fix `s3_website cfg apply` for CloudFront setup (#33)
134
+
135
+ ## 1.4.1
136
+
137
+ * Fix diff for Windows users
138
+
139
+ See
140
+ <https://github.com/laurilehmijoki/s3_website/issues/8#issuecomment-23683568>
141
+ for discussion.
142
+
143
+ ## 1.4.0
144
+
145
+ * Add setting `cloudfront_invalidate_root`
146
+
147
+ ## 1.3.2
148
+
149
+ * Move blacklist filtering into a better place
150
+
151
+ ## 1.3.1
152
+
153
+ * Print to stdout the initial state of the diff progress indicator
154
+
155
+ ## 1.3.0
156
+
157
+ * Show a progress indicator when calculating diff
158
+
159
+ ## 1.2.1
160
+
161
+ * Use `print` instead of `puts` when printing to stdout in a concurrent context
162
+
163
+ ## 1.2.0
164
+
165
+ * Use the `--config_dir` CLI option to specify the directory from where to read
166
+ the `s3_website.yml` config file
167
+
168
+ ## 1.1.2
169
+
170
+ * Mention the MIT license in the gemspec file
171
+
172
+ ## 1.1.1
173
+
174
+ * Mention the new `concurrency_level` setting in the sample config file
175
+
176
+ ## 1.1.0
177
+
178
+ * Add possibility to define the concurrency level in *s3_website.yml*
179
+
180
+ ## 1.0.3
181
+
182
+ * Reject blacklisted files in a more appropriate place
183
+
184
+ ## 1.0.2
185
+
186
+ * Never upload the file *s3_website.yml*
187
+
188
+ ## 1.0.1
189
+
190
+ * Set default concurrency level to 100. Related to issue [#6](https://github.com/laurilehmijoki/s3_website/issues/6).
191
+
192
+ ## 1.0.0
193
+
194
+ * Make 0.4.0 the version 1.0.0
195
+
196
+ ## 0.4.0
197
+
198
+ * Include the available configs in the sample s3_website.yml file
199
+
200
+ ## 0.3.0
201
+
202
+ * Add Nanoc support
203
+
204
+ ## 0.2.1
205
+
206
+ * Remove Gemfile.lock
207
+ * Rename gemspec file
208
+
209
+ ## 0.2.0
210
+
211
+ * Add command `s3_website cfg apply`
212
+
213
+ ## 0.1.0
214
+
215
+ * First version
@@ -0,0 +1,29 @@
1
+ Feature: Using s3_website as a library
2
+
3
+ As a developer
4
+ I want to programmatically use the s3_website API
5
+ So that I can extend my software with the capabilities of s3_website
6
+
7
+ @one-file-to-delete
8
+ Scenario: Developer wants feedback on how many files s3_website deleted
9
+ When my S3 website is in "features/support/test_site_dirs/unpublish-a-post.com"
10
+ And I call the push command
11
+ Then s3_website should report that it deleted 1 file from S3
12
+
13
+ @new-and-changed-files
14
+ Scenario: Developer wants feedback on how many files s3_website uploaded
15
+ When my S3 website is in "features/support/test_site_dirs/new-and-changed-files.com"
16
+ And I call the push command
17
+ Then s3_website should report that it uploaded 1 new and 1 changed files into S3
18
+
19
+ @s3-and-cloudfront-when-updating-a-file
20
+ Scenario: Developer wants feedback on how many Cloudfront items s3_website invalidated
21
+ When my S3 website is in "features/support/test_site_dirs/cdn-powered.with-one-change.blog.fi"
22
+ And I call the push command
23
+ Then s3_website should report that it invalidated 2 Cloudfront item
24
+
25
+ @create-redirect
26
+ Scenario: Developer wants feedback on how many redirects s3_website created
27
+ When my S3 website is in "features/support/test_site_dirs/create-redirects"
28
+ And I call the push command
29
+ Then s3_website should report that it created 2 new redirects