logstash-output-swift 0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: a53de853bb7f7f97356dab1053131195d5f83e56
+  data.tar.gz: 98ed226c784788a9bb4510b4208f1128b14ee315
+SHA512:
+  metadata.gz: 38d9a9e6e243e297436fa605d7e40eee0a5841f31a67617238803da04bde70f695519ec96a509fede8c331284782f992b6f6e4ce4a84d21371063a79736fa5ad
+  data.tar.gz: 56f10e27ce4efd6eaa2806673030cecd8bb7e39422012bbbfc656d19329f2e15fe6d56f2b28aaf81a7b13a7746d9bf90390ecb8db53bc763fe13f9b6d760a19b

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1
+- First release, based on lgostash-output-s3 4.0.13

data/CONTRIBUTORS

@@ -0,0 +1,22 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Chad Bean (chadbean)
+* Colin Surprenant (colinsurprenant)
+* Helmut Duregger (hduregger)
+* Jordan Sissel (jordansissel)
+* Joshua Spence (joshuaspence)
+* Kurt Hurtado (kurtado)
+* Mattia Peterle (MattiaBeast)
+* Nick Ethier (nickethier)
+* Pier-Hugues Pellerin (ph)
+* Richard Pijnenburg (electrical)
+* Jake Landis (jakelandis)
+* Armin Braun (original-brownbear)
+* Karri Niemelä (kakoni)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,15 @@
+[Missing the other part of the readme]
+
+## Running the tests
+
+```
+bundle install
+bundle exec rspec
+```
+
+If you want to run the integration test against a real bucket you need to pass
+your aws credentials to the test runner or declare it in your environment.
+
+```
+AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID=123 AWS_SECRET_ACCESS_KEY=secret AWS_LOGSTASH_TEST_BUCKET=mytest bundle exec rspec spec/integration/s3_spec.rb --tag integration
+```

data/Gemfile

@@ -0,0 +1,11 @@
+source 'https://rubygems.org'
+
+gemspec
+
+logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
+use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
+
+if Dir.exist?(logstash_path) && use_logstash_source
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end

data/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,2 @@
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,38 @@
+# logstash-output-swift
+
+This plugin is provided as an external plugin and is not part of the Logstash project.
+Its forked from [logstash-out-s3](https://github.com/logstash-plugins/logstash-output-s3) project.
+
+This plugin allows you to output to Openstack swift storage.
+
+## Changelog
+See CHANGELOG.md
+
+## Versions
+Released versions are available via rubygems.
+
+## Installation
+  - Run `bin/logstash-plugin install logstash-output-swift` in your logstash installation directory
+  - Configure
+
+## Configuration options
+
+| Option           | Type             | Description                                 | Required? | Default |
+| ------           | ----             | -----------                                 | --------- | ------- |
+| username         | String           | Openstack Username                          | Yes       |         |
+| api_key          | String           | Openstack apikey                            | Yes       |         |
+| auth_url         | String           | Keystone server                             | Yes       |         |
+| project_name     | String           | Project name                                | Yes       |         |
+| domain_name      | String           | Domain name                                 | Yes       |         |
+| container        | String           | Swift container for the logs                | Yes       |         |
+
+## Development and Running tests
+  - `bundle exec rspec`
+
+## Releasing
+  - Update Changelog
+  - Bump version in gemspec
+  - Commit
+  - Create tag `git tag v<version-number-in-gemspec>`
+  - `gem build logstash-output-swift.gemspec`
+  - `gem push`

data/lib/logstash/outputs/swift.rb

@@ -0,0 +1,360 @@
+# encoding: utf-8
+require "logstash/outputs/base"
+require "logstash/namespace"
+require "stud/temporary"
+require "stud/task"
+require "concurrent"
+require "socket"
+require "thread"
+require "tmpdir"
+require "fileutils"
+require "set"
+require "pathname"
+require "fog/openstack"
+
+# INFORMATION:
+#
+# This plugin batches and uploads logstash events into Openstack Swift.
+#
+# Requirements:
+# * Amazon S3 Bucket and S3 Access Permissions (Typically access_key_id and secret_access_key)
+# * S3 PutObject permission
+#
+# Swift outputs create temporary files into the OS' temporary directory, you can specify where to save them using the `temporary_directory` option.
+#
+# Swift output files have the following format
+#
+# ls.s3.312bc026-2f5d-49bc-ae9f-5940cf4ad9a6.2013-04-18T10.00.tag_hello.part0.txt
+#
+#
+# |=======
+# | ls.s3 | indicate logstash plugin s3 |
+# | 312bc026-2f5d-49bc-ae9f-5940cf4ad9a6 | a new, random uuid per file. |
+# | 2013-04-18T10.00 | represents the time whenever you specify time_file. |
+# | tag_hello | this indicates the event's tag. |
+# | part0 | this means if you indicate size_file then it will generate more parts if you file.size > size_file. When a file is full it will be pushed to the bucket and then deleted from the temporary directory. If a file is empty, it is simply deleted.  Empty files will not be pushed |
+# |=======
+#
+# Crash Recovery:
+# * This plugin will recover and upload temporary log files after crash/abnormal termination when using `restore` set to true
+#
+##[Note regarding time_file and size_file] :
+#
+## Both time_file and size_file settings can trigger a log "file rotation"
+## A log rotation pushes the current log "part" to s3 and deleted from local temporary storage.
+#
+## If you specify BOTH size_file and time_file then it will create file for each tag (if specified).
+## When EITHER time_file minutes have elapsed OR log file size > size_file, a log rotation is triggered.
+##
+## If you ONLY specify time_file but NOT file_size, one file for each tag (if specified) will be created.
+## When time_file minutes elapses, a log rotation will be triggered.
+#
+## If you ONLY specify size_file, but NOT time_file, one files for each tag (if specified) will be created.
+## When size of log file part > size_file, a log rotation will be triggered.
+#
+## If NEITHER size_file nor time_file is specified, ONLY one file for each tag (if specified) will be created.
+## WARNING: Since no log rotation is triggered, S3 Upload will only occur when logstash restarts.
+#
+#
+# #### Usage:
+# This is an example of logstash config:
+# [source,ruby]
+# output {
+#    swift {
+#      username => "crazy_name"                 (required)
+#      api_key => "crazy_key"                   (required)
+#      auth_url => "https://auth_tokens"        (required)
+#      project_name => "crazy project name"     (required)
+#      domain_name => "crazy domain name"       (required)
+#      container => "your_bucket_container"     (required)
+#    }
+#
+class LogStash::Outputs::Swift < LogStash::Outputs::Base
+  require "logstash/outputs/swift/writable_directory_validator"
+  require "logstash/outputs/swift/path_validator"
+  require "logstash/outputs/swift/write_container_permission_validator"
+  require "logstash/outputs/swift/size_rotation_policy"
+  require "logstash/outputs/swift/time_rotation_policy"
+  require "logstash/outputs/swift/size_and_time_rotation_policy"
+  require "logstash/outputs/swift/temporary_file"
+  require "logstash/outputs/swift/temporary_file_factory"
+  require "logstash/outputs/swift/uploader"
+  require "logstash/outputs/swift/file_repository"
+
+  #include LogStash::PluginMixins::AwsConfig::V2
+
+  PREFIX_KEY_NORMALIZE_CHARACTER = "_"
+  PERIODIC_CHECK_INTERVAL_IN_SECONDS = 15
+  CRASH_RECOVERY_THREADPOOL = Concurrent::ThreadPoolExecutor.new({
+                                                                   :min_threads => 1,
+                                                                   :max_threads => 2,
+                                                                   :fallback_policy => :caller_runs
+                                                                 })
+
+
+  config_name "swift"
+  default :codec, "line"
+
+  concurrency :shared
+
+  # Swift container
+  config :container, :validate => :string, :required => true
+
+  # Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file.
+  # If you have tags then it will generate a specific size file for every tags
+  ##NOTE: define size of file is the better thing, because generate a local temporary file on disk and then put it in bucket.
+  config :size_file, :validate => :number, :default => 1024 * 1024 * 5
+
+  # Set the time, in MINUTES, to close the current sub_time_section of bucket.
+  # If you define file_size you have a number of files in consideration of the section and the current tag.
+  # 0 stay all time on listerner, beware if you specific 0 and size_file 0, because you will not put the file on bucket,
+  # for now the only thing this plugin can do is to put the file when logstash restart.
+  config :time_file, :validate => :number, :default => 15
+
+  ## IMPORTANT: if you use multiple instance of s3, you should specify on one of them the "restore=> true" and on the others "restore => false".
+  ## This is hack for not destroy the new files after restoring the initial files.
+  ## If you do not specify "restore => true" when logstash crashes or is restarted, the files are not sent into the bucket,
+  ## for example if you have single Instance.
+  config :restore, :validate => :boolean, :default => true
+
+  # The S3 canned ACL to use when putting the file. Defaults to "private".
+  config :canned_acl, :validate => ["private", "public-read", "public-read-write", "authenticated-read"],
+         :default => "private"
+
+  # Set the directory where logstash will store the tmp files before sending it to S3
+  # default to the current OS temporary directory in linux /tmp/logstash
+  config :temporary_directory, :validate => :string, :default => File.join(Dir.tmpdir, "logstash")
+
+  # Specify a prefix to the uploaded filename, this can simulate directories on S3.  Prefix does not require leading slash.
+  # This option support string interpolation, be warned this can created a lot of temporary local files.
+  config :prefix, :validate => :string, :default => ''
+
+  # Specify how many workers to use to upload the files to S3
+  config :upload_workers_count, :validate => :number, :default => (Concurrent.processor_count * 0.5).ceil
+
+  # Number of items we can keep in the local queue before uploading them
+  config :upload_queue_size, :validate => :number, :default => 2 * (Concurrent.processor_count * 0.25).ceil
+
+  # Define tags to be appended to the file on the S3 bucket.
+  #
+  # Example:
+  # tags => ["elasticsearch", "logstash", "kibana"]
+  #
+  # Will generate this file:
+  # "ls.s3.logstash.local.2015-01-01T00.00.tag_elasticsearch.logstash.kibana.part0.txt"
+  #
+  config :tags, :validate => :array, :default => []
+
+  # Specify the content encoding. Supports ("gzip"). Defaults to "none"
+  config :encoding, :validate => ["none", "gzip"], :default => "none"
+
+  # Define the strategy to use to decide when we need to rotate the file and push it to S3,
+  # The default strategy is to check for both size and time, the first one to match will rotate the file.
+  config :rotation_strategy, :validate => ["size_and_time", "size", "time"], :default => "size_and_time"
+
+  # The common use case is to define permission on the root bucket and give Logstash full access to write its logs.
+  # In some circonstances you need finer grained permission on subfolder, this allow you to disable the check at startup.
+  config :validate_credentials_on_root_bucket, :validate => :boolean, :default => true
+
+  #Openstack username
+  config :username, :validate => :string
+
+  #Openstack api_key
+  config :api_key, :validate => :string
+
+  #Openstack auth url
+  config :auth_url, :validate => :string
+
+  #Openstack project_name
+  config :project_name, :validate => :string
+
+  #Openstack domain_name
+  config :domain_name, :validate => :string
+
+  def register
+    # I've move the validation of the items into custom classes
+    # to prepare for the new config validation that will be part of the core so the core can
+    # be moved easily.
+    unless @prefix.empty?
+      if !PathValidator.valid?(prefix)
+        raise LogStash::ConfigurationError, "Prefix must not contains: #{PathValidator::INVALID_CHARACTERS}"
+      end
+    end
+
+    if !WritableDirectoryValidator.valid?(@temporary_directory)
+      raise LogStash::ConfigurationError, "Logstash must have the permissions to write to the temporary directory: #{@temporary_directory}"
+    end
+
+    if @validate_credentials_on_root_bucket && !WriteContainerPermissionValidator.new(@logger).valid?(container_resource)
+      raise LogStash::ConfigurationError, "Logstash must have the privileges to write to root container `#{@container}`, check your credentials or your permissions."
+    end
+
+    if @time_file.nil? && @size_file.nil? || @size_file == 0 && @time_file == 0
+      raise LogStash::ConfigurationError, "The S3 plugin must have at least one of time_file or size_file set to a value greater than 0"
+    end
+
+    @file_repository = FileRepository.new(@tags, @encoding, @temporary_directory)
+
+    @rotation = rotation_strategy
+
+    executor = Concurrent::ThreadPoolExecutor.new({ :min_threads => 1,
+                                                    :max_threads => @upload_workers_count,
+                                                    :max_queue => @upload_queue_size,
+                                                    :fallback_policy => :caller_runs })
+
+    @uploader = Uploader.new(container_resource, @logger, executor)
+
+    # Restoring from crash will use a new threadpool to slowly recover
+    # New events should have more priority.
+    restore_from_crash if @restore
+
+    # If we need time based rotation we need to do periodic check on the file
+    # to take care of file that were not updated recently
+    start_periodic_check if @rotation.needs_periodic?
+  end
+
+  def multi_receive_encoded(events_and_encoded)
+    puts 'milti rrecei'
+    prefix_written_to = Set.new
+
+    events_and_encoded.each do |event, encoded|
+      prefix_key = normalize_key(event.sprintf(@prefix))
+      prefix_written_to << prefix_key
+
+      begin
+        puts 'begin'
+        @file_repository.get_file(prefix_key) { |file| file.write(encoded) }
+        # The output should stop accepting new events coming in, since it cannot do anything with them anymore.
+        # Log the error and rethrow it.
+      rescue Errno::ENOSPC => e
+        @logger.error("S3: No space left in temporary directory", :temporary_directory => @temporary_directory)
+        raise e
+      end
+    end
+
+    puts 'done'
+    # Groups IO calls to optimize fstat checks
+    rotate_if_needed(prefix_written_to)
+  end
+
+  def close
+    stop_periodic_check if @rotation.needs_periodic?
+
+    @logger.debug("Uploading current workspace")
+
+    # The plugin has stopped receiving new events, but we still have
+    # data on disk, lets make sure it get to S3.
+    # If Logstash get interrupted, the `restore_from_crash` (when set to true) method will pickup
+    # the content in the temporary directly and upload it.
+    # This will block the shutdown until all upload are done or the use force quit.
+    @file_repository.each_files do |file|
+      upload_file(file)
+    end
+
+    @file_repository.shutdown
+
+    @uploader.stop # wait until all the current upload are complete
+    @crash_uploader.stop if @restore # we might have still work to do for recovery so wait until we are done
+  end
+
+  def normalize_key(prefix_key)
+    prefix_key.gsub(PathValidator.matches_re, PREFIX_KEY_NORMALIZE_CHARACTER)
+  end
+
+  private
+  # We start a task in the background for check for stale files and make sure we rotate them to S3 if needed.
+  def start_periodic_check
+    @logger.debug("Start periodic rotation check")
+
+    @periodic_check = Concurrent::TimerTask.new(:execution_interval => PERIODIC_CHECK_INTERVAL_IN_SECONDS) do
+      @logger.debug("Periodic check for stale files")
+
+      rotate_if_needed(@file_repository.keys)
+    end
+
+    @periodic_check.execute
+  end
+
+  def stop_periodic_check
+    @periodic_check.shutdown
+  end
+
+  def container_resource
+    Excon.defaults[:ciphers] = 'DEFAULT'
+    service = Fog::Storage::OpenStack.new({
+      openstack_username: @username,
+      openstack_api_key: @api_key,
+      openstack_auth_url: @auth_url,
+      openstack_project_name: @project_name,
+      openstack_domain_name: @domain_name,
+      connection_options: {}
+    })
+
+    service.directories.get(@container)
+  end
+
+  def rotate_if_needed(prefixes)
+    puts 'rotate!'
+    prefixes.each do |prefix|
+      # Each file access is thread safe,
+      # until the rotation is done then only
+      # one thread has access to the resource.
+      @file_repository.get_factory(prefix) do |factory|
+        temp_file = factory.current
+
+        if @rotation.rotate?(temp_file)
+          @logger.debug("Rotate file",
+                        :strategy => @rotation.class.name,
+                        :key => temp_file.key,
+                        :path => temp_file.path)
+
+          upload_file(temp_file)
+          factory.rotate!
+        end
+      end
+    end
+  end
+
+  def upload_file(temp_file)
+    puts 'upload file'
+    @logger.debug("Queue for upload", :path => temp_file.path)
+
+    # if the queue is full the calling thread will be used to upload
+    temp_file.close # make sure the content is on disk
+    if temp_file.size > 0
+      @uploader.upload_async(temp_file,
+                             :on_complete => method(:clean_temporary_file))
+    end
+  end
+
+  def rotation_strategy
+    case @rotation_strategy
+    when "size"
+      SizeRotationPolicy.new(size_file)
+    when "time"
+      TimeRotationPolicy.new(time_file)
+    when "size_and_time"
+      SizeAndTimeRotationPolicy.new(size_file, time_file)
+    end
+  end
+
+  def clean_temporary_file(file)
+    @logger.debug("Removing temporary file", :file => file.path)
+    file.delete!
+  end
+
+  # The upload process will use a separate uploader/threadpool with less resource allocated to it.
+  # but it will use an unbounded queue for the work, it may take some time before all the older files get processed.
+  def restore_from_crash
+    @crash_uploader = Uploader.new(container_resource, @logger, CRASH_RECOVERY_THREADPOOL)
+
+    temp_folder_path = Pathname.new(@temporary_directory)
+    Dir.glob(::File.join(@temporary_directory, "**/*"))
+      .select { |file| ::File.file?(file) }
+      .each do |file|
+      temp_file = TemporaryFile.create_from_existing_file(file, temp_folder_path)
+      @logger.debug("Recovering from crash and uploading", :file => temp_file.path)
+      @crash_uploader.upload_async(temp_file, :on_complete => method(:clean_temporary_file))
+    end
+  end
+end