fluent-plugin-gcs 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ac0dec9f5eadfcf591d969f57d8c0299e8f9060e
+  data.tar.gz: 6ca58b3c7a54526ac95d2711407491a7f8ccfed5
+SHA512:
+  metadata.gz: f8e0681f28757ee42a811f623afdd987bcd59006574356cab3b64001f57c45ae6e64fdfb2cf6cd94eab258f554a886c79d50c52e18f8d38cc70d7f73a0980a57
+  data.tar.gz: cf27dad559db3e0306137cb395faf4268c96223d4da6dd2b06970ff2e060ecdc284a7cfaf7b4a5e158c855db6d59dbd903c337ad7ee9ec87ed7dcacb3cf85c61

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.travis.yml

@@ -0,0 +1,11 @@
+language: ruby
+
+rvm:
+  - 2.1
+  - 2.2
+  - 2.3.0
+
+before_install:
+  - gem update bundler
+
+script: bundle exec rake test

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fluent-plugin-gcs.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,13 @@
+Copyright (c) 2016 - Daichi HIRATA
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/README.md

@@ -0,0 +1,197 @@
+# fluent-plugin-gcs
+[![Build Status](https://travis-ci.org/daichirata/fluent-plugin-gcs.svg?branch=master)](https://travis-ci.org/daichirata/fluent-plugin-gcs) [![Code Climate](https://codeclimate.com/github/daichirata/fluent-plugin-gcs/badges/gpa.svg)](https://codeclimate.com/github/daichirata/fluent-plugin-gcs)
+
+Google Cloud Storage output plugin for [Fluentd](https://github.com/fluent/fluentd).
+
+## Installation
+
+``` shell
+gem install fluent-plugin-gcs
+```
+
+## Examples
+
+```
+<match pattern>
+  @type gcs
+
+  project YOUR_PROJECT
+  keyfile YOUR_KEYFILE_PATH
+  gcs_bucket YOUR_GCS_BUCKET_NAME
+  gcs_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
+  path logs/
+  buffer_path /var/log/fluent/gcs
+
+  time_slice_format %Y%m%d-%H
+  time_slice_wait 10m
+  utc
+</match>
+```
+
+## Configuration
+
+### Authentication
+
+You can provide the project and credential information to connect to the Storage
+service, or if you are running on Google Compute Engine this configuration is taken care of for you.
+
+**project**
+
+Project identifier for GCS. Project are discovered in the following order:
+* Specify project in `project`
+* Discover project in environment variables `STORAGE_PROJECT`, `GOOGLE_CLOUD_PROJECT`, `GCLOUD_PROJECT`
+* Discover GCE credentials
+
+**keyfile**
+
+Path of GCS service account credentials JSON file. Credentials are discovered in the following order:
+* Specify credentials path in `keyfile`
+* Discover credentials path in environment variables `GOOGLE_CLOUD_KEYFILE`, `GCLOUD_KEYFILE`
+* Discover credentials JSON in environment variables `GOOGLE_CLOUD_KEYFILE_JSON`, `GCLOUD_KEYFILE_JSON`
+* Discover credentials file in the Cloud SDK's path
+* Discover GCE credentials
+
+**client_retries**
+
+Number of times to retry requests on server error.
+
+**client_timeout**
+
+Default timeout to use in requests.
+
+**bucket (*required)**
+
+GCS bucket name.
+
+**store_as**
+
+Archive format on GCS. You can use serveral format:
+
+* gzip (default)
+* json
+* text
+
+**path**
+
+path prefix of the files on GCS. Default is "" (no prefix).
+
+**object_key_format**
+
+The format of GCS object keys. You can use several built-in variables:
+
+* %{path}
+* %{time_slice}
+* %{index}
+* %{file_extension}
+* %{uuid_flush}
+* %{hex_random}
+
+to decide keys dynamically.
+
+* `%{path}` is exactly the value of `path` configured in the configuration file. E.g., "logs/" in the example configuration above.
+* `%{time_slice}` is the time-slice in text that are formatted with `time_slice_format`.
+* `%{index}` is the sequential number starts from 0, increments when multiple files are uploaded to GCS in the same time slice.
+* `%{file_extention}` is changed by the value of `store_as`.
+  * gzip - gz
+  * json - json
+  * text - txt
+* `%{uuid_flush}` a uuid that is replaced everytime the buffer will be flushed
+* `%{hex_random}` a random hex string that is replaced for each buffer chunk, not assured to be unique.
+You can configure the length of string with a `hex_random_length` parameter (Default: 4).
+
+The default format is `%{path}%{time_slice}_%{index}.%{file_extension}`.
+
+**hex_random_length**
+
+The length of `%{hex_random}` placeholder.
+
+**transcoding**
+
+Enable the decompressive form of transcoding.
+
+See also [Transcoding of gzip-compressed files](https://cloud.google.com/storage/docs/transcoding).
+
+**format**
+
+Change one line format in the GCS object. You can use serveral format:
+
+* out_file (default)
+* json
+* ltsv
+* single_value
+
+See also [official Formatter article](http://docs.fluentd.org/articles/formatter-plugin-overview).
+
+**auto_create_bucket**
+
+Create GCS bucket if it does not exists. Default is true.
+
+TODO: rate limit
+
+**acl**
+
+Permission for the object in GCS. Acceptable values are:
+
+* `auth_read`       - File owner gets OWNER access, and allAuthenticatedUsers get READER access.
+* `owner_full`      - File owner gets OWNER access, and project team owners get OWNER access.
+* `owner_read`      - File owner gets OWNER access, and project team owners get READER access.
+* `private`         - File owner gets OWNER access.
+* `project_private` - File owner gets OWNER access, and project team members get access according to their roles.
+* `public_read`     - File owner gets OWNER access, and allUsers get READER access.
+
+Default is nil (bucket default object ACL). See also [official document](https://cloud.google.com/storage/docs/access-control/lists).
+
+**encryption_key**, **encryption_key_sha256**
+
+You can also choose to provide your own AES-256 key for server-side encryption. See also [Customer-supplied encryption keys](https://cloud.google.com/storage/docs/encryption#customer-supplied).
+
+**overwrite**
+
+Overwrite already existing path. Default is false, which raises an error
+if a GCS object of the same path already exists, or increment the
+`%{index}` placeholder until finding an absent path.
+
+**buffer_path (*required)**
+
+path prefix of the files to buffer logs.
+
+**time_slice_format**
+
+Format of the time used as the file name. Default is '%Y%m%d'. Use
+'%Y%m%d%H' to split files hourly.
+
+**time_slice_wait**
+
+The time to wait old logs. Default is 10 minutes. Specify larger value if
+old logs may reache.
+
+**localtime**
+
+Use Local time instead of UTC.
+
+**utc**
+
+Use UTC instead of local time.
+
+
+And see [official Time Sliced Output article](http://docs.fluentd.org/articles/output-plugin-overview#time-sliced-output-parameters)
+
+### ObjectMetadata
+
+User provided web-safe keys and arbitrary string values that will returned with requests for the file as "x-goog-meta-" response headers.
+
+```
+<match *>
+  @type gcs
+
+  <object_metadata>
+    key KEY_DATA_1
+    value VALUE_DATA_1
+  </object_metadata>
+
+  <object_metadata>
+    key KEY_DATA_2
+    value VALUE_DATA_2
+  </object_metadata>
+</match>
+```

data/Rakefile

@@ -0,0 +1,11 @@
+require "bundler"
+Bundler::GemHelper.install_tasks
+
+require "rake/testtask"
+Rake::TestTask.new(:test) do |test|
+  test.libs << "lib" << "test"
+  test.test_files = FileList["test/plugin/test_*.rb"]
+  test.verbose = true
+end
+
+task :default => [:test]

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "fluent/plugin/gcs"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/fluent-plugin-gcs.gemspec

@@ -0,0 +1,32 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'fluent/plugin/gcs/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "fluent-plugin-gcs"
+  spec.version       = Fluent::GCSPlugin::VERSION
+  spec.authors       = ["Daichi HIRATA"]
+  spec.email         = ["hirata.daichi@gmail.com"]
+  spec.summary       = "Google Cloud Storage output plugin for Fluentd"
+  spec.description   = "Google Cloud Storage output plugin for Fluentd"
+  spec.homepage      = "https://github.com/daishirata/fluent-plugin-gcs"
+  spec.license       = "Apache-2.0"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_runtime_dependency "fluentd", "~> 0.12.0"
+  spec.add_runtime_dependency "google-cloud-storage", "~> 0.21"
+
+  spec.add_development_dependency "bundler", "~> 1.13"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rr", "= 1.1.2"
+  spec.add_development_dependency "test-unit", ">= 3.0.8"
+  spec.add_development_dependency "test-unit-rr", ">= 1.0.3"
+  spec.add_development_dependency "timecop"
+end

data/lib/fluent/plugin/gcs/object_creator.rb

@@ -0,0 +1,92 @@
+require "tempfile"
+require "zlib"
+
+module Fluent
+  module GCS
+    def self.discovered_object_creator(store_as, transcoding: nil)
+      case store_as
+      when :gzip
+        Fluent::GCS::GZipObjectCreator.new(transcoding)
+      when :json
+        Fluent::GCS::JSONObjectCreator.new
+      when :text
+        Fluent::GCS::TextObjectCreator.new
+      end
+    end
+
+    class ObjectCreator
+      def content_type
+        raise NotImplementedError
+      end
+
+      def content_encoding
+        nil
+      end
+
+      def file_extension
+        raise NotImplementedError
+      end
+
+      def write(chunk, io)
+        raise NotImplementedError
+      end
+
+      def create(chunk, &block)
+        Tempfile.create("fluent-plugin-gcs") do |f|
+          f.binmode
+          f.sync = true
+          write(chunk, f)
+          block.call(f)
+        end
+      end
+    end
+
+    class GZipObjectCreator < ObjectCreator
+      def initialize(transcoding)
+        @transcoding = transcoding
+      end
+
+      def content_type
+        @transcoding ? "text/plain" : "application/gzip"
+      end
+
+      def content_encoding
+        @transcoding ? "gzip" : nil
+      end
+
+      def file_extension
+        "gz"
+      end
+
+      def write(chunk, io)
+        writer = Zlib::GzipWriter.new(io)
+        chunk.write_to(writer)
+        writer.finish
+      end
+    end
+
+    class TextObjectCreator < ObjectCreator
+      def content_type
+        "text/plain"
+      end
+
+      def file_extension
+        "txt"
+      end
+
+      def write(chunk, io)
+        chunk.write_to(io)
+      end
+    end
+
+    class JSONObjectCreator < TextObjectCreator
+      def content_type
+        "application/json"
+      end
+
+      def file_extension
+        "json"
+      end
+    end
+  end
+end

data/lib/fluent/plugin/gcs/version.rb

@@ -0,0 +1,5 @@
+module Fluent
+  module GCSPlugin
+    VERSION = "0.1.0"
+  end
+end

data/lib/fluent/plugin/out_gcs.rb

@@ -0,0 +1,161 @@
+require "securerandom"
+require "digest"
+
+require "fluent/plugin/gcs/object_creator"
+require "fluent/plugin/gcs/version"
+
+module Fluent
+  class GCSOutput < TimeSlicedOutput
+    Fluent::Plugin.register_output("gcs", self)
+
+    def initialize
+      super
+      require "google/cloud/storage"
+    end
+
+    config_param :project, :string,  default: nil,
+                 desc: "Project identifier for GCS"
+    config_param :keyfile, :string, default: nil,
+                 desc: "Path of GCS service account credentials JSON file"
+    config_param :client_retries, :integer, default: nil,
+                 desc: "Number of times to retry requests on server error"
+    config_param :client_timeout, :integer, default: nil,
+                 desc: "Default timeout to use in requests"
+    config_param :bucket, :string,
+                 desc: "Name of a GCS bucket"
+    config_param :object_key_format, :string, default: "%{path}%{time_slice}_%{index}.%{file_extension}",
+                 desc: "Format of GCS object keys"
+    config_param :path, :string, default: "",
+                 desc: "Path prefix of the files on GCS"
+    config_param :store_as, :enum, list: [:gzip, :json, :text], default: :gzip,
+                 desc: "Archive format on GCS"
+    config_param :transcoding, :bool, default: false,
+                 desc: "Enable the decompressive form of transcoding"
+    config_param :auto_create_bucket, :bool, default: true,
+                 desc: "Create GCS bucket if it does not exists"
+    config_param :hex_random_length, :integer, default: 4,
+                 desc: "Max length of `%{hex_random}` placeholder(4-16)"
+    config_param :overwrite, :bool, default: false,
+                 desc: "Overwrite already existing path"
+    config_param :format, :string, default: "out_file",
+                 desc: "Change one line format in the GCS object"
+    config_param :acl, :enum, list: [:auth_read, :owner_full, :owner_read, :private, :project_private, :public_read], default: nil,
+                 desc: "Permission for the object in GCS"
+    config_param :encryption_key, :string, default: nil, secret: true,
+                 desc: "Customer-supplied, AES-256 encryption key"
+    config_param :encryption_key_sha256, :string, default: nil, secret: true,
+                 desc: "SHA256 hash of the customer-supplied, AES-256 encryption key"
+    config_section :object_metadata, required: false do
+      config_param :key, :string, default: ""
+      config_param :value, :string, default: ""
+    end
+    # TODO: gem "google-cloud-storage" does not support object lavel storage_class yet.
+    # config_param :storage_class, :string, default: "regional"
+
+    MAX_HEX_RANDOM_LENGTH = 32
+
+    def configure(conf)
+      super
+
+      if @encryption_key && @encryption_key_sha256.nil?
+        raise Fluent::ConfigError, "encryption_key_sha256 parameter must be provided if `encryption_key` is provided."
+      end
+
+      if @hex_random_length > MAX_HEX_RANDOM_LENGTH
+        raise Fluent::ConfigError, "hex_random_length parameter should be set to #{MAX_HEX_RANDOM_LENGTH} characters or less."
+      end
+
+      # The customer-supplied, AES-256 encryption key and hash used to encrypt the file.
+      @encryption_opts = {
+        encryption_key: @encryption_key,
+        encryption_key_sha256: @encryption_key_sha256
+      }
+
+      if @object_metadata
+        @object_metadata_hash = @object_metadata.map {|m| [m.key, m.value] }.to_h
+      end
+
+      @formatter = Fluent::Plugin.new_formatter(@format)
+      @formatter.configure(conf)
+
+      @object_creator = Fluent::GCS.discovered_object_creator(@store_as, transcoding: @transcoding)
+    end
+
+    def start
+      @gcs = Google::Cloud::Storage.new(
+        project: @project,
+        keyfile: @keyfile,
+        retries: @client_retries,
+        timeout: @client_timeout
+      )
+      @gcs_bucket = @gcs.bucket(@bucket)
+
+      ensure_bucket
+      super
+    end
+
+    def format(tag, time, record)
+      @formatter.format(tag, time, record)
+    end
+
+    def write(chunk)
+      path = generate_path(chunk)
+
+      @object_creator.create(chunk) do |obj|
+        opts = {
+          metadata: @object_metadata_hash,
+          acl: @acl,
+          content_type: @object_creator.content_type,
+          content_encoding: @object_creator.content_encoding,
+        }
+        opts.merge!(@encryption_opts)
+
+        log.debug { "out_gcs: upload chunk:#{chunk.key} to gcs://#{@bucket}/#{path} options: #{opts}" }
+        @gcs_bucket.upload_file(obj.path, path, opts)
+      end
+    end
+
+    private
+
+    def ensure_bucket
+      return unless @gcs_bucket.nil?
+
+      if !@auto_create_bucket
+        raise "bucket `#{@bucket}` does not exist"
+      end
+      log.info "creating bucket `#{@bucket}`"
+      @gcs_bucket = @gcs.create_bucket(@bucket)
+    end
+
+    def hex_random(chunk)
+      Digest::MD5.hexdigest(chunk.unique_id)[0...@hex_random_length]
+    end
+
+    def format_path(chunk)
+      now = Time.strptime(chunk.key, @time_slice_format)
+      (@localtime ? now : now.utc).strftime(@path)
+    end
+
+    def generate_path(chunk, i = 0, prev = nil)
+      tags = {
+        "%{file_extension}" => @object_creator.file_extension,
+        "%{hex_random}" => hex_random(chunk),
+        "%{index}" => i,
+        "%{path}" => format_path(chunk),
+        "%{time_slice}" => chunk.key,
+        "%{uuid_flush}" => SecureRandom.uuid,
+      }
+      path = @object_key_format.gsub(Regexp.union(tags.keys), tags)
+      return path unless @gcs_bucket.find_file(path, @encryption_opts)
+
+      if path == prev
+        if @overwrite
+          log.warn "object `#{path}` already exists but overwrites it"
+          return path
+        end
+        raise "object `#{path}` already exists"
+      end
+      generate_path(chunk, i + 1, path)
+    end
+  end
+end

metadata

@@ -0,0 +1,168 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-gcs
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Daichi HIRATA
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2016-11-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.12.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.12.0
+- !ruby/object:Gem::Dependency
+  name: google-cloud-storage
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.21'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.21'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.13'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.13'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rr
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.1.2
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.1.2
+- !ruby/object:Gem::Dependency
+  name: test-unit
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 3.0.8
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 3.0.8
+- !ruby/object:Gem::Dependency
+  name: test-unit-rr
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.3
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.0.3
+- !ruby/object:Gem::Dependency
+  name: timecop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Google Cloud Storage output plugin for Fluentd
+email:
+- hirata.daichi@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".travis.yml"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- fluent-plugin-gcs.gemspec
+- lib/fluent/plugin/gcs/object_creator.rb
+- lib/fluent/plugin/gcs/version.rb
+- lib/fluent/plugin/out_gcs.rb
+homepage: https://github.com/daishirata/fluent-plugin-gcs
+licenses:
+- Apache-2.0
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: Google Cloud Storage output plugin for Fluentd
+test_files: []