fluent-plugin-redshift-v2 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 4a7f29254c4d31c42ec336bab72191989fcfe1d1
-  data.tar.gz: c021ef2cd64c797078e5c18f8010e820889e2285
+  metadata.gz: 928f3d93c7527fbba91158fe285469d0edd2188b
+  data.tar.gz: 2ef0cb1fd50773157281d9a98bb622e6a8d1a584
 SHA512:
-  metadata.gz: 700278954259afef087001238a16aecbfe41b8ab6e8ead4313d49231c1b7d441212b7ce06a81487f8e1afec10ca2a97e3e9e174ae82d97c524539f21dddb6e19
-  data.tar.gz: 7b3413c7f5fbaf66befce0dadf68d750680d5d830733ed4bf742c627849f396eb421026540364c1963b604f475a8582719d2beba3a2830f78c2b7100428fb07b
+  metadata.gz: ea6e6a79008672482e58570a13c5eac3d538a92eceb4ee1ab80a9cceeb2fa2608ec8112a9014672d6b1bf5e78b66be84de3b814fd7b95a1952af7773da022892
+  data.tar.gz: abf3cd43a5b89c6eb9be4bd24b8985d16dcf0ff263460b2ce61ab2b43ae883712f04f282532452c643b62023121712a7f9868d1b67677acb800f48f7649a3138

data/README.md

@@ -1,36 +1,140 @@
-# Fluent::Plugin::Redshift::V2
+Amazon Redshift output plugin for Fluentd
+========
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/fluent/plugin/redshift/v2`. To experiment with that code, run `bin/console` for an interactive prompt.
+## Overview
+
+Amazon Redshift output plugin uploads event logs to an Amazon Redshift Cluster. Supported data formats are csv, tsv and json. An S3 bucket and a Redshift Cluster are required to use this plugin. Forked from fluent-plugin-redshift, and re-implemented with aws-sdk-v2 and fluentd 0.14.
 
-TODO: Delete this and the text above, and describe your gem
 
 ## Installation
 
-Add this line to your application's Gemfile:
+    fluent-gem install fluent-plugin-redshift-v2
 
-```ruby
-gem 'fluent-plugin-redshift-v2'
-```
+## Configuration
+
+Format:
+
+    <match my.tag>
+        type redshift_v2
+
+        # s3 (for copying data to redshift)
+        aws_key_id YOUR_AWS_KEY_ID
+        aws_sec_key YOUR_AWS_SECRET_KEY
+        ## or Use IAM Role instead of credentials.
+        aws_iam_role arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME
+
+        s3_bucket YOUR_S3_BUCKET
+        s3_endpoint YOUR_S3_BUCKET_END_POINT
+        path YOUR_S3_PATH
+        timestamp_key_format year=%Y/month=%m/day=%d/hour=%H/%Y%m%d-%H%M
+        s3_server_side_encryption S3_SERVER_SIDE_ENCRYPTION
+
+        # redshift
+        redshift_host YOUR_AMAZON_REDSHIFT_CLUSTER_END_POINT
+        redshift_port YOUR_AMAZON_REDSHIFT_CLUSTER_PORT
+        redshift_dbname YOUR_AMAZON_REDSHIFT_CLUSTER_DATABASE_NAME
+        redshift_user YOUR_AMAZON_REDSHIFT_CLUSTER_USER_NAME
+        redshift_password YOUR_AMAZON_REDSHIFT_CLUSTER_PASSWORD
+        redshift_schemaname YOUR_AMAZON_REDSHIFT_CLUSTER_TARGET_SCHEMA_NAME
+        redshift_tablename YOUR_AMAZON_REDSHIFT_CLUSTER_TARGET_TABLE_NAME
+        redshift_copy_columns COLMUNS_FOR_COPY
+        file_type [tsv|csv|json|msgpack]
+
+        # buffer
+        buffer_type file
+        buffer_path /var/log/fluent/redshift
+        flush_interval 15m
+		retry_wait 30s
+		retry_limit 5
+        buffer_chunk_limit 1g
+		buffer_queue_limit 100
+		flush_at_shutdown true
+		num_threads 4
+    </match>
+
++ `type` (required) : The value must be `redshift_v2`.
+
++ `aws_key_id` : AWS access key id to access s3 bucket.
+
++ `aws_sec_key` : AWS secret key id to access s3 bucket.
+
++ `aws_iam_role` : AWS IAM Role name to access s3 bucket and copy into redshift.
+
++ `s3_bucket` (required) : s3 bucket name. S3 bucket must be same as the region of your Redshift cluster.
+
++ `s3_endpoint` : s3 endpoint.
+
++ `path` (required) : s3 path to input.
+
++ `timestamp_key_format` : The format of the object keys. It can include date-format directives.
+
+  - Default parameter is "year=%Y/month=%m/day=%d/hour=%H/%Y%m%d-%H%M"
+  - For example, the s3 path is as following with the above example configration.
+    <pre>
+  logs/example/year=2013/month=03/day=05/hour=12/20130305_1215_00.gz
+  logs/example/year=2013/month=03/day=05/hour=12/20130305_1230_00.gz
+</pre>
 
-And then execute:
++ `s3_server_side_encryption` : S3 Server-Side Encryption (Only aes256 is supported)
 
-    $ bundle
++ `redshift_host` (required) : the end point(or hostname) of your Amazon Redshift cluster.
 
-Or install it yourself as:
++ `redshift_port` (required) : port number.
 
-    $ gem install fluent-plugin-redshift-v2
++ `redshift_dbname` (required) : database name.
 
-## Usage
++ `redshift_user` (required) : user name.
 
-TODO: Write usage instructions here
++ `redshift_password` (required) : password for the user name.
 
-## Development
++ `redshift_tablename` (required) : table name to store data.
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
++ `redshift_schemaname` : schema name to store data. By default, this option is not set and find table without schema as your own search_path.
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
++ `redshift_connect_timeout` : maximum time to wait for connection to succeed.
 
-## Contributing
++ `redshift_copy_columns` : columns for copying. Value needs to be comma-separated like `id,name,age`
+
++ `file_type` : file format of the source data.  `csv`, `tsv`, `msgpack` or `json` are available.
+
++ `delimiter` : delimiter of the source data. This option will be ignored if `file_type` is specified.
+
++ `buffer_type` : buffer type.
+
++ `buffer_path` : path prefix of the files to buffer logs.
+
++ `flush_interval` : flush interval.
+
++ `buffer_chunk_limit` : limit buffer size to chunk.
+
++ `buffer_queue_limit` : limit buffer files to chunk.
+
++ `num_threads` : number of threads to load data to redshift.
+
++ `utc` : utc time zone. This parameter affects `timestamp_key_format`.
+
+and standard buffered output options. (see https://docs.fluentd.org/v0.14/articles/buffer-plugin-overview)
+
+## Logging examples
+```ruby
+# examples by fluent-logger
+require 'fluent-logger'
+log = Fluent::Logger::FluentLogger.new(nil, host: 'localhost', port: 24224)
+
+# file_type: csv
+log.post('your.tag', log: "12345,12345")
+
+# file_type: tsv
+log.post('your.tag', log: "12345\t12345")
+
+# file_type: json
+require 'json'
+log.post('your.tag', { user_id: 12345, data_id: 12345 }.to_json)
+
+# file_type: msgpack
+log.post('your.tag', user_id: 12345, data_id: 12345)
+```
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/fluent-plugin-redshift-v2.
+## License
 
+[Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)

data/fluent-plugin-redshift-v2.gemspec

@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
   spec.name          = "fluent-plugin-redshift-v2"
-  spec.version       = "0.1.2"
+  spec.version       = "0.1.3"
   spec.authors       = ["Jun Yokoyama"]
   spec.email         = ["jun@larus.org"]
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-redshift-v2
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.3
 platform: ruby
 authors:
 - Jun Yokoyama
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-11-02 00:00:00.000000000 Z
+date: 2017-11-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler