fluent-plugin-redshift-v2 0.1.2 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +122 -18
  3. data/fluent-plugin-redshift-v2.gemspec +1 -1
  4. metadata +2 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 4a7f29254c4d31c42ec336bab72191989fcfe1d1
4
- data.tar.gz: c021ef2cd64c797078e5c18f8010e820889e2285
3
+ metadata.gz: 928f3d93c7527fbba91158fe285469d0edd2188b
4
+ data.tar.gz: 2ef0cb1fd50773157281d9a98bb622e6a8d1a584
5
5
  SHA512:
6
- metadata.gz: 700278954259afef087001238a16aecbfe41b8ab6e8ead4313d49231c1b7d441212b7ce06a81487f8e1afec10ca2a97e3e9e174ae82d97c524539f21dddb6e19
7
- data.tar.gz: 7b3413c7f5fbaf66befce0dadf68d750680d5d830733ed4bf742c627849f396eb421026540364c1963b604f475a8582719d2beba3a2830f78c2b7100428fb07b
6
+ metadata.gz: ea6e6a79008672482e58570a13c5eac3d538a92eceb4ee1ab80a9cceeb2fa2608ec8112a9014672d6b1bf5e78b66be84de3b814fd7b95a1952af7773da022892
7
+ data.tar.gz: abf3cd43a5b89c6eb9be4bd24b8985d16dcf0ff263460b2ce61ab2b43ae883712f04f282532452c643b62023121712a7f9868d1b67677acb800f48f7649a3138
data/README.md CHANGED
@@ -1,36 +1,140 @@
1
- # Fluent::Plugin::Redshift::V2
1
+ Amazon Redshift output plugin for Fluentd
2
+ ========
2
3
 
3
- Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/fluent/plugin/redshift/v2`. To experiment with that code, run `bin/console` for an interactive prompt.
4
+ ## Overview
5
+
6
+ Amazon Redshift output plugin uploads event logs to an Amazon Redshift Cluster. Supported data formats are csv, tsv and json. An S3 bucket and a Redshift Cluster are required to use this plugin. Forked from fluent-plugin-redshift, and re-implemented with aws-sdk-v2 and fluentd 0.14.
4
7
 
5
- TODO: Delete this and the text above, and describe your gem
6
8
 
7
9
  ## Installation
8
10
 
9
- Add this line to your application's Gemfile:
11
+ fluent-gem install fluent-plugin-redshift-v2
10
12
 
11
- ```ruby
12
- gem 'fluent-plugin-redshift-v2'
13
- ```
13
+ ## Configuration
14
+
15
+ Format:
16
+
17
+ <match my.tag>
18
+ type redshift_v2
19
+
20
+ # s3 (for copying data to redshift)
21
+ aws_key_id YOUR_AWS_KEY_ID
22
+ aws_sec_key YOUR_AWS_SECRET_KEY
23
+ ## or Use IAM Role instead of credentials.
24
+ aws_iam_role arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME
25
+
26
+ s3_bucket YOUR_S3_BUCKET
27
+ s3_endpoint YOUR_S3_BUCKET_END_POINT
28
+ path YOUR_S3_PATH
29
+ timestamp_key_format year=%Y/month=%m/day=%d/hour=%H/%Y%m%d-%H%M
30
+ s3_server_side_encryption S3_SERVER_SIDE_ENCRYPTION
31
+
32
+ # redshift
33
+ redshift_host YOUR_AMAZON_REDSHIFT_CLUSTER_END_POINT
34
+ redshift_port YOUR_AMAZON_REDSHIFT_CLUSTER_PORT
35
+ redshift_dbname YOUR_AMAZON_REDSHIFT_CLUSTER_DATABASE_NAME
36
+ redshift_user YOUR_AMAZON_REDSHIFT_CLUSTER_USER_NAME
37
+ redshift_password YOUR_AMAZON_REDSHIFT_CLUSTER_PASSWORD
38
+ redshift_schemaname YOUR_AMAZON_REDSHIFT_CLUSTER_TARGET_SCHEMA_NAME
39
+ redshift_tablename YOUR_AMAZON_REDSHIFT_CLUSTER_TARGET_TABLE_NAME
40
+ redshift_copy_columns COLMUNS_FOR_COPY
41
+ file_type [tsv|csv|json|msgpack]
42
+
43
+ # buffer
44
+ buffer_type file
45
+ buffer_path /var/log/fluent/redshift
46
+ flush_interval 15m
47
+ retry_wait 30s
48
+ retry_limit 5
49
+ buffer_chunk_limit 1g
50
+ buffer_queue_limit 100
51
+ flush_at_shutdown true
52
+ num_threads 4
53
+ </match>
54
+
55
+ + `type` (required) : The value must be `redshift_v2`.
56
+
57
+ + `aws_key_id` : AWS access key id to access s3 bucket.
58
+
59
+ + `aws_sec_key` : AWS secret key id to access s3 bucket.
60
+
61
+ + `aws_iam_role` : AWS IAM Role name to access s3 bucket and copy into redshift.
62
+
63
+ + `s3_bucket` (required) : s3 bucket name. S3 bucket must be same as the region of your Redshift cluster.
64
+
65
+ + `s3_endpoint` : s3 endpoint.
66
+
67
+ + `path` (required) : s3 path to input.
68
+
69
+ + `timestamp_key_format` : The format of the object keys. It can include date-format directives.
70
+
71
+ - Default parameter is "year=%Y/month=%m/day=%d/hour=%H/%Y%m%d-%H%M"
72
+ - For example, the s3 path is as following with the above example configration.
73
+ <pre>
74
+ logs/example/year=2013/month=03/day=05/hour=12/20130305_1215_00.gz
75
+ logs/example/year=2013/month=03/day=05/hour=12/20130305_1230_00.gz
76
+ </pre>
14
77
 
15
- And then execute:
78
+ + `s3_server_side_encryption` : S3 Server-Side Encryption (Only aes256 is supported)
16
79
 
17
- $ bundle
80
+ + `redshift_host` (required) : the end point(or hostname) of your Amazon Redshift cluster.
18
81
 
19
- Or install it yourself as:
82
+ + `redshift_port` (required) : port number.
20
83
 
21
- $ gem install fluent-plugin-redshift-v2
84
+ + `redshift_dbname` (required) : database name.
22
85
 
23
- ## Usage
86
+ + `redshift_user` (required) : user name.
24
87
 
25
- TODO: Write usage instructions here
88
+ + `redshift_password` (required) : password for the user name.
26
89
 
27
- ## Development
90
+ + `redshift_tablename` (required) : table name to store data.
28
91
 
29
- After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
92
+ + `redshift_schemaname` : schema name to store data. By default, this option is not set and find table without schema as your own search_path.
30
93
 
31
- To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
94
+ + `redshift_connect_timeout` : maximum time to wait for connection to succeed.
32
95
 
33
- ## Contributing
96
+ + `redshift_copy_columns` : columns for copying. Value needs to be comma-separated like `id,name,age`
97
+
98
+ + `file_type` : file format of the source data. `csv`, `tsv`, `msgpack` or `json` are available.
99
+
100
+ + `delimiter` : delimiter of the source data. This option will be ignored if `file_type` is specified.
101
+
102
+ + `buffer_type` : buffer type.
103
+
104
+ + `buffer_path` : path prefix of the files to buffer logs.
105
+
106
+ + `flush_interval` : flush interval.
107
+
108
+ + `buffer_chunk_limit` : limit buffer size to chunk.
109
+
110
+ + `buffer_queue_limit` : limit buffer files to chunk.
111
+
112
+ + `num_threads` : number of threads to load data to redshift.
113
+
114
+ + `utc` : utc time zone. This parameter affects `timestamp_key_format`.
115
+
116
+ and standard buffered output options. (see https://docs.fluentd.org/v0.14/articles/buffer-plugin-overview)
117
+
118
+ ## Logging examples
119
+ ```ruby
120
+ # examples by fluent-logger
121
+ require 'fluent-logger'
122
+ log = Fluent::Logger::FluentLogger.new(nil, host: 'localhost', port: 24224)
123
+
124
+ # file_type: csv
125
+ log.post('your.tag', log: "12345,12345")
126
+
127
+ # file_type: tsv
128
+ log.post('your.tag', log: "12345\t12345")
129
+
130
+ # file_type: json
131
+ require 'json'
132
+ log.post('your.tag', { user_id: 12345, data_id: 12345 }.to_json)
133
+
134
+ # file_type: msgpack
135
+ log.post('your.tag', user_id: 12345, data_id: 12345)
136
+ ```
34
137
 
35
- Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/fluent-plugin-redshift-v2.
138
+ ## License
36
139
 
140
+ [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)
data/fluent-plugin-redshift-v2.gemspec CHANGED
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
4
 
5
5
  Gem::Specification.new do |spec|
6
6
  spec.name = "fluent-plugin-redshift-v2"
7
- spec.version = "0.1.2"
7
+ spec.version = "0.1.3"
8
8
  spec.authors = ["Jun Yokoyama"]
9
9
  spec.email = ["jun@larus.org"]
10
10
 
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-redshift-v2
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.2
4
+ version: 0.1.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Jun Yokoyama
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2017-11-02 00:00:00.000000000 Z
11
+ date: 2017-11-17 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler