logstash-input-rethinkdb 0.1.0 → 0.2.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (5) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +34 -60
  3. data/lib/logstash/inputs/rethinkdb.rb +15 -4
  4. data/logstash-input-rethinkdb.gemspec +2 -2
  5. metadata +37 -35
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: ee3f3933a77ac91a562be9e4a09a70663a456382
4
- data.tar.gz: a8170147fd7d527903a6bb6cc46ca40518973bbb
3
+ metadata.gz: b4f5a5863e0ceea457c8d6bfba285c4421a02305
4
+ data.tar.gz: b9137a56e7dbe3f34d8bd439c104cff2b1085172
5
5
  SHA512:
6
- metadata.gz: da1a6acba6a759f9533c3ae53473a80f1be1a2a4e0370ca0e69c3f420d6c06caf31c5b1b51f4ef2126e99b7b65c20b317a5ceb5208b6fdb00d4814595f58ddfd
7
- data.tar.gz: fe318a16b281d0558bf08aa3e504ed40ef437ab9c4282d050c230397465ca129dbfce5977f99905292de3334fab2ae333c204f3dfce073b0044b035284685d0d
6
+ metadata.gz: 067b0b57606401155fac68fbc4341b70eaad764f0a9368a9cdd4acd75c866a534d9f16c6711f655617fc2dee547ccb49644f6794985be2bd972ce2d6555132f0
7
+ data.tar.gz: f817381a2c177a65fa8f2d7f08065813e7a42753412ba06c5c103b44de589aa7d95c6fcdd07707dc61f41fcd14eab5a8bd8494cc723cafec6033bad741f500b9
data/README.md CHANGED
@@ -2,85 +2,59 @@
2
2
 
3
3
  This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
4
4
 
5
- It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
5
+ **Currently this is BETA software. It includes some limitations that won't make it suitable for use in production until RethinkDB 2.2 is released**
6
6
 
7
- ## Documentation
7
+ This plugin will eventually replace the [RethinkDB river](https://github.com/rethinkdb/elasticsearch-river-rethinkdb) as the preferred method of syncing RethinkDB and ElasticSearch. We're hoping to get early feedback on how useful the plugin is, and what can be improved (beyond the known limitations, see below).
8
8
 
9
- Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
10
9
 
11
- - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
12
- - For more asciidoc formatting tips, see the excellent reference here https://github.com/elasticsearch/docs#asciidoc-guide
10
+ ## Using the plugin
13
11
 
14
- ## Need Help?
12
+ You'll need to use this with the Logstash 1.5 release candidate, which you can [download here](https://www.elastic.co/downloads/logstash).
15
13
 
16
- Need help? Try #logstash on freenode IRC or the logstash-users@googlegroups.com mailing list.
17
-
18
- ## Developing
19
-
20
- ### 1. Plugin Developement and Testing
21
-
22
- #### Code
23
- - To get started, you'll need JRuby with the Bundler gem installed.
24
-
25
- - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
26
-
27
- - Install dependencies
14
+ - Install the plugin from the Logstash home directory
28
15
  ```sh
29
- bundle install
16
+ $ bin/plugin install logstash-input-rethinkdb
30
17
  ```
31
18
 
32
- #### Test
33
-
34
- - Update your dependencies
19
+ - Now you can test the plugin using the stdout output:
35
20
 
36
21
  ```sh
37
- bundle install
22
+ $ bin/logstash -e '
23
+ input {rethinkdb
24
+ {host => "localhost"
25
+ port => 28015
26
+ auth_key => ""
27
+ watch_dbs => ["db1", "db2"]
28
+ watch_tables => ["test.foo", "db2.baz"]
29
+ backfill => true
30
+ }}
31
+ output {stdout {codec => json_lines}}'
38
32
  ```
39
33
 
40
- - Run tests
34
+ This will immediately watch the tables `test.foo` and `db2.baz`, and it will also watch the databases `db1` and `db2` for new or dropped tables and watch or unwatch those tables appropriately. Since `backfill` is `true`, it will automatically send events for the documents that already exist in those tables during initialization.
41
35
 
42
- ```sh
43
- bundle exec rspec
44
- ```
45
36
 
46
- ### 2. Running your unpublished Plugin in Logstash
37
+ Since `backfill` is `true`, it will automatically watch any tables it finds in `db1` or `db2`, and will send events over logstash for each document already in those tables.
47
38
 
48
- #### 2.1 Run in a local Logstash clone
39
+ ### Format of the events:
49
40
 
50
- - Edit Logstash `Gemfile` and add the local plugin path, for example:
51
- ```ruby
52
- gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
53
- ```
54
- - Install plugin
55
- ```sh
56
- bin/plugin install --no-verify
57
- ```
58
- - Run Logstash with your plugin
59
- ```sh
60
- bin/logstash -e 'filter {awesome {}}'
61
- ```
62
- At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
63
-
64
- #### 2.2 Run in an installed Logstash
65
-
66
- You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
41
+ The events are encoded with the "json_lines" codec, which puts compressed json documents one event per line
67
42
 
68
- - Build your plugin gem
69
- ```sh
70
- gem build logstash-filter-awesome.gemspec
71
- ```
72
- - Install the plugin from the Logstash home
73
- ```sh
74
- bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
75
- ```
76
- - Start Logstash and proceed to test the plugin
43
+ Fields:
77
44
 
78
- ## Contributing
45
+ - **db**: the database that emitted the event
46
+ - **table**: the table that emitted the event
47
+ - **old_val**: the old value of the document (see [changefeeds](http://rethinkdb.com/docs/changefeeds/ruby/))
48
+ - **new_val**: the new value of the document
49
+ - **@timestamp**: timestamp added by logstash (so may not correspond to the rethinkdb server time the change was emitted)
50
+ - **@version**: version number added by logstash (always 1)
79
51
 
80
- All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
52
+ ## Known limitations
81
53
 
82
- Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
54
+ There are two big limitations that prevent this plugin from being useful for production:
55
+ 1. Until RethinkDB supports resuming changefeeds, this plugin cannot guarantee that no changes are missed if a connection to the database is dropped. Again, once that functionality is implemented, this plugin will be modified to provide reliable "at least once" semantics for changes (meaning once it reconnects, it can catch back up and send any changes that it missed).
56
+ 2. Documents that are deleted in RethinkDB while the LogStash plugin is disconnected will not be synchronized. This is true even if `backfill` is enabled. This limitation is a consequence of LogStash operating on a document-by-document basis.
83
57
 
84
- It is more important to the community that you are able to contribute.
58
+ ## License
85
59
 
86
- For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
60
+ Apache 2.0
data/lib/logstash/inputs/rethinkdb.rb CHANGED
@@ -19,12 +19,17 @@ class LogStash::Inputs::RethinkDB < LogStash::Inputs::Base
19
19
  # Auth key of RethinkDB server (don't provide if nil)
20
20
  config :auth_key, :validate => :string, :default => ""
21
21
  # Time period to squash changefeeds on. Defaults to no squashing.
22
- config :squash, :default => false
22
+ config :squash, :default => true
23
23
  # Which tables to watch for changes
24
24
  config :watch_tables, :validate => :array, :default => []
25
25
  # Which databases to watch for changes. Tables added or removed from
26
26
  # these databases will be watched or unwatched accordingly
27
27
  config :watch_dbs, :validate => :array, :default => []
28
+ # Whether to backfill documents from the dbs and tables when
29
+ # (re)connecting to RethinkDB. This ensures all documents in the
30
+ # RethinkDB tables will be sent over logstash, but it may cause a
31
+ # lot of traffic with very large tables and/or unstable connections.
32
+ config :backfill, :default => true
28
33
 
29
34
 
30
35
  # Part of the logstash input interface
@@ -34,6 +39,8 @@ class LogStash::Inputs::RethinkDB < LogStash::Inputs::Base
34
39
  # {db => QueryHandle}
35
40
  @db_feeds = {}
36
41
  @queue = nil
42
+ @backfill = @backfill && @backfill != 'false'
43
+ @squash = @squash && @squash != 'true'
37
44
  end
38
45
 
39
46
  # # Part of the logstash input interface
@@ -128,14 +135,18 @@ class LogStash::Inputs::RethinkDB < LogStash::Inputs::Base
128
135
  table('table_status').
129
136
  filter {|row| row['status']['all_replicas_ready']}.
130
137
  pluck('db', 'name').
131
- changes(:squash => @squash, :include_states => true).
138
+ changes(:include_initial => @backfill,
139
+ :squash => @squash,
140
+ :include_states => true).
132
141
  em_run(@conn, handler)
133
142
  end
134
143
 
135
144
  def create_table_feed(db, table, handler)
136
145
  r.db(db).
137
146
  table(table).
138
- changes(:squash => @squash, :include_states => true).
147
+ changes(:include_initial => @backfill,
148
+ :squash => @squash,
149
+ :include_states => true).
139
150
  em_run(@conn, handler)
140
151
  end
141
152
 
@@ -166,7 +177,7 @@ class TableHandler < RethinkDB::Handler
166
177
  @plugin = plugin
167
178
  end
168
179
 
169
- def on_initial(val)
180
+ def on_initial_val(val)
170
181
  @plugin.send(@db, @table, nil, val)
171
182
  end
172
183
 
data/logstash-input-rethinkdb.gemspec CHANGED
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-input-rethinkdb'
3
- s.version = '0.1.0'
3
+ s.version = '0.2.0'
4
4
  s.licenses = ['Apache License (2.0)']
5
5
  s.summary = "Streams changes from RethinkDB tables into LogStash."
6
6
  s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
@@ -20,7 +20,7 @@ Gem::Specification.new do |s|
20
20
  # Gem dependencies
21
21
  s.add_runtime_dependency "logstash-core", '>= 1.4.0', '< 2.0.0'
22
22
  s.add_runtime_dependency 'logstash-codec-plain'
23
- s.add_runtime_dependency 'rethinkdb', '~>2.0.0'
23
+ s.add_runtime_dependency 'rethinkdb', '~>2.2.0'
24
24
  s.add_runtime_dependency 'eventmachine', '~>1.0.7'
25
25
  s.add_development_dependency 'logstash-devutils'
26
26
  end
metadata CHANGED
@@ -1,98 +1,100 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-input-rethinkdb
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - RethinkDB
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-04-15 00:00:00.000000000 Z
11
+ date: 2015-11-18 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
+ name: logstash-core
14
15
  requirement: !ruby/object:Gem::Requirement
15
16
  requirements:
16
- - - '>='
17
+ - - ">="
17
18
  - !ruby/object:Gem::Version
18
19
  version: 1.4.0
19
- - - <
20
+ - - "<"
20
21
  - !ruby/object:Gem::Version
21
22
  version: 2.0.0
22
- name: logstash-core
23
- prerelease: false
24
23
  type: :runtime
24
+ prerelease: false
25
25
  version_requirements: !ruby/object:Gem::Requirement
26
26
  requirements:
27
- - - '>='
27
+ - - ">="
28
28
  - !ruby/object:Gem::Version
29
29
  version: 1.4.0
30
- - - <
30
+ - - "<"
31
31
  - !ruby/object:Gem::Version
32
32
  version: 2.0.0
33
33
  - !ruby/object:Gem::Dependency
34
+ name: logstash-codec-plain
34
35
  requirement: !ruby/object:Gem::Requirement
35
36
  requirements:
36
- - - '>='
37
+ - - ">="
37
38
  - !ruby/object:Gem::Version
38
39
  version: '0'
39
- name: logstash-codec-plain
40
- prerelease: false
41
40
  type: :runtime
41
+ prerelease: false
42
42
  version_requirements: !ruby/object:Gem::Requirement
43
43
  requirements:
44
- - - '>='
44
+ - - ">="
45
45
  - !ruby/object:Gem::Version
46
46
  version: '0'
47
47
  - !ruby/object:Gem::Dependency
48
+ name: rethinkdb
48
49
  requirement: !ruby/object:Gem::Requirement
49
50
  requirements:
50
- - - ~>
51
+ - - "~>"
51
52
  - !ruby/object:Gem::Version
52
- version: 2.0.0
53
- name: rethinkdb
54
- prerelease: false
53
+ version: 2.2.0
55
54
  type: :runtime
55
+ prerelease: false
56
56
  version_requirements: !ruby/object:Gem::Requirement
57
57
  requirements:
58
- - - ~>
58
+ - - "~>"
59
59
  - !ruby/object:Gem::Version
60
- version: 2.0.0
60
+ version: 2.2.0
61
61
  - !ruby/object:Gem::Dependency
62
+ name: eventmachine
62
63
  requirement: !ruby/object:Gem::Requirement
63
64
  requirements:
64
- - - ~>
65
+ - - "~>"
65
66
  - !ruby/object:Gem::Version
66
67
  version: 1.0.7
67
- name: eventmachine
68
- prerelease: false
69
68
  type: :runtime
69
+ prerelease: false
70
70
  version_requirements: !ruby/object:Gem::Requirement
71
71
  requirements:
72
- - - ~>
72
+ - - "~>"
73
73
  - !ruby/object:Gem::Version
74
74
  version: 1.0.7
75
75
  - !ruby/object:Gem::Dependency
76
+ name: logstash-devutils
76
77
  requirement: !ruby/object:Gem::Requirement
77
78
  requirements:
78
- - - '>='
79
+ - - ">="
79
80
  - !ruby/object:Gem::Version
80
81
  version: '0'
81
- name: logstash-devutils
82
- prerelease: false
83
82
  type: :development
83
+ prerelease: false
84
84
  version_requirements: !ruby/object:Gem::Requirement
85
85
  requirements:
86
- - - '>='
86
+ - - ">="
87
87
  - !ruby/object:Gem::Version
88
88
  version: '0'
89
- description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
89
+ description: This gem is a logstash plugin required to be installed on top of the
90
+ Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not
91
+ a stand-alone program
90
92
  email: josh@rethinkdb.com
91
93
  executables: []
92
94
  extensions: []
93
95
  extra_rdoc_files: []
94
96
  files:
95
- - .gitignore
97
+ - ".gitignore"
96
98
  - Gemfile
97
99
  - LICENSE
98
100
  - README.md
@@ -106,24 +108,24 @@ licenses:
106
108
  metadata:
107
109
  logstash_plugin: 'true'
108
110
  logstash_group: input
109
- post_install_message:
111
+ post_install_message:
110
112
  rdoc_options: []
111
113
  require_paths:
112
114
  - lib
113
115
  required_ruby_version: !ruby/object:Gem::Requirement
114
116
  requirements:
115
- - - '>='
117
+ - - ">="
116
118
  - !ruby/object:Gem::Version
117
119
  version: '0'
118
120
  required_rubygems_version: !ruby/object:Gem::Requirement
119
121
  requirements:
120
- - - '>='
122
+ - - ">="
121
123
  - !ruby/object:Gem::Version
122
124
  version: '0'
123
125
  requirements: []
124
- rubyforge_project:
125
- rubygems_version: 2.1.9
126
- signing_key:
126
+ rubyforge_project:
127
+ rubygems_version: 2.4.5.1
128
+ signing_key:
127
129
  specification_version: 4
128
130
  summary: Streams changes from RethinkDB tables into LogStash.
129
131
  test_files: