fileminer 1.0.2 → 1.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +182 -1
- data/conf/fileminer_default.yml +12 -0
- data/lib/fileminer/output/mysql.rb +22 -20
- data/lib/fileminer/version.rb +1 -1
- data/lib/fileminer.rb +13 -1
- metadata +3 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 721d984d3d2e8c905f0bf86d52b2d89fb0adbb70c651ddd7d48cac16720d3584
|
4
|
+
data.tar.gz: 12015a1aba0b44df10b4033ff476a4d7818d3db250e5bbbe9b0a9c4d1afc4c3b
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: b0c9305e8e2ca63f13812593cdd1d9697de6fd51d9e633a5249accd704dbbc487924e2a36b2d2ff40981d7a271d79325a2a607b4618d1b82ca56693d7f9dad7a
|
7
|
+
data.tar.gz: 017dddf1edc9b4859124504e05bb1d2ed46b6d52c9bec053e56b838e9ded56027d6cbeedd768578ebfa9249c6e716f2b872157f77ad0cf06073f95b471cb2510
|
data/README.md
CHANGED
@@ -1,2 +1,183 @@
|
|
1
1
|
# fileminer
|
2
|
-
|
2
|
+
[![Gem Version](https://d25lcipzij17d.cloudfront.net/badge.svg?id=rb&type=6&v=1.1.0&x2=0)](https://rubygems.org/gems/fileminer)
|
3
|
+
[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/fmjsjx/fileminer/blob/master/LICENSE)
|
4
|
+
|
5
|
+
|
6
|
+
A simple line based file/log transfer tool coding by ruby.
|
7
|
+
|
8
|
+
# quick start
|
9
|
+
1. installation
|
10
|
+
|
11
|
+
Install fileminer from RubyGems:
|
12
|
+
```
|
13
|
+
$ gem install fileminer
|
14
|
+
Fetching fileminer-1.1.0.gem
|
15
|
+
Successfully installed fileminer-1.1.0
|
16
|
+
Parsing documentation for fileminer-1.1.0
|
17
|
+
Installing ri documentation for fileminer-1.1.0
|
18
|
+
Done installing documentation for fileminer after 0 seconds
|
19
|
+
1 gem installed
|
20
|
+
$ _
|
21
|
+
```
|
22
|
+
|
23
|
+
2. setting up fileminer
|
24
|
+
|
25
|
+
* generate configuration file
|
26
|
+
|
27
|
+
At first, we should generate the fileminer configuration file.
|
28
|
+
|
29
|
+
We provide a command tool 'fileminer-genconf' to generate configurations:
|
30
|
+
```
|
31
|
+
$ fileminer-genconf -h
|
32
|
+
Usage: fileminer-genconf [options]
|
33
|
+
|
34
|
+
Options:
|
35
|
+
-t, --type fileminer|supervisor Type of the config file to be generated
|
36
|
+
Default is fileminer
|
37
|
+
-o, --out [path] Output content to a file
|
38
|
+
For type fileminer, default is /etc/fileminer/fileminer.yml
|
39
|
+
For type supervisor, default is ./fileminer.ini
|
40
|
+
-h, --help Print help
|
41
|
+
$ _
|
42
|
+
```
|
43
|
+
|
44
|
+
Generation with default options:
|
45
|
+
```
|
46
|
+
$ fileminer-genconf -o
|
47
|
+
generated config file: /etc/fileminer/fileminer.yml
|
48
|
+
$ _
|
49
|
+
```
|
50
|
+
|
51
|
+
* config input on fileminer.yml
|
52
|
+
|
53
|
+
Edit fileminer.inputs on fileminer.yml:
|
54
|
+
```yaml
|
55
|
+
fileminer.inputs:
|
56
|
+
paths:
|
57
|
+
- /path/to/*.log
|
58
|
+
```
|
59
|
+
|
60
|
+
* configure output on fileminer.yml
|
61
|
+
|
62
|
+
In current version, fileminer provides three prefab output plugins: redis, kafka & mysql. One process can only choose one of them.
|
63
|
+
|
64
|
+
**Output to redis:**
|
65
|
+
|
66
|
+
Redis output plugin now using LPUSH to send messages for each line.
|
67
|
+
|
68
|
+
Install redis client, fileminer use hiredis by default:
|
69
|
+
```
|
70
|
+
$ gem install redis hiredis
|
71
|
+
Fetching redis-4.1.0.gem
|
72
|
+
Successfully installed redis-4.1.0
|
73
|
+
Parsing documentation for redis-4.1.0
|
74
|
+
Installing ri documentation for redis-4.1.0
|
75
|
+
Done installing documentation for redis after 0 seconds
|
76
|
+
Fetching hiredis-0.6.3.gem
|
77
|
+
Building native extensions. This could take a while...
|
78
|
+
Successfully installed hiredis-0.6.3
|
79
|
+
Parsing documentation for hiredis-0.6.3
|
80
|
+
Installing ri documentation for hiredis-0.6.3
|
81
|
+
Done installing documentation for hiredis after 0 seconds
|
82
|
+
2 gems installed
|
83
|
+
$ _
|
84
|
+
```
|
85
|
+
|
86
|
+
Edit fileminer.yml:
|
87
|
+
```yaml
|
88
|
+
output.redis:
|
89
|
+
# target redis server URI
|
90
|
+
uri: redis://localhost:6379/0
|
91
|
+
# target redis key, type must be LIST
|
92
|
+
key: fileminer
|
93
|
+
```
|
94
|
+
|
95
|
+
|
96
|
+
**Output to kafka:**
|
97
|
+
|
98
|
+
Install kafka ruby client:
|
99
|
+
```
|
100
|
+
$ gem install ruby-kafka
|
101
|
+
Successfully installed ruby-kafka-0.7.5
|
102
|
+
Parsing documentation for ruby-kafka-0.7.5
|
103
|
+
Done installing documentation for ruby-kafka after 0 seconds
|
104
|
+
1 gem installed
|
105
|
+
$ _
|
106
|
+
```
|
107
|
+
|
108
|
+
Edit fileminer.yml
|
109
|
+
```yaml
|
110
|
+
output.kafka:
|
111
|
+
brokers: ['host1:9092','host2:9092','host3:9092']
|
112
|
+
client_id: fileminer
|
113
|
+
topic: fileminer
|
114
|
+
```
|
115
|
+
|
116
|
+
|
117
|
+
**Output to MySQL:**
|
118
|
+
|
119
|
+
Install mysql2:
|
120
|
+
```
|
121
|
+
$ gem install mysql2
|
122
|
+
Building native extensions. This could take a while...
|
123
|
+
Successfully installed mysql2-0.5.2
|
124
|
+
Parsing documentation for mysql2-0.5.2
|
125
|
+
Done installing documentation for mysql2 after 0 seconds
|
126
|
+
1 gem installed
|
127
|
+
$ _
|
128
|
+
```
|
129
|
+
|
130
|
+
Edit fileminer.yml
|
131
|
+
```yaml
|
132
|
+
output.mysql:
|
133
|
+
host: hostname
|
134
|
+
port: 3306
|
135
|
+
username: someuser
|
136
|
+
password: somepwd
|
137
|
+
database: somedb
|
138
|
+
table: sometable
|
139
|
+
```
|
140
|
+
|
141
|
+
|
142
|
+
*Since 1.1.0, you can also use customized output plugin.*
|
143
|
+
|
144
|
+
**Output using customized output plugin:**
|
145
|
+
|
146
|
+
Get your script ready:
|
147
|
+
```ruby
|
148
|
+
require 'fileminer/plugins'
|
149
|
+
|
150
|
+
class YourPlugin < Output::OutputPlugin
|
151
|
+
def initialize options
|
152
|
+
...
|
153
|
+
end
|
154
|
+
...
|
155
|
+
public
|
156
|
+
def send_all(lines, &listener)
|
157
|
+
...
|
158
|
+
end
|
159
|
+
...
|
160
|
+
end
|
161
|
+
```
|
162
|
+
|
163
|
+
Edit fileminer.yml
|
164
|
+
```yaml
|
165
|
+
output.script:
|
166
|
+
script: /path/to/your_script.rb
|
167
|
+
plugin_class: YourPlugin
|
168
|
+
init_options:
|
169
|
+
some_field: some value
|
170
|
+
```
|
171
|
+
|
172
|
+
3. runnning fileminer
|
173
|
+
|
174
|
+
Run in command line:
|
175
|
+
```
|
176
|
+
$ fileminer
|
177
|
+
|
178
|
+
```
|
179
|
+
or
|
180
|
+
```
|
181
|
+
$ fileminer /path/to/fileminer.yml
|
182
|
+
|
183
|
+
```
|
data/conf/fileminer_default.yml
CHANGED
@@ -139,3 +139,15 @@ output.redis:
|
|
139
139
|
|
140
140
|
# table name
|
141
141
|
#table: sometable
|
142
|
+
|
143
|
+
# using customized ruby script
|
144
|
+
#output.script:
|
145
|
+
# customized ruby script path
|
146
|
+
#script: /path/to/script.rb
|
147
|
+
|
148
|
+
# class name of the custom plugin extends Output::OutputPlugin
|
149
|
+
#plugin_class: Output::CustomPlugin
|
150
|
+
|
151
|
+
# options to be input when initialize the custom plugin
|
152
|
+
#init_options:
|
153
|
+
# some fields
|
@@ -14,7 +14,6 @@ module Output
|
|
14
14
|
ssl_mode: :disabled
|
15
15
|
}
|
16
16
|
|
17
|
-
#
|
18
17
|
# Create a mysql output plugin instance
|
19
18
|
#
|
20
19
|
# @param [Hash] options
|
@@ -37,30 +36,20 @@ module Output
|
|
37
36
|
@encoding = conf[:encoding]
|
38
37
|
conf[:ssl_mode] = :disabled if conf[:ssl_mode] != :enabled
|
39
38
|
@mysql = Mysql2::Client.new conf
|
40
|
-
@mysql_conf = conf
|
41
39
|
create_table_if_not_exists
|
42
40
|
@sqls = Hash.new { |hash, key| hash[key] = generate_batch_sql key }
|
43
41
|
end
|
44
42
|
|
45
43
|
private
|
46
44
|
def create_table_if_not_exists
|
47
|
-
|
48
|
-
rs = mysql_client.query 'SHOW TABLES'
|
45
|
+
rs = @mysql.query 'SHOW TABLES'
|
49
46
|
tables = rs.map { |row| row.values[0] }
|
50
47
|
unless tables.include? @table
|
51
48
|
sql = create_table_sql
|
52
|
-
|
49
|
+
@mysql.query sql
|
53
50
|
end
|
54
51
|
end
|
55
52
|
|
56
|
-
def get_mysql_client
|
57
|
-
mysql_client = @mysql
|
58
|
-
if mysql_client.closed?
|
59
|
-
@mysql = mysql_client = Mysql2::Client.new @mysql_conf
|
60
|
-
end
|
61
|
-
mysql_client
|
62
|
-
end
|
63
|
-
|
64
53
|
def create_table_sql
|
65
54
|
<<-EOS
|
66
55
|
CREATE TABLE `#@table` (
|
@@ -79,7 +68,14 @@ module Output
|
|
79
68
|
"INSERT IGNORE INTO `#@table`(`host`,`path`,`pos`,`end`,`data`) VALUES " << (['(?,?,?,?,?)'] * size).join(',')
|
80
69
|
end
|
81
70
|
|
82
|
-
|
71
|
+
def get_batch_sql(size)
|
72
|
+
if @sqls.key? size
|
73
|
+
@sqls[size]
|
74
|
+
else
|
75
|
+
@sqls[size] = generate_batch_sql size
|
76
|
+
end
|
77
|
+
end
|
78
|
+
|
83
79
|
# Send all lines to mysql
|
84
80
|
#
|
85
81
|
# @param [Array] lines
|
@@ -87,12 +83,18 @@ module Output
|
|
87
83
|
public
|
88
84
|
def send_all(lines, &listener)
|
89
85
|
values = lines.flat_map { |line| [line[:host], line[:path], line[:pos], line[:end], line[:data]] }
|
90
|
-
sql =
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
86
|
+
sql = get_batch_sql lines.size
|
87
|
+
@mysql.query 'BEGIN'
|
88
|
+
begin
|
89
|
+
stat = @mysql.prepare sql
|
90
|
+
stat.execute *values
|
91
|
+
stat.close
|
92
|
+
@mysql.query 'COMMIT'
|
93
|
+
listener.call
|
94
|
+
rescue => err
|
95
|
+
@mysql.query 'ROLLBACK'
|
96
|
+
raise err
|
97
|
+
end
|
96
98
|
end
|
97
99
|
|
98
100
|
end
|
data/lib/fileminer/version.rb
CHANGED
data/lib/fileminer.rb
CHANGED
@@ -13,7 +13,7 @@ class FileMiner
|
|
13
13
|
refresh_files_time_trigger: '30s',
|
14
14
|
max_time_of_each_mining: '5s',
|
15
15
|
max_lines_of_each_mining: -1,
|
16
|
-
max_lines_of_each_file: -1,
|
16
|
+
max_lines_of_each_file: -1,
|
17
17
|
}
|
18
18
|
|
19
19
|
attr_reader :miner, :output, :running
|
@@ -103,6 +103,9 @@ class FileMiner
|
|
103
103
|
when conf.key?('output.mysql')
|
104
104
|
mysql_conf = conf['output.mysql'].keys_to_sym
|
105
105
|
init_output_mysql mysql_conf
|
106
|
+
when conf.key?('output.script')
|
107
|
+
script_conf = conf['output.script'].keys_to_sym
|
108
|
+
init_output_script script_conf
|
106
109
|
else
|
107
110
|
raise 'Missing config for output'
|
108
111
|
end
|
@@ -136,6 +139,15 @@ class FileMiner
|
|
136
139
|
Output::MysqlPlugin.new mysql_conf
|
137
140
|
end
|
138
141
|
|
142
|
+
def init_output_script(script_conf)
|
143
|
+
script_path = script_conf[:script]
|
144
|
+
plugin_class_name = script_conf[:plugin_class]
|
145
|
+
init_options = script_conf[:init_options] || {}
|
146
|
+
require script_path
|
147
|
+
plugin_class = Object.const_get plugin_class_name
|
148
|
+
plugin_class.new init_options.keys_to_sym
|
149
|
+
end
|
150
|
+
|
139
151
|
def send_lines(record, lines)
|
140
152
|
if @output.batch?
|
141
153
|
@output.send_all lines do
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fileminer
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0
|
4
|
+
version: 1.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Fang MinJie
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2019-06-
|
11
|
+
date: 2019-06-20 00:00:00.000000000 Z
|
12
12
|
dependencies: []
|
13
13
|
description: A simple file/log transfer tool coding by ruby.
|
14
14
|
email:
|
@@ -54,7 +54,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
54
54
|
- !ruby/object:Gem::Version
|
55
55
|
version: '0'
|
56
56
|
requirements: []
|
57
|
-
rubygems_version: 3.0.
|
57
|
+
rubygems_version: 3.0.1
|
58
58
|
signing_key:
|
59
59
|
specification_version: 4
|
60
60
|
summary: A simple file/log transfer tool coding by ruby.
|