fluent-plugin-watch-process 0.0.1 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.travis.yml +10 -3
- data/Appraisals +7 -0
- data/README.md +73 -15
- data/example1.conf +9 -11
- data/example2.conf +3 -2
- data/fluent-plugin-watch-process.gemspec +7 -3
- data/gemfiles/fluentd_v0.14.gemfile +7 -0
- data/lib/fluent/plugin/in_watch_process.rb +178 -73
- data/test/helper.rb +1 -9
- data/test/plugin/test_in_watch_process.rb +100 -5
- metadata +80 -21
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 2d114a31b165c3feca90dac92247785d9f757e8951ded1aac49bc30806e505a1
|
4
|
+
data.tar.gz: 9a2aab2afb95873873e3e3581b2f5f23b12bd0e7062d456b7eebe257c7f8473a
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: ad4b0bffb25faaed2082c8bab5904100750ee265113097a0f643d5c13305c53731a9afcb6bd0b661fba1d012b45e74294c57defd530a774fbaf052135bcb6f43
|
7
|
+
data.tar.gz: 8f4761bfec54a36ecb2f702d7749ce4fd21bb69f35834aaeef274167456de74e0e8c4d7d09bdaaa90dc9bc3647d66eafbeb16d81ca24aede35fb4d273edd7e43
|
data/.travis.yml
CHANGED
data/Appraisals
ADDED
data/README.md
CHANGED
@@ -3,17 +3,17 @@ fluent-plugin-watch-process [![Build Status](https://travis-ci.org/y-ken/fluent-
|
|
3
3
|
|
4
4
|
## Overview
|
5
5
|
|
6
|
-
Fluentd Input plugin to collect process information via ps command.
|
6
|
+
Fluentd Input plugin to collect continual process information via ps command. It is useful for cron/barch process monitoring.
|
7
7
|
|
8
8
|
## Use Cases
|
9
9
|
|
10
|
-
* collect cron/batch process for analysis.
|
10
|
+
* collect cron/batch process for long term analysis.
|
11
11
|
* high cpu load time
|
12
12
|
* high usage of memory
|
13
13
|
* determine too long running task
|
14
14
|
|
15
15
|
* output destination example
|
16
|
-
* Elasticsearch + Kibana to visualize statistics. Example: [example1.conf](https://github.com/y-ken/fluent-plugin-watch-process/blob/master/example1.conf)
|
16
|
+
* Elasticsearch + Kibana to visualize cron/batch process statistics. Example: [example1.conf](https://github.com/y-ken/fluent-plugin-watch-process/blob/master/example1.conf)
|
17
17
|
* save process information as audit log into AWS S3 which filename isolated by hostname. Example: [example2.conf](https://github.com/y-ken/fluent-plugin-watch-process/blob/master/example2.conf)
|
18
18
|
|
19
19
|
## Installation
|
@@ -25,7 +25,10 @@ install with gem or fluent-gem command as:
|
|
25
25
|
$ gem install fluent-plugin-watch-process
|
26
26
|
|
27
27
|
# for td-agent
|
28
|
-
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-watch-process
|
28
|
+
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-watch-process -v 0.1.1
|
29
|
+
|
30
|
+
# for td-agent2 (recommend)
|
31
|
+
$ sudo td-agent-gem install fluent-plugin-watch-process -v 0.1.1
|
29
32
|
```
|
30
33
|
|
31
34
|
## Configuration
|
@@ -36,14 +39,14 @@ It is a quick sample to output log to `/var/log/td-agent/td-agent.log` with td-a
|
|
36
39
|
|
37
40
|
`````
|
38
41
|
<source>
|
39
|
-
type
|
42
|
+
@type watch_process
|
40
43
|
tag debug.batch.${hostname} # Required
|
41
44
|
lookup_user batchuser # Optional
|
42
45
|
interval 10s # Optional (default: 5s)
|
43
46
|
</source>
|
44
47
|
|
45
48
|
<match debug.**>
|
46
|
-
type
|
49
|
+
@type stdout
|
47
50
|
</match>
|
48
51
|
`````
|
49
52
|
|
@@ -51,41 +54,96 @@ After restarting td-agent, it will output process information to the td-agent.lo
|
|
51
54
|
|
52
55
|
`````
|
53
56
|
$ tail -f /var/log/td-agent/td-agent.log
|
57
|
+
...snip...
|
54
58
|
2014-01-16 14:21:34 +0900 debug.batch.localhost: {"start_time":"2014-01-16 14:21:13 +0900","user":"td-agent","pid":17486,"parent_pid":17483,"cpu_time":"00:00:00","cpu_percent":1.5,"memory_percent":3.5,"mem_rss":36068,"mem_size":60708,"state":"S","proc_name":"ruby","command":"/usr/lib64/fluent/ruby/bin/ruby /usr/sbin/td-agent --group td-agent --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid","elapsed_time":21}
|
55
59
|
`````
|
56
60
|
|
57
61
|
### Syntax
|
58
62
|
|
59
|
-
* tag
|
63
|
+
* tag (Required)
|
60
64
|
* record output destination
|
61
65
|
* supported tag placeholders are `${hostname}` and `__HOSTNAME__`.
|
62
66
|
|
63
|
-
* command
|
67
|
+
* command (Optional)
|
64
68
|
* execute ps command with some options
|
65
69
|
* [default] Linux: `LANG=en_US.UTF-8 && ps -ewwo lstart,user:20,pid,ppid,time,%cpu,%mem,rss,sz,s,comm,cmd`
|
66
70
|
* [default] MacOSX: `LANG=en_US.UTF-8 && ps -ewwo lstart,user,pid,ppid,time,%cpu,%mem,rss,vsz,state,comm,command`
|
71
|
+
* [default] Windows: described in the next section, `About Windows`.
|
67
72
|
|
68
|
-
* keys
|
73
|
+
* keys (Optional)
|
69
74
|
* output record keys of the ps command results
|
70
|
-
* [default] start_time
|
75
|
+
* [default] Linux, MacOSX: `start_time,user,pid,parent_pid,cpu_time,cpu_percent,memory_percent,mem_rss,mem_size,state,proc_name,command`
|
76
|
+
* need to modify `command` too if you modify this value.
|
77
|
+
* [default] Windows: `StartTime,UserName,SessionId,Id,CPU,WorkingSet,VirtualMemorySize,HandleCount,ProcessName`
|
78
|
+
* in Windows only, you can fix this without fixing `command`. These keys can be specified from the properties of `System.Diagnostics.Process` object of `.NET`.
|
79
|
+
* `UserName` key needs administrator privilege. You can exclude this to avoid needing administrator privilege.
|
71
80
|
|
72
|
-
* types
|
81
|
+
* types (Optional)
|
73
82
|
* settings of converting types from string to integer/float.
|
74
|
-
* [default] pid:integer
|
83
|
+
* [default] Linux, MacOSX: `pid:integer,parent_pid:integer,cpu_percent:float,memory_percent:float,mem_rss:integer,mem_size:integer`
|
84
|
+
* [default] Windows: `SessionId:integer,Id:integer,CPU:float,WorkingSet:integer,VirtualMemorySize:integer,HandleCount:integer`
|
75
85
|
|
76
|
-
* interval
|
86
|
+
* interval (Optional)
|
77
87
|
* execute interval time
|
78
88
|
* [default] 5s
|
79
89
|
|
80
|
-
* lookup_user
|
90
|
+
* lookup_user (Optional)
|
81
91
|
* filter process owner username with comma delimited
|
82
92
|
* [default] N/A
|
83
93
|
|
84
|
-
* hostname_command
|
94
|
+
* hostname_command (Optional)
|
85
95
|
* settings for tag placeholder, `${hostname}` and `__HOSTNAME__`. By default, it using long hostname.
|
86
96
|
* to use short hostname, set `hostname -s` for this option on linux/mac.
|
87
97
|
* [default] `hostname`
|
88
98
|
|
99
|
+
* powershell_command (Optional)
|
100
|
+
* settings for powershell command name. PowerShell Core had been renamed its command to `pwsh` and PowerShell 7 continues to use `pwsh` as its command name.
|
101
|
+
* [default] `powershell`
|
102
|
+
* [avaliables] `powershell`, `pwsh`
|
103
|
+
|
104
|
+
### About Windows
|
105
|
+
|
106
|
+
Default `command` preset for Windows provides many of keys as below. Generally, you can pick up the columns with `keys` option.
|
107
|
+
If you need additional keys, consider to update `command` option.
|
108
|
+
|
109
|
+
`````powershell
|
110
|
+
powershell -command "Get-Process -IncludeUserName
|
111
|
+
| ?{$_.StartTime -ne $NULL -and $_.CPU -ne $NULL}
|
112
|
+
| Select-Object -Property StartTime,UserName,SessionId,Id,CPU,WorkingSet,VirtualMemorySize,HandleCount,ProcessName
|
113
|
+
| %{$_.StartTime = $_.StartTime.ToString('o'); return $_;}
|
114
|
+
| ConvertTo-Csv -NoTypeInformation"
|
115
|
+
`````
|
116
|
+
|
117
|
+
Confirmed versions are:
|
118
|
+
|
119
|
+
| Windows version | PowerShell version information | Note |
|
120
|
+
| ------------------------------ | ------------------------------------------------------------------------- |-----------------------------------------------|
|
121
|
+
| Windows 10 10.0.19042 (20H2) | PSVersion: 5.1.19041.906 (default installed version), PSEdition: Desktop | `powershell_command` as `powershell` (default)|
|
122
|
+
| Windows 10 10.0.19042 (20H2) | PSVersion: 7.1.2, PSEdition: Core | `powershell_command` as `pwsh` |
|
123
|
+
|
124
|
+
|
125
|
+
Here are details of this default command.
|
126
|
+
|
127
|
+
* `Get-Process -IncludeUserName`
|
128
|
+
* `Get-Process` powershell command takes `System.Diagnostics.Process` objects.
|
129
|
+
* `IncludeUserName` option is needed to take `UserName`.
|
130
|
+
* this needs administrator privilege.
|
131
|
+
* this will be omitted if `keys` does not contain `UserName`.
|
132
|
+
* ` | ?{$_.StartTime -ne $NULL -and $_.CPU -ne $NULL}`
|
133
|
+
* this exlcludes some special processes that don't have some properties, such as the "Idle" process in Windows.
|
134
|
+
* ` | Select-Object -Property ...`
|
135
|
+
* this takes the necessary parameters from `System.Diagnostics.Process` objects.
|
136
|
+
* `...` part will be automatically fixed by `keys`.
|
137
|
+
* ` | %{$_.StartTime = $_.StartTime.ToString('o'); return $_;}`
|
138
|
+
* this fixes the format of `StartTime` value.
|
139
|
+
* note: in Windows, setting the "$env:Lang" environment variable is not effective in changing the format of the output.
|
140
|
+
* ` | ConvertTo-Csv -NoTypeInformation`
|
141
|
+
* this formats objects to csv strings.
|
142
|
+
* currently, it is needed that `command` outputs the results in csv format.
|
143
|
+
* this is because white space delimiter is not suitable for Windows, in which empty values are often mixed.
|
144
|
+
|
145
|
+
**Note:** When using with PowerShell 7 which is previously known as PowerShell Core, you must specify `powershell_command` parameter as `pwsh`. Otherwise, this plugin does not work correctly on PowerShell 7 (pwsh). This is because PowerShell Core and PowerShell 7 use different command name which is `pwsh` not `powershell`.
|
146
|
+
|
89
147
|
## FAQ
|
90
148
|
|
91
149
|
* I need hostname key in the record.
|
data/example1.conf
CHANGED
@@ -6,10 +6,10 @@
|
|
6
6
|
# * fluent-plugin-elasticsearch
|
7
7
|
|
8
8
|
<source>
|
9
|
-
type watch_process
|
9
|
+
@type watch_process
|
10
10
|
|
11
11
|
# specify output tag
|
12
|
-
tag
|
12
|
+
tag batch.process
|
13
13
|
|
14
14
|
# filter specific user owned process. if no need to filter, delete this line.
|
15
15
|
lookup_user batchuser
|
@@ -18,18 +18,16 @@
|
|
18
18
|
interval 10s
|
19
19
|
</source>
|
20
20
|
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
enable_ruby false
|
21
|
+
# add hostname key into record
|
22
|
+
<filter batch.process>
|
23
|
+
@type record_transformer
|
25
24
|
<record>
|
26
|
-
|
27
|
-
hostname ${hostname}
|
25
|
+
hostname ${hostname}
|
28
26
|
</record>
|
29
|
-
</
|
27
|
+
</filter>
|
30
28
|
|
31
|
-
<match
|
32
|
-
type
|
29
|
+
<match batch.*>
|
30
|
+
@type elasticsearch
|
33
31
|
host localhost
|
34
32
|
port 9200
|
35
33
|
logstash_format true
|
data/example2.conf
CHANGED
@@ -5,9 +5,10 @@
|
|
5
5
|
# * fluent-plugin-s3
|
6
6
|
|
7
7
|
<source>
|
8
|
-
type
|
8
|
+
@type watch_process
|
9
9
|
|
10
10
|
# use ${hostname} placeholder to use filename
|
11
|
+
# If you want to customize it, Use #{Socket.gethostname} or #{Socket.gethostname.split('.').first} snippet for tag.
|
11
12
|
tag batch_process.${hostname}
|
12
13
|
|
13
14
|
# filter specific user owned process. if no need to filter, delete this line.
|
@@ -18,7 +19,7 @@
|
|
18
19
|
</source>
|
19
20
|
|
20
21
|
<match batch_process.*>
|
21
|
-
type s3
|
22
|
+
@type s3
|
22
23
|
|
23
24
|
aws_key_id YOUR_AWS_KEY_ID
|
24
25
|
aws_sec_key YOUR_AWS_SECRET/KEY
|
@@ -3,11 +3,11 @@ $:.push File.expand_path("../lib", __FILE__)
|
|
3
3
|
|
4
4
|
Gem::Specification.new do |s|
|
5
5
|
s.name = "fluent-plugin-watch-process"
|
6
|
-
s.version = "0.0
|
6
|
+
s.version = "0.2.0"
|
7
7
|
s.authors = ["Kentaro Yoshida"]
|
8
8
|
s.email = ["y.ken.studio@gmail.com"]
|
9
9
|
s.homepage = "https://github.com/y-ken/fluent-plugin-watch-process"
|
10
|
-
s.summary = %q{Fluentd Input plugin to collect process information via ps command.}
|
10
|
+
s.summary = %q{Fluentd Input plugin to collect continual process information via ps command or PowerShell pwsh command for Linux/osx/Windows. It is useful for cron/barch process monitoring.}
|
11
11
|
|
12
12
|
s.files = `git ls-files`.split("\n")
|
13
13
|
s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
|
@@ -16,5 +16,9 @@ Gem::Specification.new do |s|
|
|
16
16
|
|
17
17
|
# specify any dependencies:
|
18
18
|
s.add_development_dependency "rake"
|
19
|
-
s.
|
19
|
+
s.add_development_dependency "test-unit", ">= 3.1.0"
|
20
|
+
s.add_development_dependency "appraisal"
|
21
|
+
s.add_runtime_dependency "fluentd", [">= 0.14.0", "< 2"]
|
22
|
+
s.add_runtime_dependency "fluent-mixin-rewrite-tag-name"
|
23
|
+
s.add_runtime_dependency "fluent-mixin-type-converter", ">= 0.1.0"
|
20
24
|
end
|
@@ -1,116 +1,221 @@
|
|
1
|
-
|
2
|
-
|
3
|
-
|
1
|
+
require 'time'
|
2
|
+
require 'csv' if Fluent.windows?
|
3
|
+
require "fluent/plugin/input"
|
4
|
+
require 'fluent/mixin/rewrite_tag_name'
|
5
|
+
require 'fluent/mixin/type_converter'
|
6
|
+
|
7
|
+
module Fluent::Plugin
|
8
|
+
class WatchProcessInput < Fluent::Plugin::Input
|
9
|
+
Fluent::Plugin.register_input('watch_process', self)
|
10
|
+
|
11
|
+
helpers :timer
|
12
|
+
|
13
|
+
DEFAULT_KEYS = %w(start_time user pid parent_pid cpu_time cpu_percent memory_percent mem_rss mem_size state proc_name command)
|
14
|
+
DEFAULT_TYPES = %w(
|
15
|
+
pid:integer
|
16
|
+
parent_pid:integer
|
17
|
+
cpu_percent:float
|
18
|
+
memory_percent:float
|
19
|
+
mem_rss:integer
|
20
|
+
mem_size:integer
|
21
|
+
).join(",")
|
4
22
|
|
5
23
|
config_param :tag, :string
|
6
24
|
config_param :command, :string, :default => nil
|
7
|
-
config_param :keys, :
|
8
|
-
config_param :
|
9
|
-
config_param :
|
10
|
-
config_param :lookup_user, :string, :default => nil
|
25
|
+
config_param :keys, :array, :default => nil
|
26
|
+
config_param :interval, :time, :default => '5s'
|
27
|
+
config_param :lookup_user, :array, :default => nil
|
11
28
|
config_param :hostname_command, :string, :default => 'hostname'
|
29
|
+
config_param :powershell_command, :enum, list: [:powershell, :pwsh], :default => :powershell
|
12
30
|
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
'float' => lambda { |v| v.to_f },
|
17
|
-
'bool' => lambda { |v|
|
18
|
-
case v.downcase
|
19
|
-
when 'true', 'yes', '1'
|
20
|
-
true
|
21
|
-
else
|
22
|
-
false
|
23
|
-
end
|
24
|
-
},
|
25
|
-
'time' => lambda { |v, time_parser|
|
26
|
-
time_parser.parse(v)
|
27
|
-
},
|
28
|
-
'array' => lambda { |v, delimiter|
|
29
|
-
v.to_s.split(delimiter)
|
30
|
-
}
|
31
|
-
}
|
31
|
+
include Fluent::HandleTagNameMixin
|
32
|
+
include Fluent::Mixin::RewriteTagName
|
33
|
+
include Fluent::Mixin::TypeConverter
|
32
34
|
|
33
35
|
def initialize
|
34
36
|
super
|
35
|
-
require 'time'
|
36
37
|
end
|
37
38
|
|
38
39
|
def configure(conf)
|
39
40
|
super
|
40
41
|
|
41
|
-
@
|
42
|
-
@keys
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
@interval = Config.time_value(@interval)
|
47
|
-
@hostname = `#{@hostname_command}`.chomp
|
48
|
-
$log.info "watch_process: polling start. :tag=>#{@tag} :lookup_user=>#{@lookup_user} :interval=>#{@interval} :command=>#{@command}"
|
42
|
+
@windows_watcher = WindowsWatcher.new(@keys, @command, @lookup_user, @powershell_command) if Fluent.windows?
|
43
|
+
@keys ||= Fluent.windows? ? @windows_watcher.keys : DEFAULT_KEYS
|
44
|
+
@command ||= get_ps_command
|
45
|
+
apply_default_types
|
46
|
+
log.info "watch_process: polling start. :tag=>#{@tag} :lookup_user=>#{@lookup_user} :interval=>#{@interval} :command=>#{@command}"
|
49
47
|
end
|
50
48
|
|
51
49
|
def start
|
52
|
-
|
50
|
+
super
|
51
|
+
timer_execute(:in_watch_process, @interval, &method(:on_timer))
|
53
52
|
end
|
54
53
|
|
55
54
|
def shutdown
|
56
|
-
|
55
|
+
super
|
56
|
+
end
|
57
|
+
|
58
|
+
def apply_default_types
|
59
|
+
return unless @types.nil?
|
60
|
+
@types = Fluent.windows? ? @windows_watcher.default_types : DEFAULT_TYPES
|
61
|
+
@type_converters = parse_types_parameter unless @types.nil?
|
57
62
|
end
|
58
63
|
|
59
|
-
def
|
60
|
-
|
61
|
-
|
64
|
+
def on_timer
|
65
|
+
io = IO.popen(@command, 'r')
|
66
|
+
begin
|
62
67
|
io.gets
|
63
68
|
while result = io.gets
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
tag = @tag.gsub(/(\${[a-z]+}|__[A-Z]+__)/, get_placeholder)
|
75
|
-
Engine.emit(tag, Engine.now, data)
|
69
|
+
if Fluent.windows?
|
70
|
+
data = @windows_watcher.parse_line(result)
|
71
|
+
next unless @windows_watcher.match_look_up_user?(data)
|
72
|
+
else
|
73
|
+
data = parse_line(result)
|
74
|
+
next unless match_look_up_user?(data)
|
75
|
+
end
|
76
|
+
emit_tag = tag.dup
|
77
|
+
filter_record(emit_tag, Fluent::Engine.now, data)
|
78
|
+
router.emit(emit_tag, Fluent::Engine.now, data)
|
76
79
|
end
|
80
|
+
ensure
|
77
81
|
io.close
|
78
|
-
sleep @interval
|
79
82
|
end
|
83
|
+
rescue StandardError => e
|
84
|
+
log.error "watch_process: error has occured. #{e.message}"
|
85
|
+
end
|
86
|
+
|
87
|
+
def parse_line(line)
|
88
|
+
keys_size = @keys.size
|
89
|
+
if line =~ /(?<lstart>(^\w+\s+\w+\s+\d+\s+\d\d:\d\d:\d\d \d+))/
|
90
|
+
lstart = Time.parse($~[:lstart])
|
91
|
+
line = line.sub($~[:lstart], '')
|
92
|
+
keys_size -= 1
|
93
|
+
end
|
94
|
+
values = [lstart.to_s, line.chomp.strip.split(/\s+/, keys_size)]
|
95
|
+
data = Hash[@keys.zip(values.reject(&:empty?).flatten)]
|
96
|
+
data['elapsed_time'] = (Time.now - Time.parse(data['start_time'])).to_i if data['start_time']
|
97
|
+
data
|
98
|
+
end
|
99
|
+
|
100
|
+
def match_look_up_user?(data)
|
101
|
+
return true if @lookup_user.nil?
|
102
|
+
|
103
|
+
@lookup_user.include?(data['user'])
|
80
104
|
end
|
81
105
|
|
82
106
|
def get_ps_command
|
83
|
-
if
|
84
|
-
"LANG=en_US.UTF-8 && ps -ewwo lstart,user:20,pid,ppid,time,%cpu,%mem,rss,sz,s,comm,cmd"
|
85
|
-
elsif OS.mac?
|
107
|
+
if mac?
|
86
108
|
"LANG=en_US.UTF-8 && ps -ewwo lstart,user,pid,ppid,time,%cpu,%mem,rss,vsz,state,comm,command"
|
109
|
+
elsif Fluent.windows?
|
110
|
+
@windows_watcher.command
|
111
|
+
else
|
112
|
+
"LANG=en_US.UTF-8 && ps -ewwo lstart,user:20,pid,ppid,time,%cpu,%mem,rss,sz,s,comm,cmd"
|
87
113
|
end
|
88
114
|
end
|
89
115
|
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
116
|
+
def mac?
|
117
|
+
(/darwin/ =~ RUBY_PLATFORM) != nil
|
118
|
+
end
|
119
|
+
|
120
|
+
class WindowsWatcher
|
121
|
+
# Keys are from the "System.Diagnostics.Process" object properties that can be taken by the "Get-Process" command.
|
122
|
+
# You can check the all properties by the "(Get-Process)[0] | Get-Member" command.
|
123
|
+
DEFAULT_KEYS = %w(StartTime UserName SessionId Id CPU WorkingSet VirtualMemorySize HandleCount ProcessName)
|
124
|
+
|
125
|
+
DEFAULT_TYPES = %w(
|
126
|
+
SessionId:integer
|
127
|
+
Id:integer
|
128
|
+
CPU:float
|
129
|
+
WorkingSet:integer
|
130
|
+
VirtualMemorySize:integer
|
131
|
+
HandleCount:integer
|
132
|
+
).join(",")
|
133
|
+
|
134
|
+
attr_reader :keys
|
135
|
+
attr_reader :command
|
136
|
+
|
137
|
+
def initialize(keys, command, lookup_user, powershell_command)
|
138
|
+
@keys = keys || DEFAULT_KEYS
|
139
|
+
@powershell_command = powershell_command
|
140
|
+
@command = command || default_command
|
141
|
+
@lookup_user = lookup_user
|
94
142
|
end
|
95
143
|
|
96
|
-
def
|
97
|
-
|
144
|
+
def default_types
|
145
|
+
DEFAULT_TYPES
|
98
146
|
end
|
99
147
|
|
100
|
-
def
|
101
|
-
|
148
|
+
def parse_line(line)
|
149
|
+
values = line.chomp.strip.parse_csv.map { |e| e ? e : "" }
|
150
|
+
data = Hash[@keys.zip(values)]
|
151
|
+
|
152
|
+
unless data["StartTime"].nil?
|
153
|
+
start_time = Time.parse(data['StartTime'])
|
154
|
+
data['ElapsedTime'] = (Time.now - start_time).to_i
|
155
|
+
data["StartTime"] = start_time.to_s
|
156
|
+
end
|
157
|
+
|
158
|
+
data
|
159
|
+
end
|
160
|
+
|
161
|
+
def match_look_up_user?(data)
|
162
|
+
return true if @lookup_user.nil?
|
163
|
+
|
164
|
+
@lookup_user.include?(data["UserName"])
|
102
165
|
end
|
103
166
|
|
104
|
-
def
|
105
|
-
|
167
|
+
def default_command
|
168
|
+
command = [
|
169
|
+
command_ps,
|
170
|
+
pipe_filtering_normal_ps,
|
171
|
+
pipe_select_columns,
|
172
|
+
pipe_fixing_locale,
|
173
|
+
pipe_formatting_output,
|
174
|
+
].join
|
175
|
+
"#{@powershell_command} -command \"#{command}\""
|
106
176
|
end
|
107
|
-
end
|
108
177
|
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
178
|
+
def command_ps
|
179
|
+
if @keys.include?("UserName")
|
180
|
+
# The "IncludeUserName" option is needed to get the username, but this option requires Administrator privilege.
|
181
|
+
"Get-Process -IncludeUserName"
|
182
|
+
else
|
183
|
+
"Get-Process"
|
184
|
+
end
|
185
|
+
end
|
186
|
+
|
187
|
+
private
|
188
|
+
|
189
|
+
def pipe_filtering_normal_ps
|
190
|
+
# There are some special processes that don't have some properties, such as the "Idle" process.
|
191
|
+
# Normally, these are specific to the system and are not important, so exclude them.
|
192
|
+
# Note: The same situation can occur in some processes if there are no Administrator privilege.
|
193
|
+
" | ?{$_.StartTime -ne $NULL -and $_.CPU -ne $NULL}"
|
194
|
+
end
|
195
|
+
|
196
|
+
def pipe_select_columns
|
197
|
+
if @keys.nil? || @keys.empty?
|
198
|
+
raise "The 'keys' parameter is not specified correctly. [keys: #{@keys}]"
|
199
|
+
end
|
200
|
+
|
201
|
+
" | Select-Object -Property #{@keys.join(',')}"
|
202
|
+
end
|
203
|
+
|
204
|
+
def pipe_fixing_locale()
|
205
|
+
# In Windows, setting the "$env:Lang" environment variable is not effective in changing the format of the output.
|
206
|
+
# You can use "Datetime.ToString" method to change format of datetime values in for-each pipe.
|
207
|
+
# Note: About "DateTime.ToString" method: https://docs.microsoft.com/en-us/dotnet/api/system.datetime.tostring
|
208
|
+
return "" unless @keys.include?("StartTime")
|
209
|
+
|
210
|
+
" | %{$_.StartTime = $_.StartTime.ToString('o'); return $_;}"
|
211
|
+
end
|
212
|
+
|
213
|
+
def pipe_formatting_output
|
214
|
+
# In the "ConvertTo-Csv" command, there are 2 lines of type info and header info at the beginning in the outputs.
|
215
|
+
# By the "NoTypeInformation" option, the line of type info is excluded.
|
216
|
+
# This enables you to skip the just first line for parsing, like linux or mac.
|
217
|
+
" | ConvertTo-Csv -NoTypeInformation"
|
218
|
+
end
|
219
|
+
end
|
115
220
|
end
|
116
221
|
end
|
data/test/helper.rb
CHANGED
@@ -12,15 +12,7 @@ require 'test/unit'
|
|
12
12
|
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
|
13
13
|
$LOAD_PATH.unshift(File.dirname(__FILE__))
|
14
14
|
require 'fluent/test'
|
15
|
-
|
16
|
-
nulllogger = Object.new
|
17
|
-
nulllogger.instance_eval {|obj|
|
18
|
-
def method_missing(method, *args)
|
19
|
-
# pass
|
20
|
-
end
|
21
|
-
}
|
22
|
-
$log = nulllogger
|
23
|
-
end
|
15
|
+
require 'fluent/test/driver/input'
|
24
16
|
|
25
17
|
require 'fluent/plugin/in_watch_process'
|
26
18
|
|
@@ -7,11 +7,10 @@ class WatchProcessInputTest < Test::Unit::TestCase
|
|
7
7
|
|
8
8
|
CONFIG = %[
|
9
9
|
tag input.watch_process
|
10
|
-
lookup_user apache, mycron
|
11
10
|
]
|
12
11
|
|
13
|
-
def create_driver(conf=CONFIG
|
14
|
-
Fluent::Test::
|
12
|
+
def create_driver(conf = CONFIG)
|
13
|
+
Fluent::Test::Driver::Input.new(Fluent::Plugin::WatchProcessInput).configure(conf)
|
15
14
|
end
|
16
15
|
|
17
16
|
def test_configure
|
@@ -22,9 +21,105 @@ class WatchProcessInputTest < Test::Unit::TestCase
|
|
22
21
|
tag input.watch_process
|
23
22
|
lookup_user apache, mycron
|
24
23
|
]
|
25
|
-
d.instance.inspect
|
26
24
|
assert_equal 'input.watch_process', d.instance.tag
|
27
25
|
assert_equal ['apache', 'mycron'], d.instance.lookup_user
|
26
|
+
assert_equal :powershell, d.instance.powershell_command
|
28
27
|
end
|
29
|
-
end
|
30
28
|
|
29
|
+
def test_windows_configure
|
30
|
+
omit "Only for UNIX like." unless Fluent.windows?
|
31
|
+
d = create_driver %[
|
32
|
+
tag input.watch_process
|
33
|
+
lookup_user apache, mycron
|
34
|
+
powershell_command pwsh
|
35
|
+
]
|
36
|
+
assert_equal 'input.watch_process', d.instance.tag
|
37
|
+
assert_equal ['apache', 'mycron'], d.instance.lookup_user
|
38
|
+
assert_equal :pwsh, d.instance.powershell_command
|
39
|
+
end
|
40
|
+
|
41
|
+
def test_unixlike
|
42
|
+
omit "Only for UNIX like." if Fluent.windows?
|
43
|
+
whoami = `whoami`
|
44
|
+
d = create_driver %[
|
45
|
+
tag input.watch_process
|
46
|
+
lookup_user #{whoami}
|
47
|
+
interval 1s
|
48
|
+
]
|
49
|
+
d.run(expect_emits: 1, timeout: 3)
|
50
|
+
assert(d.events.size > 1)
|
51
|
+
end
|
52
|
+
|
53
|
+
sub_test_case "Windows" do
|
54
|
+
def test_windows_default
|
55
|
+
omit "Only for Windows." unless Fluent.windows?
|
56
|
+
d = create_driver %[
|
57
|
+
tag input.watch_process
|
58
|
+
interval 1s
|
59
|
+
]
|
60
|
+
default_keys = Fluent::Plugin::WatchProcessInput::WindowsWatcher::DEFAULT_KEYS + ["ElapsedTime"]
|
61
|
+
|
62
|
+
d.run(expect_records: 1, timeout: 10);
|
63
|
+
|
64
|
+
assert d.events.size > 0
|
65
|
+
|
66
|
+
tag, time, record = d.events[0]
|
67
|
+
|
68
|
+
assert_equal "input.watch_process", tag
|
69
|
+
assert time.is_a?(Fluent::EventTime)
|
70
|
+
assert_equal default_keys, record.keys
|
71
|
+
end
|
72
|
+
|
73
|
+
def test_windows_customized
|
74
|
+
omit "Only for Windows." unless Fluent.windows?
|
75
|
+
custom_keys = ["PM", "Id", "UserProcessorTime", "Path"]
|
76
|
+
d = create_driver %[
|
77
|
+
tag input.watch_process
|
78
|
+
interval 1s
|
79
|
+
keys #{custom_keys.join(",")}
|
80
|
+
types Id:integer
|
81
|
+
]
|
82
|
+
|
83
|
+
d.run(expect_records: 1, timeout: 10);
|
84
|
+
|
85
|
+
assert d.events.size > 0
|
86
|
+
|
87
|
+
tag, time, record = d.events[0]
|
88
|
+
|
89
|
+
assert_equal "input.watch_process", tag
|
90
|
+
assert time.is_a?(Fluent::EventTime)
|
91
|
+
assert_equal custom_keys, record.keys
|
92
|
+
assert record["PM"].is_a?(String)
|
93
|
+
assert record["Id"].is_a?(Integer)
|
94
|
+
end
|
95
|
+
|
96
|
+
def test_windows_lookup
|
97
|
+
omit "Only for Windows." unless Fluent.windows?
|
98
|
+
d = create_driver %[
|
99
|
+
tag input.watch_process
|
100
|
+
interval 1s
|
101
|
+
]
|
102
|
+
d.run(expect_records: 1, timeout: 10);
|
103
|
+
|
104
|
+
assert d.events.size > 0
|
105
|
+
|
106
|
+
tag, time, record = d.events[0]
|
107
|
+
lookup_user = record["UserName"]
|
108
|
+
|
109
|
+
d = create_driver %[
|
110
|
+
tag input.watch_process
|
111
|
+
interval 1s
|
112
|
+
lookup_user #{lookup_user}
|
113
|
+
]
|
114
|
+
d.run(expect_records: 1, timeout: 10);
|
115
|
+
|
116
|
+
assert d.events.size > 0
|
117
|
+
|
118
|
+
other_user_records = d.events.reject do |tag, time, record|
|
119
|
+
lookup_user.include?(record["UserName"])
|
120
|
+
end
|
121
|
+
|
122
|
+
assert other_user_records.size == 0
|
123
|
+
end
|
124
|
+
end
|
125
|
+
end
|
metadata
CHANGED
@@ -1,48 +1,105 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-watch-process
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0
|
5
|
-
prerelease:
|
4
|
+
version: 0.2.0
|
6
5
|
platform: ruby
|
7
6
|
authors:
|
8
7
|
- Kentaro Yoshida
|
9
8
|
autorequire:
|
10
9
|
bindir: bin
|
11
10
|
cert_chain: []
|
12
|
-
date:
|
11
|
+
date: 2021-05-30 00:00:00.000000000 Z
|
13
12
|
dependencies:
|
14
13
|
- !ruby/object:Gem::Dependency
|
15
14
|
name: rake
|
16
15
|
requirement: !ruby/object:Gem::Requirement
|
17
|
-
none: false
|
18
16
|
requirements:
|
19
|
-
- -
|
17
|
+
- - ">="
|
20
18
|
- !ruby/object:Gem::Version
|
21
19
|
version: '0'
|
22
20
|
type: :development
|
23
21
|
prerelease: false
|
24
22
|
version_requirements: !ruby/object:Gem::Requirement
|
25
|
-
none: false
|
26
23
|
requirements:
|
27
|
-
- -
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '0'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: test-unit
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: 3.1.0
|
34
|
+
type: :development
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: 3.1.0
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: appraisal
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
28
53
|
- !ruby/object:Gem::Version
|
29
54
|
version: '0'
|
30
55
|
- !ruby/object:Gem::Dependency
|
31
56
|
name: fluentd
|
32
57
|
requirement: !ruby/object:Gem::Requirement
|
33
|
-
none: false
|
34
58
|
requirements:
|
35
|
-
- -
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: 0.14.0
|
62
|
+
- - "<"
|
63
|
+
- !ruby/object:Gem::Version
|
64
|
+
version: '2'
|
65
|
+
type: :runtime
|
66
|
+
prerelease: false
|
67
|
+
version_requirements: !ruby/object:Gem::Requirement
|
68
|
+
requirements:
|
69
|
+
- - ">="
|
70
|
+
- !ruby/object:Gem::Version
|
71
|
+
version: 0.14.0
|
72
|
+
- - "<"
|
73
|
+
- !ruby/object:Gem::Version
|
74
|
+
version: '2'
|
75
|
+
- !ruby/object:Gem::Dependency
|
76
|
+
name: fluent-mixin-rewrite-tag-name
|
77
|
+
requirement: !ruby/object:Gem::Requirement
|
78
|
+
requirements:
|
79
|
+
- - ">="
|
36
80
|
- !ruby/object:Gem::Version
|
37
81
|
version: '0'
|
38
82
|
type: :runtime
|
39
83
|
prerelease: false
|
40
84
|
version_requirements: !ruby/object:Gem::Requirement
|
41
|
-
none: false
|
42
85
|
requirements:
|
43
|
-
- -
|
86
|
+
- - ">="
|
44
87
|
- !ruby/object:Gem::Version
|
45
88
|
version: '0'
|
89
|
+
- !ruby/object:Gem::Dependency
|
90
|
+
name: fluent-mixin-type-converter
|
91
|
+
requirement: !ruby/object:Gem::Requirement
|
92
|
+
requirements:
|
93
|
+
- - ">="
|
94
|
+
- !ruby/object:Gem::Version
|
95
|
+
version: 0.1.0
|
96
|
+
type: :runtime
|
97
|
+
prerelease: false
|
98
|
+
version_requirements: !ruby/object:Gem::Requirement
|
99
|
+
requirements:
|
100
|
+
- - ">="
|
101
|
+
- !ruby/object:Gem::Version
|
102
|
+
version: 0.1.0
|
46
103
|
description:
|
47
104
|
email:
|
48
105
|
- y.ken.studio@gmail.com
|
@@ -50,8 +107,9 @@ executables: []
|
|
50
107
|
extensions: []
|
51
108
|
extra_rdoc_files: []
|
52
109
|
files:
|
53
|
-
- .gitignore
|
54
|
-
- .travis.yml
|
110
|
+
- ".gitignore"
|
111
|
+
- ".travis.yml"
|
112
|
+
- Appraisals
|
55
113
|
- Gemfile
|
56
114
|
- LICENSE.txt
|
57
115
|
- README.md
|
@@ -59,33 +117,34 @@ files:
|
|
59
117
|
- example1.conf
|
60
118
|
- example2.conf
|
61
119
|
- fluent-plugin-watch-process.gemspec
|
120
|
+
- gemfiles/fluentd_v0.14.gemfile
|
62
121
|
- lib/fluent/plugin/in_watch_process.rb
|
63
122
|
- test/helper.rb
|
64
123
|
- test/plugin/test_in_watch_process.rb
|
65
124
|
homepage: https://github.com/y-ken/fluent-plugin-watch-process
|
66
125
|
licenses: []
|
126
|
+
metadata: {}
|
67
127
|
post_install_message:
|
68
128
|
rdoc_options: []
|
69
129
|
require_paths:
|
70
130
|
- lib
|
71
131
|
required_ruby_version: !ruby/object:Gem::Requirement
|
72
|
-
none: false
|
73
132
|
requirements:
|
74
|
-
- -
|
133
|
+
- - ">="
|
75
134
|
- !ruby/object:Gem::Version
|
76
135
|
version: '0'
|
77
136
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
78
|
-
none: false
|
79
137
|
requirements:
|
80
|
-
- -
|
138
|
+
- - ">="
|
81
139
|
- !ruby/object:Gem::Version
|
82
140
|
version: '0'
|
83
141
|
requirements: []
|
84
|
-
|
85
|
-
rubygems_version: 1.8.23
|
142
|
+
rubygems_version: 3.1.6
|
86
143
|
signing_key:
|
87
|
-
specification_version:
|
88
|
-
summary: Fluentd Input plugin to collect process information via ps command
|
144
|
+
specification_version: 4
|
145
|
+
summary: Fluentd Input plugin to collect continual process information via ps command
|
146
|
+
or PowerShell pwsh command for Linux/osx/Windows. It is useful for cron/barch process
|
147
|
+
monitoring.
|
89
148
|
test_files:
|
90
149
|
- test/helper.rb
|
91
150
|
- test/plugin/test_in_watch_process.rb
|