ext_batch 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.gitignore +9 -0
- data/.ruby-gemset +1 -0
- data/.ruby-version +1 -0
- data/Gemfile +5 -0
- data/LICENSE.txt +21 -0
- data/README.md +83 -0
- data/Rakefile +2 -0
- data/bin/console +14 -0
- data/bin/setup +8 -0
- data/ext_batch.gemspec +39 -0
- data/lib/ext_batch/common.rb +6 -0
- data/lib/ext_batch/main.rb +437 -0
- data/lib/ext_batch/version.rb +3 -0
- data/lib/ext_batch.rb +7 -0
- data/lib/tasks/ext_batch.rake +30 -0
- data/lib/templates/workers/ext_batch_monitor.rb +34 -0
- metadata +132 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 161f89b793ce8e0c47982974ae4ce6a871ebbb34
|
4
|
+
data.tar.gz: bafb5ea1e0f452fbc4e69a95e1b67a7a3e90d24e
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 2803a11495151399f2fc7b50d7a4a22c9a925a5549c892f5489a9d39f45f540d55b1b220c7a41f5ec84bc7097895facc1466c6fabe21f4617c3c2abb30d757ec
|
7
|
+
data.tar.gz: fcaa5df84a98c00f184a925da462a1a9c567e7582f3775eeacbcc08e054bc8d249d5a11b4fd2de1e21efee44db872246c96ac13c1b52aa069ee88860bcd05117
|
data/.gitignore
ADDED
data/.ruby-gemset
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
gem_ext_batch
|
data/.ruby-version
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
2.3.1
|
data/Gemfile
ADDED
data/LICENSE.txt
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2018 Ruby
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
13
|
+
all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,83 @@
|
|
1
|
+
# ExtBatch
|
2
|
+
|
3
|
+
Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/ext_batch`. To experiment with that code, run `bin/console` for an interactive prompt.
|
4
|
+
|
5
|
+
```explain
|
6
|
+
Extended the ext_logger gem package, services for Ruby batch and Ruby rake.
|
7
|
+
Other extensions see version updates.
|
8
|
+
2019-01-20(0.1.0):
|
9
|
+
dependency: ext_logger ~> v0.2
|
10
|
+
|
11
|
+
|
12
|
+
```
|
13
|
+
|
14
|
+
## Installation
|
15
|
+
|
16
|
+
Add this line to your application's Gemfile:
|
17
|
+
|
18
|
+
```ruby
|
19
|
+
gem 'ext_batch'
|
20
|
+
```
|
21
|
+
|
22
|
+
And then execute:
|
23
|
+
|
24
|
+
$ bundle
|
25
|
+
|
26
|
+
Or install it yourself as:
|
27
|
+
|
28
|
+
$ gem install ext_batch
|
29
|
+
|
30
|
+
## Usage
|
31
|
+
|
32
|
+
1. Initialization object (It automatically creates non-existent directories based on paths):
|
33
|
+
|
34
|
+
```ruby
|
35
|
+
batch = ExtBatch.new(task_name, {log: {age: 7/'weekly', size: 1024000, split_format: '%Y%m%d%H%M', log_max_num: 5}})
|
36
|
+
# task_name: task object or task name string, if there is ":", it will automatically change to "/", and finally generate the log directory.
|
37
|
+
```
|
38
|
+
|
39
|
+
2. Output log (Extended ext_logger.info):
|
40
|
+
|
41
|
+
```ruby
|
42
|
+
batch.info("log content")
|
43
|
+
batch.debug("log content")
|
44
|
+
batch.error("log content")
|
45
|
+
batch.warn("log content")
|
46
|
+
batch.fatal("log content")
|
47
|
+
batch.exception(e, {send_mail: ExtBatch::SEND_MAIL_NO/ExtBatch::SEND_MAIL_YES}) # when rescure Exception => e, send_mail default ExtBatch::SEND_MAIL_NO
|
48
|
+
```
|
49
|
+
|
50
|
+
3. Send mail:
|
51
|
+
|
52
|
+
```ruby
|
53
|
+
batch.mail_ok({content: "mail content"})
|
54
|
+
batch.mail_error({content: "mail content"})
|
55
|
+
batch.mail_warn({content: "mail content"})
|
56
|
+
batch.mail_exception(e, {content: "mail content"}) # when rescure Exception => e
|
57
|
+
```
|
58
|
+
|
59
|
+
4. Real-time display time:
|
60
|
+
|
61
|
+
```ruby
|
62
|
+
batch.show_use_time
|
63
|
+
```
|
64
|
+
|
65
|
+
5. Batch termination processing:
|
66
|
+
|
67
|
+
```ruby
|
68
|
+
batch.end
|
69
|
+
```
|
70
|
+
|
71
|
+
## Development
|
72
|
+
|
73
|
+
After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
74
|
+
|
75
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
76
|
+
|
77
|
+
## Contributing
|
78
|
+
|
79
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/ext_batch.
|
80
|
+
|
81
|
+
## License
|
82
|
+
|
83
|
+
The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
|
data/Rakefile
ADDED
data/bin/console
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require "bundler/setup"
|
4
|
+
require "ext_batch"
|
5
|
+
|
6
|
+
# You can add fixtures and/or initialization code here to make experimenting
|
7
|
+
# with your gem easier. You can also use a different console, if you like.
|
8
|
+
|
9
|
+
# (If you use this, don't forget to add pry to your Gemfile!)
|
10
|
+
# require "pry"
|
11
|
+
# Pry.start
|
12
|
+
|
13
|
+
require "irb"
|
14
|
+
IRB.start(__FILE__)
|
data/bin/setup
ADDED
data/ext_batch.gemspec
ADDED
@@ -0,0 +1,39 @@
|
|
1
|
+
# coding: utf-8
|
2
|
+
lib = File.expand_path("../lib", __FILE__)
|
3
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
+
require "ext_batch/version"
|
5
|
+
|
6
|
+
Gem::Specification.new do |spec|
|
7
|
+
spec.name = "ext_batch"
|
8
|
+
spec.version = ExtBatch::VERSION
|
9
|
+
spec.authors = ["Ruby"]
|
10
|
+
spec.email = ["wu.chao@maisiteng.com"]
|
11
|
+
|
12
|
+
spec.summary = %q{Depending on the ext_logger gem package, create for batch. }
|
13
|
+
spec.description = %q{Depending on the ext_logger gem package, create for batch. The batch resource consumption can be recorded, and the specified mail can be sent according to the implementation of batch, and the mail format is fixed.}
|
14
|
+
spec.homepage = ""
|
15
|
+
spec.license = "MIT"
|
16
|
+
|
17
|
+
# # Prevent pushing this gem to RubyGems.org. To allow pushes either set the 'allowed_push_host'
|
18
|
+
# # to allow pushing to a single host or delete this section to allow pushing to any host.
|
19
|
+
# if spec.respond_to?(:metadata)
|
20
|
+
# spec.metadata["allowed_push_host"] = "TODO: Set to 'http://mygemserver.com'"
|
21
|
+
# else
|
22
|
+
# raise "RubyGems 2.0 or newer is required to protect against " \
|
23
|
+
# "public gem pushes."
|
24
|
+
# end
|
25
|
+
|
26
|
+
spec.files = `git ls-files -z`.split("\x0").reject do |f|
|
27
|
+
f.match(%r{^(test|spec|features)/})
|
28
|
+
end
|
29
|
+
spec.bindir = "exe"
|
30
|
+
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
31
|
+
spec.require_paths = ["lib"]
|
32
|
+
|
33
|
+
spec.add_development_dependency "bundler", "~> 1.15"
|
34
|
+
spec.add_development_dependency "rake", "~> 10.0"
|
35
|
+
|
36
|
+
spec.add_dependency "mail", "~> 2.5"
|
37
|
+
spec.add_dependency "sys-proctable", "~> 1.2"
|
38
|
+
spec.add_dependency "ext_logger", "~> 0.2"
|
39
|
+
end
|
@@ -0,0 +1,437 @@
|
|
1
|
+
|
2
|
+
# require 'rails'
|
3
|
+
require 'ext_logger'
|
4
|
+
# require 'ext_batch/common'
|
5
|
+
require 'sys/proctable' if !defined? ProcTable
|
6
|
+
|
7
|
+
class ExtBatch
|
8
|
+
include Sys
|
9
|
+
include ExtLogger::Common
|
10
|
+
# include Common
|
11
|
+
|
12
|
+
LABEL_OK = 'ok'
|
13
|
+
LABEL_ERROR = 'error'
|
14
|
+
LABEL_WARN = 'warning'
|
15
|
+
|
16
|
+
SEND_MAIL_NO = 'no'
|
17
|
+
SEND_MAIL_YES = 'yes'
|
18
|
+
|
19
|
+
# Initialize, next is parameters:
|
20
|
+
# 1rst: task object or batch name(string) (Default parameter)
|
21
|
+
# 2end: option (Influencing log size, file name format, etc.)
|
22
|
+
# :log -- Inheriting ExtLogger classes, hash class, value exp:
|
23
|
+
# {
|
24
|
+
# :age -- Split file with N days
|
25
|
+
# :size -- Split file with file size
|
26
|
+
# :split_format -- Split file format. exp: '%Y....'
|
27
|
+
# :log_max_num -- Number of log files saved
|
28
|
+
# :debug -- Is debug
|
29
|
+
# }
|
30
|
+
def initialize(*args)
|
31
|
+
func_name = "[#{get_class_name}.#{__method__.to_s}]"
|
32
|
+
output_debug "#{func_name} args.size: #{args.size}"
|
33
|
+
|
34
|
+
# Instance variables
|
35
|
+
@task_name = ''
|
36
|
+
@ext_logger = nil
|
37
|
+
@batch_log_name = ''
|
38
|
+
@batch_message ||= ''
|
39
|
+
@start_time ||= Time.now
|
40
|
+
@last_time ||= @start_time
|
41
|
+
|
42
|
+
@pid_file = {
|
43
|
+
path: '',
|
44
|
+
name: ''
|
45
|
+
}
|
46
|
+
|
47
|
+
init(args[0]) if args.size == 1
|
48
|
+
|
49
|
+
if args.size == 2
|
50
|
+
opt = args[1].clone
|
51
|
+
if opt.is_a?(Hash)
|
52
|
+
init(args[0], opt)
|
53
|
+
else
|
54
|
+
init(args[0])
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
# Get the message
|
60
|
+
def get_message
|
61
|
+
return @batch_message
|
62
|
+
end
|
63
|
+
|
64
|
+
# Set the message
|
65
|
+
def set_message(str=nil)
|
66
|
+
@batch_message = str if !is_blank?(str)
|
67
|
+
end
|
68
|
+
|
69
|
+
# Record start time
|
70
|
+
def start_time
|
71
|
+
return @start_time
|
72
|
+
end
|
73
|
+
|
74
|
+
# Real-time display of usage time
|
75
|
+
def show_use_time
|
76
|
+
func_name = "[#{get_instance_name}.#{__method__.to_s}]"
|
77
|
+
now_time = Time.now
|
78
|
+
|
79
|
+
output_debug "#{func_name} @start_time:#{@start_time}, @last_time:#{@last_time}, now_time:#{now_time}"
|
80
|
+
|
81
|
+
use_time = format_sub_time(@last_time, now_time)
|
82
|
+
message = "Use time: #{use_time}#{get_total_time(@start_time, now_time)}"
|
83
|
+
@ext_logger.info message
|
84
|
+
|
85
|
+
@last_time = now_time
|
86
|
+
end
|
87
|
+
|
88
|
+
# batch end
|
89
|
+
def end(opt={})
|
90
|
+
func_name = "[#{get_instance_name}.end]"
|
91
|
+
message = "#{@task_name} END."
|
92
|
+
# display usage time
|
93
|
+
# opt[:show_use_time] = true if opt[:show_use_time] != false
|
94
|
+
message += get_total_time(@start_time) #if opt[:show_use_time]
|
95
|
+
|
96
|
+
delete_pid_file
|
97
|
+
@ext_logger.info message
|
98
|
+
@ext_logger.record_memory('end') if defined?(@ext_logger.record_memory)
|
99
|
+
end
|
100
|
+
|
101
|
+
# Inheriting ExtLogger instance method: info
|
102
|
+
def info(*args)
|
103
|
+
@ext_logger.info(*args)
|
104
|
+
end
|
105
|
+
|
106
|
+
# Inheriting ExtLogger instance method: info
|
107
|
+
def debug(*args)
|
108
|
+
@ext_logger.debug(*args)
|
109
|
+
end
|
110
|
+
|
111
|
+
# Inheriting ExtLogger instance method: info
|
112
|
+
def error(*args)
|
113
|
+
@ext_logger.error(*args)
|
114
|
+
end
|
115
|
+
|
116
|
+
# Inheriting ExtLogger instance method: info
|
117
|
+
def warn(*args)
|
118
|
+
@ext_logger.warn(*args)
|
119
|
+
end
|
120
|
+
|
121
|
+
# Inheriting ExtLogger instance method: info
|
122
|
+
def fatal(*args)
|
123
|
+
@ext_logger.fatal(*args)
|
124
|
+
end
|
125
|
+
|
126
|
+
# Inheriting ExtLogger instance method: info
|
127
|
+
def exception(e, opt={})
|
128
|
+
opt[:send_mail] ||= SEND_MAIL_NO
|
129
|
+
opt[:status] = LABEL_ERROR
|
130
|
+
content = []
|
131
|
+
content << "#{@log_name} #{opt[:status]} (#{Time.now})"
|
132
|
+
content << analysis_exception(e)
|
133
|
+
content << opt[:content] if !is_blank?(opt[:content])
|
134
|
+
|
135
|
+
case opt[:send_mail]
|
136
|
+
when SEND_MAIL_YES
|
137
|
+
mail_error({content: content.join("\n")})
|
138
|
+
end
|
139
|
+
error(content)
|
140
|
+
end
|
141
|
+
|
142
|
+
# batch: send ok mail
|
143
|
+
def mail_ok(opt={})
|
144
|
+
func_name = "[#{get_instance_name}.#{__method__.to_s}]"
|
145
|
+
opt[:content] ||= ''
|
146
|
+
opt[:status] = LABEL_OK
|
147
|
+
return mail_tips(@log_name, opt)
|
148
|
+
end
|
149
|
+
|
150
|
+
# batch: send error mail
|
151
|
+
def mail_error(opt={})
|
152
|
+
func_name = "[#{get_instance_name}.#{__method__.to_s}]"
|
153
|
+
opt[:content] ||= ''
|
154
|
+
opt[:status] = LABEL_ERROR
|
155
|
+
return mail_tips(@log_name, opt)
|
156
|
+
end
|
157
|
+
|
158
|
+
# batch: send warning mail
|
159
|
+
def mail_warn(opt={})
|
160
|
+
func_name = "[#{get_instance_name}.#{__method__.to_s}]"
|
161
|
+
opt[:content] ||= ''
|
162
|
+
opt[:status] = LABEL_WARN
|
163
|
+
return mail_tips(@log_name, opt)
|
164
|
+
end
|
165
|
+
|
166
|
+
# batch: send exception mail
|
167
|
+
def mail_exception(e, opt={})
|
168
|
+
func_name = "[#{get_instance_name}.#{__method__.to_s}]"
|
169
|
+
opt[:status] = LABEL_ERROR
|
170
|
+
content = []
|
171
|
+
content << "#{@log_name} #{opt[:status]} (#{Time.now})"
|
172
|
+
content << analysis_exception(e)
|
173
|
+
content << opt[:content] if !is_blank?(opt[:content])
|
174
|
+
opt[:content] = content.join("\n")
|
175
|
+
|
176
|
+
return mail_tips(@log_name, opt)
|
177
|
+
end
|
178
|
+
|
179
|
+
# Set file path for batch process pid
|
180
|
+
def set_pid_file_path(str)
|
181
|
+
@pid_file[:path] = str if str.is_a?(String) && !is_blank?(str)
|
182
|
+
end
|
183
|
+
|
184
|
+
# Get file path for batch process pid
|
185
|
+
def get_pid_file_path
|
186
|
+
return @pid_file[:path]
|
187
|
+
end
|
188
|
+
|
189
|
+
# Set file name for batch process pid
|
190
|
+
def set_pid_file_name(str)
|
191
|
+
@pid_file[:name] = str if str.is_a?(String) && !is_blank?(str)
|
192
|
+
end
|
193
|
+
|
194
|
+
# Get file name for batch process pid
|
195
|
+
def get_pid_file_name
|
196
|
+
return @pid_file[:name]
|
197
|
+
end
|
198
|
+
|
199
|
+
# Get test for pid
|
200
|
+
def get_pid_test(opt={})
|
201
|
+
|
202
|
+
# Everything
|
203
|
+
# ProcTable.ps{ |p|
|
204
|
+
# puts p.pid.to_s
|
205
|
+
# puts p.comm
|
206
|
+
# # ...
|
207
|
+
# }
|
208
|
+
|
209
|
+
# Just one process
|
210
|
+
# s = ProcTable.ps(pid: 2123)
|
211
|
+
# puts s.pid.to_s
|
212
|
+
# puts s.comm
|
213
|
+
# ...
|
214
|
+
|
215
|
+
# Return the results as an array of ProcTableStructs
|
216
|
+
puts "@task_name ### #{@task_name}"
|
217
|
+
|
218
|
+
task_name = 'irb'
|
219
|
+
pid = ''
|
220
|
+
a = ProcTable.ps
|
221
|
+
a.each_with_index do |p, idx|
|
222
|
+
if p.cmdline.include?(task_name)
|
223
|
+
@ext_logger.info "pid:#{p.pid}, cmd:#{p.comm}, cmdline: #{p.cmdline}"
|
224
|
+
pid = p.pid
|
225
|
+
break
|
226
|
+
end
|
227
|
+
# puts "pid:#{p.pid}, cmd:#{p.comm}, cmdline: #{p.cmdline}" if p.cmdline.include?(@task_name)
|
228
|
+
# puts "p.class: #{p.class}"
|
229
|
+
# puts p.to_s
|
230
|
+
# break if idx >= 10
|
231
|
+
# ...
|
232
|
+
end
|
233
|
+
|
234
|
+
puts "#{task_name}.pid: #{pid}"
|
235
|
+
|
236
|
+
File.new()
|
237
|
+
end
|
238
|
+
|
239
|
+
private
|
240
|
+
# Private function for initialize
|
241
|
+
# batch initialize
|
242
|
+
def init(task, opt={})
|
243
|
+
require 'rake'
|
244
|
+
func_name = "[#{get_class_name}:private #{__method__.to_s}]"
|
245
|
+
set_debug(true) if opt[:debug]
|
246
|
+
|
247
|
+
output_debug "#{func_name} task_name: #{task}, opt: #{opt}"
|
248
|
+
|
249
|
+
# Get log name & task name
|
250
|
+
if task.is_a?(Rake::Task)
|
251
|
+
@log_name = task.name
|
252
|
+
|
253
|
+
elsif task.is_a?(String)
|
254
|
+
@log_name = task
|
255
|
+
|
256
|
+
else
|
257
|
+
@log_name = "ext_batch_log_#{Time.now.to_i}"
|
258
|
+
end
|
259
|
+
@task_name = @log_name.clone
|
260
|
+
@batch_message = @log_name.clone
|
261
|
+
|
262
|
+
# Format log path & name
|
263
|
+
@log_name = @log_name.gsub(':', '/')
|
264
|
+
while @log_name.include?('//')
|
265
|
+
@log_name.gsub!('//', '/')
|
266
|
+
end
|
267
|
+
if @log_name[0] == '/'
|
268
|
+
@log_name = @log_name[1, @log_name.size - 1]
|
269
|
+
end
|
270
|
+
|
271
|
+
# Full log file name
|
272
|
+
log_path = rails_root + '/log/' + @log_name + '.log'
|
273
|
+
mkdir_more(log_path)
|
274
|
+
output_debug "#{func_name} Current log path: #{log_path}"
|
275
|
+
|
276
|
+
opt[:log] ||= {}
|
277
|
+
opt[:log][:debug] = get_debug
|
278
|
+
@ext_logger = ExtLogger.new(log_path, log: opt[:log])
|
279
|
+
@ext_logger.record_memory()
|
280
|
+
@batch_message = ''
|
281
|
+
|
282
|
+
if task.nil?
|
283
|
+
@ext_logger.error "#{func_name} Object Error! [task] = nil"
|
284
|
+
return
|
285
|
+
|
286
|
+
else
|
287
|
+
if task.class != String
|
288
|
+
if is_blank?(task.name)
|
289
|
+
@ext_logger.error "#{func_name} Object Error! [task] = #{task.to_json}"
|
290
|
+
return
|
291
|
+
end
|
292
|
+
end
|
293
|
+
end
|
294
|
+
|
295
|
+
# # Additional ENV parameters to message
|
296
|
+
# if !is_blank?(opt[:env]) && opt[:env].class == Array
|
297
|
+
# args = []
|
298
|
+
# opt[:env].each do |item|
|
299
|
+
# args << "#{item}=#{ENV[item]}" if !is_blank?(ENV[item])
|
300
|
+
# end
|
301
|
+
# @batch_message += "[#{args.join(',')}]" if args.size > 0
|
302
|
+
# end
|
303
|
+
|
304
|
+
# Additional content to message
|
305
|
+
if opt[:message].is_a?(String) && !is_blank?(opt[:message])
|
306
|
+
@batch_message += opt[:message]
|
307
|
+
end
|
308
|
+
|
309
|
+
@pid_file = {
|
310
|
+
path: rails_root + '/tmp/batch_pids/', # Prevent duplication after file sharing by multiple servers
|
311
|
+
name: @task_name
|
312
|
+
}
|
313
|
+
create_pid_file
|
314
|
+
|
315
|
+
message = "(#{rails_env}) #{@task_name} START"
|
316
|
+
@ext_logger.info message
|
317
|
+
end
|
318
|
+
|
319
|
+
# batch: send tips mail
|
320
|
+
def mail_tips(task_name, opt={})
|
321
|
+
func_name = "[#{get_class_name}:private #{__method__.to_s}]"
|
322
|
+
opt[:content] ||= ''
|
323
|
+
_status = opt[:status] || LABEL_OK
|
324
|
+
|
325
|
+
result = []
|
326
|
+
result << "System time: #{Time.now}, #{get_total_time(@start_time)}"
|
327
|
+
result << ''
|
328
|
+
result << "BATCH: #{task_name}"
|
329
|
+
result << "STATUS: #{_status}"
|
330
|
+
result << ''
|
331
|
+
result << "NOTE: #{opt[:content]}" if !is_blank?(opt[:content])
|
332
|
+
|
333
|
+
message_body = result.join("\n")
|
334
|
+
# info message_body
|
335
|
+
|
336
|
+
_option = {
|
337
|
+
title: "BATCH #{_status}: #{task_name} (#{Time.now})",
|
338
|
+
body: result.collect{|t| t.gsub('\n', '<br />')}.join("<br />"),
|
339
|
+
status: _status
|
340
|
+
}
|
341
|
+
|
342
|
+
_option = _option.merge({body: message_body})
|
343
|
+
message = "### Next is detail of mail ###\n"
|
344
|
+
message += "[TITLE]: #{_option[:title]}\n"
|
345
|
+
message += "[BODY]:\n #{_option[:body]}\n"
|
346
|
+
|
347
|
+
case _status
|
348
|
+
when LABEL_ERROR
|
349
|
+
@ext_logger.error message
|
350
|
+
when LABEL_OK
|
351
|
+
@ext_logger.info message
|
352
|
+
when LABEL_WARN
|
353
|
+
@ext_logger.warn message
|
354
|
+
else
|
355
|
+
@ext_logger.info message
|
356
|
+
end
|
357
|
+
|
358
|
+
if defined? ActionMailer
|
359
|
+
mail_flag = true
|
360
|
+
if is_blank?(opt[:to]) && is_blank?(opt[:from]) && is_blank?(opt[:title]) && is_blank?(opt[:body])
|
361
|
+
@ext_logger.error "#{func_name} Mail delivery failed, parameters error. opt: #{opt}"
|
362
|
+
mail_flag = false
|
363
|
+
end
|
364
|
+
|
365
|
+
if mail_flag
|
366
|
+
mail_option = {
|
367
|
+
to: opt[:to],
|
368
|
+
subject: opt[:title],
|
369
|
+
content_type: 'text/html'
|
370
|
+
}
|
371
|
+
# logger.info "set_mail", mail_hash
|
372
|
+
mail_option = mail_option.merge(:from => opt[:from]) if !opt[:from].empty?
|
373
|
+
mail_option = mail_option.merge(:body => opt[:body]) if !opt[:body].empty?
|
374
|
+
mail_option = mail_option.merge(:template_path => opt[:template_path]) if !opt[:template_path].empty?
|
375
|
+
mail_option = mail_option.merge(:template_name => opt[:template_name]) if !opt[:template_name].empty?
|
376
|
+
|
377
|
+
mail(mail_option)
|
378
|
+
end
|
379
|
+
else
|
380
|
+
@ext_logger.warn "#{func_name} Mail delivery failed, not found ActionMailer class."
|
381
|
+
end
|
382
|
+
end
|
383
|
+
|
384
|
+
# Create batch process pid file
|
385
|
+
def create_pid_file
|
386
|
+
func_name = "[#{get_class_name}:private #{__method__.to_s}]"
|
387
|
+
if !is_blank?(@pid_file[:name]) && !is_blank?(@pid_file[:path])
|
388
|
+
file_path = File.join(@pid_file[:path], @pid_file[:name].gsub(' ', '_'))
|
389
|
+
|
390
|
+
pid = nil
|
391
|
+
a = ProcTable.ps
|
392
|
+
a.each_with_index do |p, idx|
|
393
|
+
if p.cmdline.include?(@pid_file[:name])
|
394
|
+
@ext_logger.info "pid:#{p.pid}, cmd:#{p.comm}, cmdline: #{p.cmdline}"
|
395
|
+
pid = p.pid
|
396
|
+
break
|
397
|
+
end
|
398
|
+
# puts "pid:#{p.pid}, cmd:#{p.comm}, cmdline: #{p.cmdline}" if p.cmdline.include?(@task_name)
|
399
|
+
# puts "p.class: #{p.class}"
|
400
|
+
# puts p.to_s
|
401
|
+
# break if idx >= 10
|
402
|
+
# ...
|
403
|
+
end
|
404
|
+
|
405
|
+
|
406
|
+
if pid
|
407
|
+
@ext_logger.info "#{func_name} #{@pid_file[:name]}.pid:#{pid}, pid_file_path:#{file_path}"
|
408
|
+
mkdir_more(file_path)
|
409
|
+
|
410
|
+
File.open(file_path, 'w') do |f|
|
411
|
+
f.write pid
|
412
|
+
f.close
|
413
|
+
end
|
414
|
+
|
415
|
+
else
|
416
|
+
@ext_logger.warn("#{func_name} Pid file generation failed, process[#{@pid_file[:name]}] not found.")
|
417
|
+
end
|
418
|
+
end
|
419
|
+
end
|
420
|
+
|
421
|
+
# Delete batch process pid file
|
422
|
+
def delete_pid_file
|
423
|
+
func_name = "[#{get_class_name}:private #{__method__.to_s}]"
|
424
|
+
if !is_blank?(@pid_file[:name]) && !is_blank?(@pid_file[:path])
|
425
|
+
file_path = File.join(@pid_file[:path], @pid_file[:name].gsub(' ', '_'))
|
426
|
+
|
427
|
+
File.delete(file_path) if File.exist?(file_path)
|
428
|
+
end
|
429
|
+
end
|
430
|
+
|
431
|
+
def get_total_time(s_time, e_time=Time.now)
|
432
|
+
func_name = "[#{get_instance_name}.#{__method__.to_s}]"
|
433
|
+
total_use_time = format_sub_time(s_time, e_time)
|
434
|
+
return "[total use time: #{total_use_time}]"
|
435
|
+
end
|
436
|
+
|
437
|
+
end
|
data/lib/ext_batch.rb
ADDED
@@ -0,0 +1,30 @@
|
|
1
|
+
namespace :ext_batch do
|
2
|
+
|
3
|
+
desc 'Initialize and install necessary workers to monitor files and corresponding processes in batch_pids'
|
4
|
+
task install: :environment do |task|
|
5
|
+
require 'FileUtils'
|
6
|
+
require 'ext_logger/common'
|
7
|
+
message = "[#{task.name}]"
|
8
|
+
# Next is task logic.
|
9
|
+
begin
|
10
|
+
puts "#{message} START"
|
11
|
+
|
12
|
+
if !defined? Rails
|
13
|
+
puts "#{message} ERROR: Rails class not found."
|
14
|
+
exit!(0)
|
15
|
+
end
|
16
|
+
|
17
|
+
batch_file_path = "../../templates/workers/ext_batch_monitor.rb"
|
18
|
+
target_path = "app/workers"
|
19
|
+
|
20
|
+
FileUtils.mkdir_p(File.dirname(target_path))
|
21
|
+
|
22
|
+
FileUtils.cp_r(batch_file_path, target_path)
|
23
|
+
|
24
|
+
puts "#{message} END"
|
25
|
+
rescue Exception => e
|
26
|
+
puts e.message + "\n" + e.backtrace.join("\n")
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
class BitflyerFxPairWorker
|
2
|
+
include Sidekiq::Worker
|
3
|
+
sidekiq_options :retry => false
|
4
|
+
|
5
|
+
def perform(*args)
|
6
|
+
# 这个worker取bitFlyer的FX数据
|
7
|
+
# 只显示在BTC_JPY的页面
|
8
|
+
# 只存入redis
|
9
|
+
logger = TickApi.new_logger(File.basename(__FILE__).gsub('.rb', ''))
|
10
|
+
begin
|
11
|
+
pair_code = 'fx_btc_jpy'
|
12
|
+
Yml::TICK_API[:pair][pair_code].each do |item|
|
13
|
+
company = Yml::TICK_API[:company][item]
|
14
|
+
next if company.blank?
|
15
|
+
|
16
|
+
code = company[:code][pair_code]
|
17
|
+
code = pair_code if code == ''
|
18
|
+
|
19
|
+
# 请求api
|
20
|
+
json = TickApi.get_tick_data(company[:api], code, {logger: logger})
|
21
|
+
return if json.blank?
|
22
|
+
|
23
|
+
# 保存到缓存
|
24
|
+
# 数据存入到btc_jpy的key里面用于tick.coffee实时更新
|
25
|
+
TickApi.save_tick_cache('btc_jpy', item, json, {logger: logger})
|
26
|
+
# 数据存入到fx_btc_jpy的key里面用于页面刷新
|
27
|
+
ticks = TickApi.save_tick_cache(pair_code, item, json, {logger: logger})
|
28
|
+
logger.info "[main]#{pair_code} ### worker done." if !ticks.blank?
|
29
|
+
end
|
30
|
+
rescue Exception => e
|
31
|
+
logger.error e.message
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
metadata
ADDED
@@ -0,0 +1,132 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: ext_batch
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Ruby
|
8
|
+
autorequire:
|
9
|
+
bindir: exe
|
10
|
+
cert_chain: []
|
11
|
+
date: 2019-01-22 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: bundler
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '1.15'
|
20
|
+
type: :development
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '1.15'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: rake
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '10.0'
|
34
|
+
type: :development
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "~>"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '10.0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: mail
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - "~>"
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '2.5'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - "~>"
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '2.5'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: sys-proctable
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - "~>"
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '1.2'
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - "~>"
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '1.2'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: ext_logger
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - "~>"
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0.2'
|
76
|
+
type: :runtime
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - "~>"
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '0.2'
|
83
|
+
description: Depending on the ext_logger gem package, create for batch. The batch
|
84
|
+
resource consumption can be recorded, and the specified mail can be sent according
|
85
|
+
to the implementation of batch, and the mail format is fixed.
|
86
|
+
email:
|
87
|
+
- wu.chao@maisiteng.com
|
88
|
+
executables: []
|
89
|
+
extensions: []
|
90
|
+
extra_rdoc_files: []
|
91
|
+
files:
|
92
|
+
- ".gitignore"
|
93
|
+
- ".ruby-gemset"
|
94
|
+
- ".ruby-version"
|
95
|
+
- Gemfile
|
96
|
+
- LICENSE.txt
|
97
|
+
- README.md
|
98
|
+
- Rakefile
|
99
|
+
- bin/console
|
100
|
+
- bin/setup
|
101
|
+
- ext_batch.gemspec
|
102
|
+
- lib/ext_batch.rb
|
103
|
+
- lib/ext_batch/common.rb
|
104
|
+
- lib/ext_batch/main.rb
|
105
|
+
- lib/ext_batch/version.rb
|
106
|
+
- lib/tasks/ext_batch.rake
|
107
|
+
- lib/templates/workers/ext_batch_monitor.rb
|
108
|
+
homepage: ''
|
109
|
+
licenses:
|
110
|
+
- MIT
|
111
|
+
metadata: {}
|
112
|
+
post_install_message:
|
113
|
+
rdoc_options: []
|
114
|
+
require_paths:
|
115
|
+
- lib
|
116
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
117
|
+
requirements:
|
118
|
+
- - ">="
|
119
|
+
- !ruby/object:Gem::Version
|
120
|
+
version: '0'
|
121
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
122
|
+
requirements:
|
123
|
+
- - ">="
|
124
|
+
- !ruby/object:Gem::Version
|
125
|
+
version: '0'
|
126
|
+
requirements: []
|
127
|
+
rubyforge_project:
|
128
|
+
rubygems_version: 2.6.12
|
129
|
+
signing_key:
|
130
|
+
specification_version: 4
|
131
|
+
summary: Depending on the ext_logger gem package, create for batch.
|
132
|
+
test_files: []
|