hutils 0.0.1 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +129 -0
- data/bin/lcut +2 -2
- data/bin/lfmt +38 -0
- data/bin/ltap +13 -0
- data/bin/lviz +6 -7
- data/lib/hutils/ltap/conf.rb +6 -0
- data/lib/hutils/ltap/paper_trail_drainer.rb +37 -6
- data/lib/hutils/ltap/splunk_drainer.rb +16 -5
- data/lib/hutils/ltap/time_bound_parser.rb +41 -0
- data/lib/hutils/ltap.rb +1 -0
- data/lib/hutils/text_visualizer.rb +60 -34
- data/lib/hutils.rb +9 -5
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 6a480d2cd70d3539d12e0fcf0fbcdafa071df4a7
|
4
|
+
data.tar.gz: 13f269ea287bfc976fbcbcb60832a02374e21a49
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 69a4a20b2ed69f2fe760d1e058dd6b30dfd62556cb559bbdd79a2518a680563436c483ff4ce168f7ce2b354453b224949725b789187d94567a5eb3233bf6b0b3
|
7
|
+
data.tar.gz: 5ed33ee6fc8a50e4c340f97e2f1d9c2a7c8d10e0c97c837ad20e64e8fac0c66933b751ed9176cbe6a6f85eb7e618451276c596a78423ef5eae0a00f026a2a969
|
data/README.md
CHANGED
@@ -8,6 +8,135 @@ A small collection of utilies for [logfmt](http://brandur.org/logfmt) processing
|
|
8
8
|
gem install hutils
|
9
9
|
```
|
10
10
|
|
11
|
+
## Utilities
|
12
|
+
|
13
|
+
### lcut
|
14
|
+
|
15
|
+
`lcut` extracts values from a logfmt trace based on some field name.
|
16
|
+
|
17
|
+
```
|
18
|
+
$ ltap 'instrumentation app=api earliest=-1m at=finish' | lcut method path
|
19
|
+
GET /providers/users/search
|
20
|
+
GET /vendor/resources/6307854
|
21
|
+
GET /health
|
22
|
+
GET /vendor/resources/6007506
|
23
|
+
GET /vendor/resources/7117492
|
24
|
+
```
|
25
|
+
|
26
|
+
### lfmt
|
27
|
+
|
28
|
+
`lfmt` prettifies logfmt lines as they emerge from a stream, and highlights their key sections.
|
29
|
+
|
30
|
+
(Note that the example below doesn't demonstrate color, which is one of the more important features of `logfmt`.)
|
31
|
+
|
32
|
+
```
|
33
|
+
$ ltap 'instrumentation app=api earliest=-1m at=finish' | lfmt
|
34
|
+
api.108081@heroku.com app: api at: finish component: manager_apiauthorized elapsed: 0.008 instance_name: api.108081 instrumentation length: 339 method: GET path: /providers/users/search request_id: ef82825d-4c10-41f3-89ed-6bf805aa4513 status: 200 user: heroku-postgresql@addons.heroku.com user_id: 105750 version: 1
|
35
|
+
api.136540@heroku.com app: api at: finish elapsed: 0.001 instance_name: api.136540 instrumentation method: GET path: /vendor/resources/6307854 request_id: 055df716-fc62-4554-b976-e2fe2472e107 status: 200 user: 3paccounts@dwnldmedia.com user_id: 97546 version: 2
|
36
|
+
api.93579@heroku.com app: api at: finish elapsed: 0.000 instance_name: api.93579 instrumentation method: GET path: /health request_id: 6af07088-82af-4f50-87c1-8b5d248807f0 status: 200 user: heroku-postgresql@addons.heroku.com user_id: 105750 version: 1
|
37
|
+
api.108081@heroku.com app: api at: finish elapsed: 0.174 instance_name: api.108081 instrumentation method: GET path: /vendor/resources/6007506 request_id: ef82825d-4c10-41f3-89ed-6bf805aa4513 status: 200 user: heroku-postgresql@addons.heroku.com user_id: 105750 version: 1
|
38
|
+
api.108081@heroku.com app: api at: finish elapsed: 0.162 instance_name: api.108081 instrumentation method: GET path: /vendor/resources/7117492 request_id: 7480424d-5a8a-488a-a32a-55812fde5f4b status: 200 user: heroku-postgresql@addons.heroku.com user_id: 105750 version: 1
|
39
|
+
```
|
40
|
+
|
41
|
+
### ltap
|
42
|
+
|
43
|
+
`ltap` accesses messages from popular log providers in a consistent way so that it can easily be parsed by other utilities that operate on logfmt traces. Currently supported providers are Papertrail and Splunk.
|
44
|
+
|
45
|
+
```
|
46
|
+
$ ltap 'instrumentation app=api earliest=-1m at=finish'
|
47
|
+
api.108081@heroku.com instrumentation method=GET path=/providers/users/search request_id=d5c373fd-d1ec-4986-bc43-2617431116f2 at=finish elapsed=0.008 length=339 status=200 app=api instance_name=api.108081 version=1 component=manager_apiauthorized app=api instance_name=api.108081 request_id=ef82825d-4c10-41f3-89ed-6bf805aa4513 version=1 user=heroku-postgresql@addons.heroku.com user_id=105750
|
48
|
+
api.136540@heroku.com instrumentation method=GET path=/vendor/resources/6307854 request_id=d2f25032-9aaa-41e9-8aaf-9a46a44523d1 at=finish elapsed=0.110 status=200 app=api instance_name=api.136540 version=1 user=heroku-postgresql@addons.heroku.com user_id=105750step=check_oauth_scope! request_id=055df716-fc62-4554-b976-e2fe2472e107 version=2 user=account@example.com user_id=97546 app=api instance_name=api.136540 at=finish elapsed=0.001
|
49
|
+
api.93579@heroku.com instrumentation method=GET path=/health request_id=6af07088-82af-4f50-87c1-8b5d248807f0 at=finish elapsed=0.000 status=200 app=api instance_name=api.93579 version=1 user=heroku-postgresql@addons.heroku.com user_id=105750
|
50
|
+
api.108081@heroku.com instrumentation method=GET path=/vendor/resources/6007506 request_id=ef82825d-4c10-41f3-89ed-6bf805aa4513 at=finish elapsed=0.174 status=200 app=api instance_name=api.108081 version=1 user=heroku-postgresql@addons.heroku.com user_id=105750
|
51
|
+
api.108081@heroku.com instrumentation method=GET path=/vendor/resources/7117492 request_id=7480424d-5a8a-488a-a32a-55812fde5f4b at=finish elapsed=0.162 status=200 app=api instance_name=api.108081 version=1 user=heroku-postgresql@addons.heroku.com user_id=105750
|
52
|
+
```
|
53
|
+
|
54
|
+
`ltap` can be configured using `~/.ltap`. A sample Papertrail configuration looks like the following:
|
55
|
+
|
56
|
+
```
|
57
|
+
[global]
|
58
|
+
profile = my_papertrail
|
59
|
+
|
60
|
+
[my_papertrail]
|
61
|
+
key = an-api-key
|
62
|
+
type = papertrail
|
63
|
+
```
|
64
|
+
|
65
|
+
A sample Splunk configuration:
|
66
|
+
|
67
|
+
```
|
68
|
+
[global]
|
69
|
+
profile = my_splunk
|
70
|
+
|
71
|
+
[my_splunk]
|
72
|
+
earliest = -24h
|
73
|
+
type = splunk
|
74
|
+
url = https://brandur:an-api-key@splunk.example.com:8089
|
75
|
+
```
|
76
|
+
|
77
|
+
### lviz
|
78
|
+
|
79
|
+
`lviz` helps to visualize logfmt output by building a tree out of some set of data by combining common sets of key/value pairs into shared parent nodes. Messages remain ordered by time, which removes some potential for commonality, but in many cases a disproportionate number of attributes can be moved up to nodes close to the top of the tree. Output is colorized and important keys are highlighted to make traces more easily digestible.
|
80
|
+
|
81
|
+
```
|
82
|
+
$ ltap 'instrumentation app=api earliest=-1m at=finish' | lviz
|
83
|
+
+ app: api
|
84
|
+
instrumentation
|
85
|
+
method: GET
|
86
|
+
status: 200
|
87
|
+
|
88
|
+
+ api.108081@heroku.com
|
89
|
+
component: manager_apiauthorized
|
90
|
+
elapsed: 0.008
|
91
|
+
instance_name: api.108081
|
92
|
+
length: 339
|
93
|
+
path: /providers/users/search
|
94
|
+
request_id: ef82825d-4c10-41f3-89ed-6bf805aa4513
|
95
|
+
user: heroku-postgresql@addons.heroku.com
|
96
|
+
user_id: 105750
|
97
|
+
version: 1
|
98
|
+
|
99
|
+
+ api.136540@heroku.com
|
100
|
+
elapsed: 0.001
|
101
|
+
instance_name: api.136540
|
102
|
+
path: /vendor/resources/6307854
|
103
|
+
request_id: 055df716-fc62-4554-b976-e2fe2472e107
|
104
|
+
user: account@example.com
|
105
|
+
user_id: 97546
|
106
|
+
version: 2
|
107
|
+
|
108
|
+
+ user: heroku-postgresql@addons.heroku.com
|
109
|
+
user_id: 105750
|
110
|
+
version: 1
|
111
|
+
|
112
|
+
+ api.93579@heroku.com
|
113
|
+
elapsed: 0.000
|
114
|
+
instance_name: api.93579
|
115
|
+
path: /health
|
116
|
+
request_id: 6af07088-82af-4f50-87c1-8b5d248807f0
|
117
|
+
|
118
|
+
+ api.108081@heroku.com
|
119
|
+
instance_name: api.108081
|
120
|
+
|
121
|
+
+ elapsed: 0.174
|
122
|
+
path: /vendor/resources/6007506
|
123
|
+
request_id: ef82825d-4c10-41f3-89ed-6bf805aa4513
|
124
|
+
|
125
|
+
+ elapsed: 0.162
|
126
|
+
path: /vendor/resources/7117492
|
127
|
+
request_id: 7480424d-5a8a-488a-a32a-55812fde5f4b
|
128
|
+
```
|
129
|
+
|
130
|
+
`lviz` can be configured with `~/.lviz`. For example:
|
131
|
+
|
132
|
+
```
|
133
|
+
[global]
|
134
|
+
highlights = path,user
|
135
|
+
ignore = at
|
136
|
+
```
|
137
|
+
|
138
|
+
`lviz` can also produce a compact mode of output using `-c` or `--compact`.
|
139
|
+
|
11
140
|
## Testing
|
12
141
|
|
13
142
|
```
|
data/bin/lcut
CHANGED
@@ -32,8 +32,8 @@ end
|
|
32
32
|
|
33
33
|
file = file ? File.open(file) : $stdin
|
34
34
|
file.each_line do |line|
|
35
|
-
|
36
|
-
|
35
|
+
events = Hutils::Parser.new(line).parse
|
36
|
+
events.each do |message, _|
|
37
37
|
values = ARGV.map { |f| message[f] ? message[f] : nil }
|
38
38
|
if allow_empty || !values.all? { |v| v.nil? }
|
39
39
|
puts values.join(delimiter)
|
data/bin/lfmt
ADDED
@@ -0,0 +1,38 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require "optparse"
|
4
|
+
|
5
|
+
require_relative "../lib/hutils"
|
6
|
+
|
7
|
+
colors = $stdout.tty?
|
8
|
+
highlights = %w(endpoint path route_signature user user_id)
|
9
|
+
|
10
|
+
opts = OptionParser.new do |opts|
|
11
|
+
opts.banner = "Usage: lsel [options] <fields>"
|
12
|
+
opts.on("-d", "--delimiter [DELIMITER]", "Delimiter separating output") { |d|
|
13
|
+
delimiter = d
|
14
|
+
}
|
15
|
+
opts.on("-e", "--allow-empty", "Output empty lines") { |e| allow_empty = e }
|
16
|
+
opts.on("-f", "--file [FILE]", "File to read") { |f| file = f }
|
17
|
+
opts.on("-h", "--help", "Show this help string") { |h|
|
18
|
+
if h
|
19
|
+
puts(opts.help)
|
20
|
+
exit(0)
|
21
|
+
end
|
22
|
+
}
|
23
|
+
opts.on("-v", "--verbose", "Verbose mode") { |v| verbose = v }
|
24
|
+
end
|
25
|
+
opts.parse!
|
26
|
+
|
27
|
+
line_visualizer = Hutils::TextVisualizer::LineVisualizer.new(
|
28
|
+
colors: colors,
|
29
|
+
compact: true,
|
30
|
+
highlights: highlights,
|
31
|
+
out: $stdout)
|
32
|
+
|
33
|
+
ARGF.each_line do |line|
|
34
|
+
events = Hutils::Parser.new(line).parse
|
35
|
+
events.each do |message, time|
|
36
|
+
line_visualizer.display(message, time: time)
|
37
|
+
end
|
38
|
+
end
|
data/bin/ltap
CHANGED
@@ -16,9 +16,13 @@ opts = OptionParser.new do |opts|
|
|
16
16
|
exit(0)
|
17
17
|
end
|
18
18
|
}
|
19
|
+
opts.on("-e", "--earliest [TIME]", "Bound for earliest message") { |e|
|
20
|
+
conf.earliest = e
|
21
|
+
}
|
19
22
|
opts.on("-k", "--key [KEY]", "Service API key") { |t| conf.timeout = t }
|
20
23
|
opts.on("-p", "--profile [PROFILE]", "Conf profile") { |p| conf.profile = p }
|
21
24
|
opts.on("-t", "--timeout [TIMEOUT]", "Job timeout") { |t| conf.timeout = t }
|
25
|
+
opts.on("-s", "--timestamps", "Output timestamps") { |t| conf.timestamps = t }
|
22
26
|
opts.on("-y", "--type [TYPE]", "Service type") { |t| conf.timeout = t }
|
23
27
|
opts.on("-u", "--url [URL]", "Service API URL") { |u| conf.url = u }
|
24
28
|
opts.on("-v", "--verbose", "Verbose mode") { |v| conf.verbose = v }
|
@@ -38,6 +42,13 @@ unless ARGV.first
|
|
38
42
|
abort(opts.help)
|
39
43
|
end
|
40
44
|
|
45
|
+
parser = Hutils::Ltap::TimeBoundParser.new
|
46
|
+
if !(earliest = parser.parse(conf.earliest, from: Time.now.getutc))
|
47
|
+
$stderr.puts %{Couldn't parse bound "#{conf.earliest}", reverting to default}
|
48
|
+
# duplicated from conf, which is not ideal
|
49
|
+
earliest = Time.now.getutc - 60 * 60 * 24
|
50
|
+
end
|
51
|
+
|
41
52
|
drainer = case conf.type
|
42
53
|
when "papertrail"
|
43
54
|
Hutils::Ltap::PaperTrailDrainer
|
@@ -48,9 +59,11 @@ else
|
|
48
59
|
end
|
49
60
|
|
50
61
|
drainer = drainer.new(
|
62
|
+
earliest: earliest,
|
51
63
|
key: conf.key,
|
52
64
|
query: ARGV.first,
|
53
65
|
timeout: conf.timeout,
|
66
|
+
timestamps: conf.timestamps,
|
54
67
|
url: conf.url,
|
55
68
|
verbose: conf.verbose
|
56
69
|
)
|
data/bin/lviz
CHANGED
@@ -7,7 +7,7 @@ require_relative "../lib/hutils"
|
|
7
7
|
colors = $stdout.tty?
|
8
8
|
compact = false
|
9
9
|
highlights = %w(endpoint path route_signature user user_id)
|
10
|
-
ignore = %w(
|
10
|
+
ignore = %w()
|
11
11
|
interactive = false
|
12
12
|
|
13
13
|
ini = IniFile.load(ENV["HOME"] + "/.lviz")
|
@@ -34,15 +34,16 @@ opts = OptionParser.new do |opts|
|
|
34
34
|
ignore = i.split(",") rescue []
|
35
35
|
}
|
36
36
|
opts.on("-i", "--interactive", "Interactive mode") { |i| interactive = i }
|
37
|
-
opts.on("--no-color", "Disable colors") { |c| colors =
|
37
|
+
opts.on("--no-color", "Disable colors") { |c| colors = false }
|
38
38
|
end
|
39
39
|
opts.parse!
|
40
40
|
|
41
41
|
# ARGF will read from stdin if available, and otherwise fall back to files
|
42
42
|
# named as parameters (similar to the behavior of `cat`)
|
43
|
-
|
44
|
-
|
45
|
-
|
43
|
+
events = Hutils::Parser.new(ARGF.read).parse
|
44
|
+
messages = events.map { |e| e[0] }
|
45
|
+
Hutils::Stripper.new(messages, ignore).run
|
46
|
+
root = Hutils::TreeBuilder.new(messages).build
|
46
47
|
|
47
48
|
if interactive
|
48
49
|
require_relative "../lib/hutils/curses_visualizer"
|
@@ -59,8 +60,6 @@ if interactive
|
|
59
60
|
root: root
|
60
61
|
).run
|
61
62
|
else
|
62
|
-
require_relative "../lib/hutils/text_visualizer"
|
63
|
-
|
64
63
|
Hutils::TextVisualizer.new(
|
65
64
|
colors: colors,
|
66
65
|
compact: compact,
|
data/lib/hutils/ltap/conf.rb
CHANGED
@@ -2,16 +2,20 @@ require "inifile"
|
|
2
2
|
|
3
3
|
module Hutils::Ltap
|
4
4
|
class Conf
|
5
|
+
attr_accessor :earliest
|
5
6
|
attr_accessor :key
|
6
7
|
attr_accessor :profile
|
7
8
|
attr_accessor :timeout
|
9
|
+
attr_accessor :timestamps
|
8
10
|
attr_accessor :type
|
9
11
|
attr_accessor :url
|
10
12
|
attr_accessor :verbose
|
11
13
|
|
12
14
|
def initialize
|
13
15
|
@ini = IniFile.load(ENV["HOME"] + "/.ltap")
|
16
|
+
self.earliest = "-24h"
|
14
17
|
self.timeout = 60
|
18
|
+
self.timestamps = false
|
15
19
|
self.verbose = false
|
16
20
|
end
|
17
21
|
|
@@ -21,9 +25,11 @@ module Hutils::Ltap
|
|
21
25
|
|
22
26
|
def load_section(name)
|
23
27
|
if section = @ini && @ini[name]
|
28
|
+
load_value(section, :earliest)
|
24
29
|
load_value(section, :key)
|
25
30
|
load_value(section, :profile)
|
26
31
|
load_value(section, :timeout)
|
32
|
+
load_value(section, :timestamps)
|
27
33
|
load_value(section, :type)
|
28
34
|
load_value(section, :url)
|
29
35
|
load_value(section, :verbose)
|
@@ -4,13 +4,15 @@ module Hutils::Ltap
|
|
4
4
|
class PaperTrailDrainer
|
5
5
|
PAPER_TRAIL_URL = "https://papertrailapp.com"
|
6
6
|
|
7
|
-
def initialize(key:, timeout:, query:, url:, verbose:)
|
7
|
+
def initialize(earliest:, key:, timeout:, query:, timestamps:, url:, verbose:)
|
8
8
|
@api = Excon.new(PAPER_TRAIL_URL,
|
9
9
|
headers: {
|
10
10
|
"X-Papertrail-Token" => key
|
11
11
|
})
|
12
|
+
@earliest = earliest
|
12
13
|
@query = query
|
13
14
|
@timeout = timeout
|
15
|
+
@timestamps = timestamps
|
14
16
|
@verbose = verbose
|
15
17
|
end
|
16
18
|
|
@@ -20,12 +22,25 @@ module Hutils::Ltap
|
|
20
22
|
start = Time.now
|
21
23
|
|
22
24
|
loop do
|
23
|
-
new_messages, reached_beginning, min_id = fetch_page(min_id)
|
25
|
+
new_messages, reached_beginning, min_id, min_time = fetch_page(min_id)
|
24
26
|
messages += new_messages
|
25
27
|
|
26
28
|
# break if PaperTrail has indicated that we've reached the beginning of
|
27
|
-
# our results
|
28
|
-
if reached_beginning
|
29
|
+
# our results
|
30
|
+
if reached_beginning
|
31
|
+
debug("breaking: reached beginning")
|
32
|
+
break
|
33
|
+
end
|
34
|
+
|
35
|
+
# or if we've reached back before our earliest
|
36
|
+
if min_time && min_time < @earliest
|
37
|
+
debug("breaking: before earliest: #{@earliest}")
|
38
|
+
break
|
39
|
+
end
|
40
|
+
|
41
|
+
# or if we've approximately hit our timeout
|
42
|
+
if (Time.now - start).to_i > @timeout
|
43
|
+
debug("breaking: reached timeout")
|
29
44
|
break
|
30
45
|
end
|
31
46
|
end
|
@@ -45,6 +60,16 @@ module Hutils::Ltap
|
|
45
60
|
class RateLimited < StandardError
|
46
61
|
end
|
47
62
|
|
63
|
+
def build_message(event)
|
64
|
+
message = event["message"].strip
|
65
|
+
if @timestamps
|
66
|
+
# it's already in ISO8601, but let's make it UTC
|
67
|
+
t = Time.parse(event["received_at"]).getutc.iso8601
|
68
|
+
message = "#{t}: #{message}"
|
69
|
+
end
|
70
|
+
message
|
71
|
+
end
|
72
|
+
|
48
73
|
def debug(str)
|
49
74
|
if @verbose
|
50
75
|
puts str
|
@@ -65,11 +90,17 @@ module Hutils::Ltap
|
|
65
90
|
end
|
66
91
|
|
67
92
|
data = JSON.parse(resp.body)
|
93
|
+
events = data["events"]
|
68
94
|
debug("backend_timeout: #{data["backend_timeout"] || false} " +
|
69
95
|
"min_id: #{data["min_id"]} " +
|
70
96
|
"reached_beginning: #{data["reached_beginning"] || false}")
|
71
|
-
|
72
|
-
[
|
97
|
+
|
98
|
+
[
|
99
|
+
events.map { |e| build_message(e) },
|
100
|
+
data["reached_beginning"],
|
101
|
+
data["min_id"],
|
102
|
+
events.last ? Time.parse(events.last["received_at"]) : nil
|
103
|
+
]
|
73
104
|
end
|
74
105
|
end
|
75
106
|
end
|
@@ -1,13 +1,16 @@
|
|
1
1
|
require "csv"
|
2
2
|
require "excon"
|
3
3
|
require "json"
|
4
|
+
require "time"
|
4
5
|
require "uri"
|
5
6
|
|
6
7
|
module Hutils::Ltap
|
7
8
|
class SplunkDrainer
|
8
|
-
def initialize(key:, timeout:, query:, url:, verbose:)
|
9
|
+
def initialize(earliest:, key:, timeout:, query:, timestamps:, url:, verbose:)
|
10
|
+
@earliest = earliest
|
9
11
|
@timeout = timeout
|
10
12
|
@query = query
|
13
|
+
@timestamps = timestamps
|
11
14
|
@verbose = verbose
|
12
15
|
|
13
16
|
@user = URI.parse(url).user
|
@@ -24,6 +27,7 @@ module Hutils::Ltap
|
|
24
27
|
|
25
28
|
# finalize the job if we've broken our timeout point
|
26
29
|
if (Time.now - start).to_i > @timeout
|
30
|
+
debug("breaking: reached timeout")
|
27
31
|
finalize_job
|
28
32
|
break
|
29
33
|
end
|
@@ -51,6 +55,7 @@ module Hutils::Ltap
|
|
51
55
|
path: "/servicesNS/#{@user}/search/search/jobs",
|
52
56
|
expects: 201,
|
53
57
|
body: URI.encode_www_form({
|
58
|
+
earliest_time: @earliest.iso8601,
|
54
59
|
output_mode: "json",
|
55
60
|
search: "search #{query}"
|
56
61
|
})
|
@@ -85,6 +90,8 @@ module Hutils::Ltap
|
|
85
90
|
expects: [200, 204],
|
86
91
|
body: URI.encode_www_form({
|
87
92
|
action: "finalize",
|
93
|
+
# tell Splunk to give us all results
|
94
|
+
count: 0,
|
88
95
|
output_mode: "csv"
|
89
96
|
})
|
90
97
|
)
|
@@ -93,15 +100,19 @@ module Hutils::Ltap
|
|
93
100
|
|
94
101
|
rows = CSV.parse(resp.body)
|
95
102
|
return [] if rows.count < 1
|
96
|
-
|
103
|
+
raw_field = rows[0].index("_raw") || raise("no _raw field detected in Splunk response")
|
104
|
+
time_field = rows[0].index("_time") || raise("no _time field detected in Splunk response")
|
97
105
|
|
98
106
|
# skip the first line as its used for CSV headers
|
99
107
|
rows[1..-1].
|
100
|
-
map { |l| l[
|
108
|
+
map { |l| [l[raw_field], l[time_field]] }.
|
101
109
|
# 2014-08-15T19:01:15.476590+00:00 54.197.117.24 local0.notice
|
102
110
|
# api-web-1[23399]: - api.108080@heroku.com ...
|
103
|
-
map { |l| l.gsub(/^.*: - /, "") }.
|
104
|
-
map { |l| l.strip }.
|
111
|
+
map { |l, t| [l.gsub(/^.*: - /, ""), t] }.
|
112
|
+
map { |l, t| [l.strip, t] }.
|
113
|
+
# format timestamps consistently (+00:00 --> Z)
|
114
|
+
map { |l, t| [l, Time.parse(t).getutc.iso8601] }.
|
115
|
+
map { |l, t| @timestamps ? "#{t}: #{l}" : l }.
|
105
116
|
# results come in from newest to oldest; flip that
|
106
117
|
reverse
|
107
118
|
end
|
@@ -0,0 +1,41 @@
|
|
1
|
+
module Hutils::Ltap
|
2
|
+
# A parser designed to support the same style of relative time that Splunk
|
3
|
+
# does, but note that absolute dates and "snap to" times are not supported.
|
4
|
+
#
|
5
|
+
# See here for more information on this format:
|
6
|
+
#
|
7
|
+
# http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchReference/SearchTimeModifiers
|
8
|
+
class TimeBoundParser
|
9
|
+
def parse(str, from: Time.now.getutc)
|
10
|
+
if str =~ /^([+\-])([0-9]+)?([a-z]+)$/
|
11
|
+
to_date($2 || 1, $3, from: from)
|
12
|
+
else
|
13
|
+
nil
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def to_date(num, unit, from: from)
|
20
|
+
unit_time = case unit
|
21
|
+
when "s", "sec", "secs", "second", "seconds"
|
22
|
+
1
|
23
|
+
when "m", "min", "minute", "minutes"
|
24
|
+
60
|
25
|
+
when "h", "hr", "hrs", "hour", "hours"
|
26
|
+
60 * 60
|
27
|
+
when "d", "day", "days"
|
28
|
+
60 * 60 * 24
|
29
|
+
when "w", "week", "weeks"
|
30
|
+
60 * 60 * 24 * 7
|
31
|
+
when "mon", "month", "months"
|
32
|
+
60 * 60 * 24 * 7 * 30
|
33
|
+
when "q", "qtr", "qtrs", "quarter", "quarters"
|
34
|
+
60 * 60 * 24 * 7 * 30 * 3
|
35
|
+
when "y", "yr", "yrs", "year", "years"
|
36
|
+
60 * 60 * 24 * 7 * 30 * 365
|
37
|
+
end
|
38
|
+
from - num.to_f * unit_time
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
data/lib/hutils/ltap.rb
CHANGED
@@ -1,8 +1,58 @@
|
|
1
1
|
require "term/ansicolor"
|
2
|
+
require "time"
|
2
3
|
|
3
4
|
module Hutils
|
4
5
|
class TextVisualizer
|
5
|
-
|
6
|
+
class LineVisualizer
|
7
|
+
include Term::ANSIColor
|
8
|
+
|
9
|
+
def initialize(colors:, compact:, highlights:, out:)
|
10
|
+
@colors = colors
|
11
|
+
@compact = compact
|
12
|
+
@highlights = highlights
|
13
|
+
@out = out
|
14
|
+
end
|
15
|
+
|
16
|
+
def display(message, indent: "", time: nil)
|
17
|
+
if time
|
18
|
+
@out.print "#{colorize(:cyan, time.iso8601)} "
|
19
|
+
end
|
20
|
+
message.to_a.sort_by { |k, v| k }.map { |k, v|
|
21
|
+
pair_to_string(k, v)
|
22
|
+
}.each_with_index { |display, i|
|
23
|
+
if @compact
|
24
|
+
@out.print(indent) if i == 0
|
25
|
+
@out.print("#{display} ")
|
26
|
+
else
|
27
|
+
marker = i == 0 ? "+ " : " "
|
28
|
+
@out.puts "#{indent}#{marker}#{display}"
|
29
|
+
end
|
30
|
+
}
|
31
|
+
@out.puts ""
|
32
|
+
end
|
33
|
+
|
34
|
+
private
|
35
|
+
|
36
|
+
def colorize(method, str)
|
37
|
+
if @colors
|
38
|
+
send(method, str)
|
39
|
+
else
|
40
|
+
str
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
def pair_to_string(k, v)
|
45
|
+
if v == true
|
46
|
+
colorize(:green, k)
|
47
|
+
else
|
48
|
+
if @highlights.include?(k)
|
49
|
+
colorize(:on_yellow, colorize(:black, "#{k}: #{v}"))
|
50
|
+
else
|
51
|
+
"#{colorize(:green, k)}: #{v}"
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
6
56
|
|
7
57
|
def initialize(colors:, compact:, highlights:, root:, out:)
|
8
58
|
@colors = colors
|
@@ -10,52 +60,28 @@ module Hutils
|
|
10
60
|
@highlights = highlights
|
11
61
|
@out = out
|
12
62
|
@root = root
|
63
|
+
|
64
|
+
@line_visualizer = LineVisualizer.new(
|
65
|
+
colors: colors,
|
66
|
+
compact: compact,
|
67
|
+
highlights: highlights,
|
68
|
+
out: out
|
69
|
+
)
|
13
70
|
end
|
14
71
|
|
15
72
|
def display
|
16
73
|
display_node(@root)
|
17
74
|
end
|
18
75
|
|
19
|
-
private
|
20
|
-
|
21
|
-
def colorize(method, str)
|
22
|
-
if @colors
|
23
|
-
send(method, str)
|
24
|
-
else
|
25
|
-
str
|
26
|
-
end
|
27
|
-
end
|
76
|
+
private
|
28
77
|
|
29
78
|
def display_node(node)
|
30
79
|
if !node.common.empty?
|
31
80
|
# the "- 1" is because the root node is empty
|
32
81
|
indent = "\t" * (node.depth - 1)
|
33
|
-
node.common
|
34
|
-
pair_to_string(k, v)
|
35
|
-
}.each_with_index { |display, i|
|
36
|
-
if @compact
|
37
|
-
@out.print(indent) if i == 0
|
38
|
-
@out.print("#{display} ")
|
39
|
-
else
|
40
|
-
marker = i == 0 ? "+ " : " "
|
41
|
-
@out.puts "#{indent}#{marker}#{display}"
|
42
|
-
end
|
43
|
-
}
|
44
|
-
@out.puts ""
|
82
|
+
@line_visualizer.display(node.common, indent: indent)
|
45
83
|
end
|
46
84
|
node.slots.each { |slot| display_node(slot) }
|
47
85
|
end
|
48
|
-
|
49
|
-
def pair_to_string(k, v)
|
50
|
-
if v == true
|
51
|
-
colorize(:green, k)
|
52
|
-
else
|
53
|
-
if @highlights.include?(k)
|
54
|
-
colorize(:on_yellow, colorize(:black, "#{k}: #{v}"))
|
55
|
-
else
|
56
|
-
"#{colorize(:green, k)}: #{v}"
|
57
|
-
end
|
58
|
-
end
|
59
|
-
end
|
60
86
|
end
|
61
87
|
end
|
data/lib/hutils.rb
CHANGED
@@ -1,4 +1,7 @@
|
|
1
|
+
require "time"
|
2
|
+
|
1
3
|
require_relative "hutils/stripper"
|
4
|
+
require_relative "hutils/text_visualizer"
|
2
5
|
|
3
6
|
module Hutils
|
4
7
|
class Node
|
@@ -63,16 +66,16 @@ module Hutils
|
|
63
66
|
end
|
64
67
|
|
65
68
|
def parse
|
66
|
-
|
67
|
-
|
68
|
-
pairs =
|
69
|
+
events = @str.split("\n").map { |line| normalize(line) }
|
70
|
+
events.map! do |message, time|
|
71
|
+
pairs = message.scan(/(?:['"](?:\\.|[^'"])*['"]|[^'" ])+/).map do |pair|
|
69
72
|
key, value = pair.split("=")
|
70
73
|
[key, value].each do |str|
|
71
74
|
str.gsub!(/^['"]?(.*?)['"]?$/, '\1') if str
|
72
75
|
end
|
73
76
|
[key, value || true]
|
74
77
|
end
|
75
|
-
Hash[pairs]
|
78
|
+
[Hash[pairs], time]
|
76
79
|
end
|
77
80
|
end
|
78
81
|
|
@@ -80,7 +83,8 @@ module Hutils
|
|
80
83
|
|
81
84
|
def normalize(line)
|
82
85
|
line = line.strip
|
83
|
-
line.gsub(/^[
|
86
|
+
line = line.gsub(/^([TZ0-9\-:+.]+)( [a-z]+\[[a-z0-9\-_.]+\])?: /, '')
|
87
|
+
[line, $1 ? Time.parse($1).getutc : nil]
|
84
88
|
end
|
85
89
|
end
|
86
90
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: hutils
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0
|
4
|
+
version: 0.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Brandur
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2014-08-
|
11
|
+
date: 2014-08-27 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: excon
|
@@ -74,6 +74,7 @@ description:
|
|
74
74
|
email: brandur@mutelight.org
|
75
75
|
executables:
|
76
76
|
- lcut
|
77
|
+
- lfmt
|
77
78
|
- ltap
|
78
79
|
- lviz
|
79
80
|
extensions: []
|
@@ -85,11 +86,13 @@ files:
|
|
85
86
|
- "./lib/hutils/ltap/conf.rb"
|
86
87
|
- "./lib/hutils/ltap/paper_trail_drainer.rb"
|
87
88
|
- "./lib/hutils/ltap/splunk_drainer.rb"
|
89
|
+
- "./lib/hutils/ltap/time_bound_parser.rb"
|
88
90
|
- "./lib/hutils/node_navigator.rb"
|
89
91
|
- "./lib/hutils/stripper.rb"
|
90
92
|
- "./lib/hutils/text_visualizer.rb"
|
91
93
|
- README.md
|
92
94
|
- bin/lcut
|
95
|
+
- bin/lfmt
|
93
96
|
- bin/ltap
|
94
97
|
- bin/lviz
|
95
98
|
homepage: https://github.com/brandur/hutils
|