syslogstash 1.3.0 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 4fae50a372aca78fab4d20cb3a21ff4248988abf
4
- data.tar.gz: 391ae0a267341f5571e9c90427e83b2862628c2f
3
+ metadata.gz: 8dd70d42345ffba77d56d3511d5220e2ab399ce9
4
+ data.tar.gz: 98f6175cd0cc98d9ca6a51593657b7e3c09f178c
5
5
  SHA512:
6
- metadata.gz: 6b0ab3566b3ce68964cfcb34be6efbdc3799394c76e72d672c53bb9a66e723e888df989c8f39155662d8151f6dbfd3b1915a08d212c92d8ca33cc4e0ad20bacc
7
- data.tar.gz: 8798ef4d1a150cbf105028e5b6d7dcd9d72ed8309f0e998da6f3d0e8989b0e47016b9a9ba02f018d031d627687f28ccdf8a9450f622efcdf1763cef07dead38c
6
+ metadata.gz: 9fe1db70bce8d062dbc84b3c24e5650ed489b9ff89f7502225b2673f848e3c4abde63d6369cd14b3235f1adb9bfb15ceb8b217eaf4392a52badbfdd906ecbf03
7
+ data.tar.gz: 1e4163fa6ccd9c6f6402be87d571f1d67e80584935e7a7772084f3c3dc968f321acbdd0f2ecf7065f9b5f295825d2dd9f997189f7866cd0b9847dc03152a3224
data/Dockerfile ADDED
@@ -0,0 +1,13 @@
1
+ FROM ruby:2.3-alpine
2
+
3
+ ARG GEM_VERSION="> 0"
4
+
5
+ COPY pkg/syslogstash-$GEM_VERSION.gem /tmp/syslogstash.gem
6
+
7
+ RUN apk update \
8
+ && apk add build-base \
9
+ && gem install /tmp/syslogstash.gem \
10
+ && apk del build-base \
11
+ && rm -f /var/cache/apk/* /tmp/syslogstash.gem
12
+
13
+ ENTRYPOINT ["/usr/local/bundle/bin/syslogstash"]
data/Makefile ADDED
@@ -0,0 +1,14 @@
1
+ IMAGE := discourse/syslogstash
2
+ TAG := $(shell date -u +%Y%m%d.%H%M%S)
3
+
4
+ .PHONY: default
5
+ default: push
6
+ @printf "${IMAGE}:${TAG} ready\n"
7
+
8
+ .PHONY: push
9
+ push: build
10
+ docker push ${IMAGE}:${TAG}
11
+
12
+ .PHONY: build
13
+ build:
14
+ docker build --build-arg=http_proxy=${http_proxy} -t ${IMAGE}:${TAG} .
data/README.md CHANGED
@@ -1,4 +1,21 @@
1
- Feed everything from one or more syslog pipes to a logstash server.
1
+ Syslogstash is intended to provide a syslog-compatible socket for one or
2
+ more applications to send their syslog messages to. The messages are then
3
+ parsed and sent to a logstash server for posterity. No more needing to run
4
+ a syslog server that writes to a file, just to have a second program that
5
+ reads those files again. With syslogstash, everything is in one neat little
6
+ package.
7
+
8
+ If you're running a containerised environment, there's a reasonable chance
9
+ you've got multiple things that want to log to syslog, but you want to keep
10
+ them organised and separate. That's easy: just run multiple syslogstash
11
+ instances, one per "virtual syslog socket" you want to provide. Multiple
12
+ containers can share the same socket, they'll just share a logstash
13
+ connection and have the same metadata / extra tags.
14
+
15
+ For maximum flexibility, you can optionally feed the syslog messages to one
16
+ or more other "downstream" sockets, and/or print all the log messages to
17
+ stdout for ad-hoc "local" debugging.
18
+
2
19
 
3
20
  # Installation
4
21
 
@@ -17,74 +34,172 @@ If you're the sturdy type that likes to run from git:
17
34
  Or, if you've eschewed the convenience of Rubygems entirely, then you
18
35
  presumably know what to do already.
19
36
 
37
+ ## Docker
38
+
39
+ Published image at https://hub.docker.com/r/discourse/syslogstash/
40
+
41
+ To build a new Docker image, run `rake docker:build`. A `rake docker:push`
42
+ will push out a new release.
43
+
20
44
 
21
45
  # Usage
22
46
 
23
- Write a configuration file, then start `syslogstash` giving the name of the
24
- config file as an argument:
47
+ Syslogstash is configured by means of environment variables. At the very
48
+ least, `syslogstash` needs to know where logstash is (`LOGSTASH_SERVER`),
49
+ and the socket to listen on for syslog messages (`SYSLOG_SOCKET`). You
50
+ specify those on the command line, like so:
51
+
52
+ LOGSTASH_SERVER=logstash-json \
53
+ SYSLOG_SOCKET=/dev/log \
54
+ syslogstash
55
+
56
+ The full set of environment variables, and their meaning, is described in
57
+ the "Syslogstash Configuration" section, below.
58
+
59
+
60
+ ## Logstash server setup
25
61
 
26
- syslogstash /etc/syslogstash.conf
62
+ The logstash server(s) you send the collected messages to must be configured
63
+ to listen on a TCP port with the `json_lines` codec. This can be done quite
64
+ easily as follows:
27
65
 
28
- ## Config File Format
66
+ tcp {
67
+ port => 5151
68
+ codec => "json_lines"
69
+ }
29
70
 
30
- The file which describes how `syslogstash` will operate is a fairly simple
31
- YAML file. It consists of two sections, `sockets` and `servers`, which list
32
- the UNIX sockets to listen for syslog messages on, and the URLs of logstash
33
- servers to send the resulting log entries to. Optionally, you can specify
34
- additional fields to insert into every message received from each syslog
35
- socket.
71
+ Adjust the port number to taste.
36
72
 
37
- It looks like this:
38
73
 
39
- sockets:
40
- # These sockets have no additional fields
41
- /tmp/sock1:
42
- /tmp/sock2:
74
+ ## Signals
43
75
 
44
- # This socket will have some fields added to its messages, and will
45
- # send all messages to a couple of other sockets, too
46
- /tmp/supersock:
47
- add_fields:
48
- foo: bar
49
- baz: wombat
50
- relay_to:
51
- - /tmp/relaysock1
52
- - /tmp/relaysock2
76
+ There are a few signals that syslogstash recognises, to control various
77
+ aspects of runtime operation. They are:
53
78
 
54
- # Every log entry received will be sent to *exactly* one of these
55
- # servers. This provides high availability for your log messages.
56
- # NOTE: Only tcp:// URLs are supported.
57
- servers:
58
- - tcp://10.0.0.1:5151
59
- - tcp://10.0.0.2:5151
79
+ * **`SIGUSR1`** / **`SIGUSR2`** -- tell syslogstash to increase (`USR1`) or
80
+ decrease (`USR2`) the verbosity of its own internal logging. This doesn't
81
+ change in *any* way the nature or volume of syslog messages that are
82
+ processed and sent to logstash, it is *only* for syslogstash's own internal
83
+ operational logging.
60
84
 
85
+ * **`SIGURG`** -- toggle whether or not relaying to stdout is enabled or
86
+ disabled.
61
87
 
62
- ### Socket configuration
63
88
 
64
- Each socket has a configuration associated with it. Using this
65
- configuration, you can add logstash fields to each entry, and configure
66
- socket relaying.
89
+ ## Use with Docker
67
90
 
68
- The following keys are available under each socket's path:
91
+ For convenience, `syslogstash` is available in a Docker container,
92
+ `discourse/syslogstash:v2`. It requires a bit of gymnastics to get the
93
+ syslog socket from the `syslogstash` container to whatever container you
94
+ want to capture syslog messages from. Typically, you'll want to share a
95
+ volume between the two containers, tell `syslogstash` to create its socket
96
+ there, and then symlink `/dev/log` from the other container to there.
69
97
 
70
- * `add_fields` -- A hash of additional fields to add to every log entry that
71
- is received on this socket, before it is passed on to logstash.
98
+ For example, you might start the syslogstash container like this:
72
99
 
73
- * `relay_to` -- A list of sockets to send all received messages to. This is
74
- useful in a very limited range of circumstances, when (for instance) you
75
- have another syslog socket consumer that wants to get in on the act, like
76
- a legacy syslogd.
100
+ docker run -v /srv/docker/syslogstash:/syslogstash \
101
+ -e LOGSTASH_SERVER=logstash-json \
102
+ -e SYSLOG_SOCKET=/syslogstash/log.sock \
103
+ discourse/syslogstash:v2
77
104
 
105
+ Then use the same volume in your other container:
78
106
 
79
- ## Logstash server configuration
107
+ docker run -v /srv/docker/syslogstash:/syslogstash something/funny
80
108
 
81
- You'll need to setup a TCP input, with the `json_lines` codec, for
82
- `syslogstash` to send log entries to. It can look as simple as this:
109
+ In the other container's startup script, include the following command:
83
110
 
84
- tcp {
85
- port => 5151
86
- codec => "json_lines"
87
- }
111
+ ln -sf /syslogstash/log.sock /dev/log
112
+
113
+ ... and everything will work nicely.
114
+
115
+ If you feel like playing on nightmare mode, you can also mount the log
116
+ socket directly into the other container, like this:
117
+
118
+ docker run -v /srv/docker/syslogstash/log.sock:/dev/log something/funny
119
+
120
+ This allows you to deal with poorly-implemented containers which run
121
+ software that logs to syslog but doesn't provide a way to override where
122
+ `/dev/log` points. *However*, due to the way bind mounts and Unix sockets
123
+ interact, if the syslogstash container restarts *for any reason*, you also
124
+ need to restart any containers that have the socket itself as a volume. If
125
+ you can coax your container management system into satisfying that
126
+ condition, then you're golden.
127
+
128
+
129
+ # Syslogstash Configuration
130
+
131
+ All configuration of syslogstash is done by placing values in environment
132
+ variables. The environment variables that syslogstash recognises are listed
133
+ below.
134
+
135
+ * **`LOGSTASH_SERVER`** (required) -- the domain name or address of the
136
+ logstash server(s) you wish to send entries to. This can be any of:
137
+
138
+ * An IPv4 address and port, separated by a colon. For example,
139
+ `192.0.2.42:5151`. The port *must* be specified.
140
+
141
+ * An IPv6 address (enclosed in square brackets) and port, separated by a
142
+ colon. For example, `[2001:db8::42]:5151`. The port *must* be
143
+ specified.
144
+
145
+ * A fully-qualified or relative domain name and port, separated by a
146
+ colon. The name given will be resolved and all IPv4 and IPv6
147
+ addresses returned will be tried in random order until a successful
148
+ connection is made to one of them. The port *must* be specified.
149
+
150
+ * A fully-qualified or relative domain name *without a port*. In this
151
+ case, the name given will be resolved as a SRV record, and the names and
152
+ ports returned will be used.
153
+
154
+ In all cases, syslogstash respects DNS record TTLs and SRV record
155
+ weight/priority selection rules. We're not monsters.
156
+
157
+ * **`SYSLOG_SOCKET`** (required) -- the absolute path to the socket which
158
+ syslogstash should create and listen on for syslog format messages.
159
+
160
+ * **`BACKLOG_SIZE`** (optional; default `"1000000"`) -- the maximum number of
161
+ messages to queue if the logstash servers are unavailable. Under normal
162
+ operation, syslog messages are immediately relayed to the logstash server
163
+ as they are received. However, if no logstash servers are available,
164
+ syslogstash will maintain a backlog of up to this many syslog messages,
165
+ and will send the entire backlog once a logstash server becomes available
166
+ again.
167
+
168
+ In the event that the queue size limit is reached, the oldest messages
169
+ will be dropped to make way for the new ones.
170
+
171
+ * **`RELAY_TO_STDOUT`** (optional; default `"no"`) -- if set to a
172
+ true-ish string (any of `true`, `yes`, `on`, or `1`, compared
173
+ case-insensitively), then all the syslog messages which are received will
174
+ be printed to stdout (with the priority/facility prefix removed). This
175
+ isn't a replacement for a fully-featured syslog server, merely a quick way
176
+ to dump messages if absolutely required.
177
+
178
+ * **`STATS_SERVER`** (optional; default `"no"`) -- if set to a true-ish
179
+ string (any of `true`, `yes`, `on`, or `1`, compared case-insensitively),
180
+ then a Prometheus-compatible statistics exporter will be started,
181
+ listening on all interfaces on port 9159.
182
+
183
+ * **`ADD_FIELD_<name>`** (optional) -- if you want to add extra fields to
184
+ the entries which are forwarded to logstash, you can specify them here,
185
+ for example:
186
+
187
+ ADD_FIELD_foo=bar ADD_FIELD_baz=wombat [...] syslogstash
188
+
189
+ This will cause all entries sent to logstash to contain `"foo": "bar"`
190
+ and `"baz": "wombat"`, in addition to the rest of the fields usually
191
+ created by syslogstash. Note that nested fields, and value types other
192
+ than strings, are not supported. Also, if you specify a field name also
193
+ used by syslogstash, the results are explicitly undefined.
194
+
195
+ * **`RELAY_SOCKETS`** (optional; default `""`) -- on the off-chance you want
196
+ to feed the syslog messages that syslogstash receives to another
197
+ syslog-compatible consumer (say, an old-school syslogd) you can specify
198
+ additional filenames to use here. Multiple socket filenames can be
199
+ specified by separating each file name with a colon. Syslogstash will open
200
+ each of the specified sockets, if they exist, and write each received
201
+ message to the socket. If the socket does not exist, or the open or write
202
+ operations fail, syslogstash **will not** retry.
88
203
 
89
204
 
90
205
  # Contributing
@@ -100,7 +215,7 @@ request](https://github.com/discourse/syslogstash/pulls].
100
215
  Unless otherwise stated, everything in this repo is covered by the following
101
216
  copyright notice:
102
217
 
103
- Copyright (C) 2015 Civilized Discourse Construction Kit Inc.
218
+ Copyright (C) 2015, 2018 Civilized Discourse Construction Kit Inc.
104
219
 
105
220
  This program is free software: you can redistribute it and/or modify it
106
221
  under the terms of the GNU General Public License version 3, as
data/bin/syslogstash CHANGED
@@ -1,51 +1,56 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
3
  require 'syslogstash'
4
- require 'yaml'
4
+ require 'logger'
5
5
 
6
- if ARGV.length != 1
7
- $stderr.puts <<-EOF.gsub(/^\t\t/, '')
8
- Invalid usage
6
+ logger = Logger.new($stderr)
7
+ logger.formatter = ->(s, t, p, m) { "#{s[0]} [#{p}] #{m}\n" }
8
+ logger.level = Logger.const_get(ENV['SYSLOGSTASH_LOG_LEVEL'] || "INFO")
9
9
 
10
- Usage:
11
- #{$0} <configfile>
12
- EOF
13
-
14
- exit 1
15
- end
16
-
17
- unless File.exist?(ARGV[0])
18
- $stderr.puts "Config file #{ARGV[0]} does not exist"
19
- exit 1
10
+ begin
11
+ cfg = Syslogstash::Config.new(ENV, logger: logger)
12
+ rescue Syslogstash::Config::ConfigurationError => ex
13
+ $stderr.puts "Error in configuration: #{ex.message}"
14
+ exit 1
20
15
  end
21
16
 
22
- unless File.readable?(ARGV[0])
23
- $stderr.puts "Config file #{ARGV[0]} not readable"
24
- exit 1
25
- end
17
+ syslogstash = Syslogstash.new(cfg)
26
18
 
27
- cfg = YAML.load_file(ARGV[0])
19
+ sig_r, sig_w = IO.pipe
28
20
 
29
- unless cfg.is_a? Hash
30
- $stderr.puts "Config file #{ARGV[0]} does not contain a YAML hash"
31
- exit 1
21
+ Signal.trap("USR1") do
22
+ sig_w.print '1'
23
+ end
24
+ Signal.trap("USR2") do
25
+ sig_w.print '2'
26
+ end
27
+ Signal.trap("URG") do
28
+ sig_w.print 'U'
32
29
  end
33
30
 
34
- %w{sockets servers}.each do |section|
35
- unless cfg.has_key?(section)
36
- $stderr.puts "Config file #{ARGV[0]} does not have a '#{section}' section"
37
- exit 1
38
- end
39
-
40
- unless cfg[section].respond_to?(:empty?)
41
- $stderr.puts "Config file #{ARGV[0]} has a malformed '#{section}' section"
42
- exit 1
43
- end
44
-
45
- if cfg[section].empty?
46
- $stderr.puts "Config file #{ARGV[0]} has an empty '#{section}' section"
47
- exit 1
48
- end
31
+ Thread.new do
32
+ loop do
33
+ begin
34
+ c = sig_r.getc
35
+ if c == '1'
36
+ logger.level -= 1 unless logger.level == Logger::DEBUG
37
+ logger.info("SignalHandler") { "Received SIGUSR1; log level is now #{Logger::SEV_LABEL[logger.level]}." }
38
+ elsif c == '2'
39
+ logger.level += 1 unless logger.level == Logger::ERROR
40
+ logger.info("SignalHandler") { "Received SIGUSR2; log level is now #{Logger::SEV_LABEL[logger.level]}." }
41
+ elsif c == 'U'
42
+ cfg.relay_to_stdout = !cfg.relay_to_stdout
43
+ logger.info("SignalHandler") { "Received SIGURG; Relaying to stdout is now #{cfg.relay_to_stdout ? "enabled" : "disabled"}" }
44
+ else
45
+ logger.error("SignalHandler") { "Got an unrecognised character from signal pipe: #{c.inspect}" }
46
+ end
47
+ rescue StandardError => ex
48
+ logger.error("SignalHandler") { (["Exception raised: #{ex.message} (#{ex.class})"] + ex.backtrace).join("\n ") }
49
+ rescue Exception => ex
50
+ $stderr.puts (["Fatal exception in syslogstash signal handler: #{ex.message} (#{ex.class})"] + ex.backtrace).join("\n ")
51
+ exit 42
52
+ end
53
+ end
49
54
  end
50
55
 
51
- Syslogstash.new(cfg['sockets'], cfg['servers'], cfg.fetch('backlog', 1_000_000)).run
56
+ syslogstash.run
data/lib/syslogstash.rb CHANGED
@@ -7,44 +7,45 @@ require 'thwait'
7
7
  # server.
8
8
  #
9
9
  class Syslogstash
10
- def initialize(sockets, servers, backlog)
11
- @metrics = PrometheusExporter.new
12
-
13
- @writer = LogstashWriter.new(servers, backlog, @metrics)
14
-
15
- @readers = sockets.map { |f, cfg| SyslogReader.new(f, cfg, @writer, @metrics) }
10
+ def initialize(cfg)
11
+ @cfg = cfg
12
+ @stats = PrometheusExporter.new(cfg)
13
+ @writer = LogstashWriter.new(cfg, @stats)
14
+ @reader = SyslogReader.new(cfg, @writer, @stats)
15
+ @logger = cfg.logger
16
16
  end
17
17
 
18
18
  def run
19
- @metrics.run
20
- @writer.run
21
- @readers.each { |w| w.run }
19
+ if @cfg.stats_server
20
+ @logger.debug("main") { "Running stats server" }
21
+ @stats.run
22
+ end
22
23
 
23
- tw = ThreadsWait.new(@metrics.thread, @writer.thread, *(@readers.map { |r| r.thread }))
24
+ @writer.run
25
+ @reader.run
24
26
 
25
- dead_thread = tw.next_wait
27
+ dead_thread = ThreadsWait.new(@reader.thread, @writer.thread).next_wait
26
28
 
27
29
  if dead_thread == @writer.thread
28
- $stderr.puts "[Syslogstash] Writer thread crashed."
29
- elsif dead_thread == @metrics.thread
30
- $stderr.puts "[Syslogstash] Metrics exporter thread crashed."
30
+ @logger.error("main") { "Writer thread crashed." }
31
+ elsif dead_thread == @reader.thread
32
+ @logger.error("main") { "Reader thread crashed." }
31
33
  else
32
- reader = @readers.find { |r| r.thread == dead_thread }
33
-
34
- $stderr.puts "[Syslogstash] Reader thread for #{reader.file} crashed."
34
+ @logger.fatal("main") { "ThreadsWait#next_wait returned unexpected value #{dead_thread.inspect}" }
35
+ exit 1
35
36
  end
36
37
 
37
38
  begin
38
39
  dead_thread.join
39
40
  rescue Exception => ex
40
- $stderr.puts "[Syslogstash] Exception in thread was: #{ex.message} (#{ex.class})"
41
- $stderr.puts ex.backtrace.map { |l| " #{l}" }.join("\n")
41
+ @logger.error("main") { (["Exception in crashed thread was: #{ex.message} (#{ex.class})"] + ex.backtrace).join("\n ") }
42
42
  end
43
43
 
44
44
  exit 1
45
45
  end
46
46
  end
47
47
 
48
+ require_relative 'syslogstash/config'
48
49
  require_relative 'syslogstash/syslog_reader'
49
50
  require_relative 'syslogstash/logstash_writer'
50
51
  require_relative 'syslogstash/prometheus_exporter'
@@ -0,0 +1,118 @@
1
+ require 'logger'
2
+
3
+ class Syslogstash::Config
4
+ class ConfigurationError < StandardError; end
5
+
6
+ # Raised if any problems were found with the config
7
+ class InvalidEnvironmentError < StandardError; end
8
+
9
+ attr_reader :logstash_server,
10
+ :syslog_socket,
11
+ :backlog_size,
12
+ :stats_server,
13
+ :add_fields,
14
+ :relay_sockets
15
+
16
+ attr_reader :logger
17
+
18
+ attr_accessor :relay_to_stdout
19
+
20
+ # Create a new syslogstash config based on environment variables.
21
+ #
22
+ # Examines the environment passed in, and then creates a new config
23
+ # object if all is well.
24
+ #
25
+ # @param env [Hash] the set of environment variables to use.
26
+ #
27
+ # @param logger [Logger] the logger to which all diagnostic and error
28
+ # data will be sent.
29
+ #
30
+ # @raise [ConfigurationError] if any problems are detected with the
31
+ # environment variables found.
32
+ #
33
+ def initialize(env, logger:)
34
+ @logger = logger
35
+
36
+ parse_env(env)
37
+ end
38
+
39
+ private
40
+
41
+ def parse_env(env)
42
+ @logger.info("config") { "Parsing environment:\n" + env.map { |k, v| "#{k}=#{v.inspect}" }.join("\n") }
43
+
44
+ @logstash_server = pluck_string(env, "LOGSTASH_SERVER")
45
+ @syslog_socket = pluck_string(env, "SYSLOG_SOCKET")
46
+ @relay_to_stdout = pluck_boolean(env, "RELAY_TO_STDOUT", default: false)
47
+ @stats_server = pluck_boolean(env, "STATS_SERVER", default: false)
48
+ @backlog_size = pluck_integer(env, "BACKLOG_SIZE", valid_range: 0..(2**31 - 1), default: 1_000_000)
49
+ @add_fields = pluck_prefix_list(env, "ADD_FIELD_")
50
+ @relay_sockets = pluck_path_list(env, "RELAY_SOCKETS", default: [])
51
+ end
52
+
53
+ def pluck_string(env, key, default: nil)
54
+ maybe_default(env, key, default) { env[key] }
55
+ end
56
+
57
+ def pluck_boolean(env, key, default: nil)
58
+ maybe_default(env, key, default) do
59
+ case env[key]
60
+ when /\A(no|off|0|false)\z/
61
+ false
62
+ when /\A(yes|on|1|true)\z/
63
+ true
64
+ else
65
+ raise ConfigurationError,
66
+ "Value for #{key} (#{env[key].inspect}) is not a valid boolean"
67
+ end
68
+ end
69
+ end
70
+
71
+ def pluck_integer(env, key, valid_range: nil, default: nil)
72
+ maybe_default(env, key, default) do
73
+ if env[key] !~ /\A\d+\z/
74
+ raise InvalidEnvironmentError,
75
+ "Value for #{key} (#{env[key].inspect}) is not an integer"
76
+ end
77
+
78
+ env[key].to_i.tap do |v|
79
+ unless valid_range.nil? || !valid_range.include?(v)
80
+ raise InvalidEnvironmentError,
81
+ "Value for #{key} (#{env[key]}) out of range (must be between #{valid_range.first} and #{valid_range.last} inclusive)"
82
+ end
83
+ end
84
+ end
85
+ end
86
+
87
+ def pluck_prefix_list(env, prefix)
88
+ {}.tap do |list|
89
+ env.each do |k, v|
90
+ next unless k.start_with? prefix
91
+ key = k.sub(prefix, '')
92
+ list[key] = v
93
+ end
94
+
95
+ @logger.debug("config") { "Prefix list for #{prefix.inspect} is #{list.inspect}" }
96
+ end
97
+ end
98
+
99
+ def pluck_path_list(env, key, default: nil)
100
+ maybe_default(env, key, default) do
101
+ env[key].split(":")
102
+ end
103
+ end
104
+
105
+ def maybe_default(env, key, default)
106
+ if env[key].nil? || env[key].empty?
107
+ if default.nil?
108
+ raise ConfigurationError,
109
+ "Required environment variable #{key} not specified"
110
+ else
111
+ @logger.debug("config") { "Using default value #{default.inspect} for config parameter #{key}" }
112
+ default
113
+ end
114
+ else
115
+ yield.tap { |v| @logger.debug("config") { "Using plucked value #{v.inspect} for config parameter #{key}" } }
116
+ end
117
+ end
118
+ end
@@ -1,22 +1,21 @@
1
- require_relative 'worker'
1
+ require 'resolv'
2
+ require 'ipaddr'
2
3
 
3
- # Write messages to one of a collection of logstash servers.
4
+ # Write messages to a logstash server.
4
5
  #
5
6
  class Syslogstash::LogstashWriter
6
- include Syslogstash::Worker
7
+ Target = Struct.new(:hostname, :port)
8
+
9
+ attr_reader :thread
7
10
 
8
11
  # Create a new logstash writer.
9
12
  #
10
- # Give it a list of servers, and your writer will be ready to go.
11
- # No messages will actually be *delivered*, though, until you call #run.
13
+ # Once the object is created, you're ready to give it messages by
14
+ # calling #send_entry. No messages will actually be *delivered* to
15
+ # logstash, though, until you call #run.
12
16
  #
13
- def initialize(servers, backlog, metrics)
14
- @servers, @backlog, @metrics = servers.map { |s| URI(s) }, backlog, metrics
15
-
16
- unless @servers.all? { |url| url.scheme == 'tcp' }
17
- raise ArgumentError,
18
- "Unsupported URL scheme: #{@servers.select { |url| url.scheme != 'tcp' }.join(', ')}"
19
- end
17
+ def initialize(cfg, stats)
18
+ @server_name, @logger, @backlog, @stats = cfg.logstash_server, cfg.logger, cfg.backlog_size, stats
20
19
 
21
20
  @entries = []
22
21
  @entries_mutex = Mutex.new
@@ -31,18 +30,19 @@ class Syslogstash::LogstashWriter
31
30
  @entries << { content: e, arrival_timestamp: Time.now }
32
31
  while @entries.length > @backlog
33
32
  @entries.shift
34
- @metrics.dropped
33
+ @stats.dropped
35
34
  end
36
35
  end
37
- @worker.run if @worker
36
+
37
+ @thread.run if @thread
38
38
  end
39
39
 
40
40
  # Start sending messages to logstash servers. This method will return
41
41
  # almost immediately, and actual message sending will occur in a
42
- # separate worker thread.
42
+ # separate thread.
43
43
  #
44
44
  def run
45
- @worker = Thread.new { send_messages }
45
+ @thread = Thread.new { send_messages }
46
46
  end
47
47
 
48
48
  private
@@ -57,16 +57,14 @@ class Syslogstash::LogstashWriter
57
57
 
58
58
  current_server do |s|
59
59
  s.puts entry[:content]
60
+ @stats.sent(server_id(s), entry[:arrival_timestamp])
60
61
  end
61
62
 
62
- @metrics.sent(@servers.last, entry[:arrival_timestamp])
63
-
64
63
  # If we got here, we sent successfully, so we don't want
65
64
  # to put the entry back on the queue in the ensure block
66
65
  entry = nil
67
66
  rescue StandardError => ex
68
- log { "Unhandled exception: #{ex.message} (#{ex.class})" }
69
- $stderr.puts ex.backtrace.map { |l| " #{l}" }.join("\n")
67
+ @logger.error("writer") { (["Unhandled exception while writing entry: #{ex.message} (#{ex.class})"] + ex.backtrace).join("\n ") }
70
68
  ensure
71
69
  @entries_mutex.synchronize { @entries.unshift if entry }
72
70
  end
@@ -91,30 +89,100 @@ class Syslogstash::LogstashWriter
91
89
  until done
92
90
  if @current_server
93
91
  begin
94
- debug { "Using current server" }
92
+ @logger.debug("writer") { "Using current server #{server_id(@current_server)}" }
95
93
  yield @current_server
96
94
  done = true
97
95
  rescue SystemCallError => ex
98
96
  # Something went wrong during the send; disconnect from this
99
97
  # server and recycle
100
- debug { "Error while writing to current server: #{ex.message} (#{ex.class})" }
98
+ @logger.debug("writer") { "Error while writing to current server: #{ex.message} (#{ex.class})" }
101
99
  @current_server.close
102
100
  @current_server = nil
103
101
  sleep 0.1
104
102
  end
105
103
  else
104
+ candidates = resolve_server_name
105
+
106
106
  begin
107
- # Pick another server to connect to at random
108
- next_server = @servers.sort { rand }.first
109
- debug { "Trying to connect to #{next_server.to_s}" }
110
- @current_server = TCPSocket.new(next_server.hostname, next_server.port)
107
+ next_server = candidates.shift
108
+
109
+ if next_server
110
+ @logger.debug("writer") { "Trying to connect to #{next_server.to_s}" }
111
+ @current_server = TCPSocket.new(next_server.hostname, next_server.port)
112
+ else
113
+ @logger.debug("writer") { "Could not connect to any server; pausing before trying again" }
114
+ @current_server = nil
115
+ sleep 5
116
+ end
111
117
  rescue SystemCallError => ex
112
- # Connection failed for any number of reasons; try again
113
- debug { "Failed to connect to #{next_server.to_s}: #{ex.message} (#{ex.class})" }
118
+ # Connection failed for any number of reasons; try the next one in the list
119
+ @logger.warn("writer") { "Failed to connect to #{next_server.to_s}: #{ex.message} (#{ex.class})" }
114
120
  sleep 0.1
115
121
  retry
116
122
  end
117
123
  end
118
124
  end
119
125
  end
126
+
127
+ def server_id(s)
128
+ pa = s.peeraddr
129
+ if pa[0] == "AF_INET6"
130
+ "[#{pa[3]}]:#{pa[1]}"
131
+ else
132
+ "#{pa[3]}:#{pa[1]}"
133
+ end
134
+ end
135
+
136
+ def resolve_server_name
137
+ return [static_target] if static_target
138
+
139
+ # The IPv6 literal case should have been taken care of by
140
+ # static_target, so the only two cases we have to deal with
141
+ # here are specified-port (assume A/AAAA) or no port (assume SRV).
142
+ if @server_name =~ /:/
143
+ host, port = @server_name.split(":", 2)
144
+ addrs = Resolv::DNS.new.getaddresses(host)
145
+ if addrs.empty?
146
+ @logger.warn("writer") { "No addresses resolved for server_name #{host.inspect}" }
147
+ end
148
+ addrs.map { |a| Target.new(a.to_s, port.to_i) }
149
+ else
150
+ # SRV records ftw
151
+ [].tap do |list|
152
+ left = Resolv::DNS.new.getresources(@server_name, Resolv::DNS::Resource::IN::SRV)
153
+ if left.empty?
154
+ @logger.warn("writer") { "No SRV records found for server_name #{@server_name.inspect}" }
155
+ end
156
+ until left.empty?
157
+ prio = left.map { |rr| rr.priority }.uniq.min
158
+ candidates = left.select { |rr| rr.priority == prio }
159
+ left -= candidates
160
+ candidates.sort_by! { |rr| [rr.weight, rr.target.to_s] }
161
+ until candidates.empty?
162
+ selector = rand(candidates.inject(1) { |n, rr| n + rr.weight })
163
+ chosen = candidates.inject(0) do |n, rr|
164
+ break rr if n + rr.weight >= selector
165
+ n + rr.weight
166
+ end
167
+ candidates.delete(chosen)
168
+ list << Target.new(chosen.target.to_s, chosen.port)
169
+ end
170
+ end
171
+ end
172
+ end
173
+ end
174
+
175
+ def static_target
176
+ @static_target ||= begin
177
+ if @server_name =~ /\A(.*):(\d+)\z/
178
+ begin
179
+ Target.new(IPAddr.new($1).to_s, $2.to_i)
180
+ rescue ArgumentError
181
+ # Whatever is on the LHS isn't a recognisable address;
182
+ # assume hostname and continue
183
+ nil
184
+ end
185
+ end
186
+ end
187
+ end
120
188
  end
@@ -1,40 +1,33 @@
1
- require 'rack'
2
- require 'prometheus/middleware/exporter'
3
- require 'rack/handler/webrick'
1
+ require 'frankenstein/server'
4
2
  require 'logger'
5
3
 
6
4
  class Syslogstash::PrometheusExporter
7
5
  attr_reader :thread
8
6
 
9
- def initialize
10
- @msg_in = prom.counter(:syslogstash_messages_received, "The number of syslog messages received from each log socket")
11
- @msg_out = prom.counter(:syslogstash_messages_sent, "The number of logstash messages sent to each logstash server")
12
- @lag = prom.gauge(:syslogstash_lag_ms, "How far behind we are in relaying messages")
13
- @queue = prom.gauge(:syslogstash_queue_size, "How many messages are queued to be sent")
7
+ def initialize(cfg)
8
+ @stats_server = Frankenstein::Server.new(port: 9159, logger: cfg.logger, metrics_prefix: "syslogstash_server")
9
+
10
+ @msg_in = prom.counter(:syslogstash_messages_received_total, "The number of syslog messages received from the log socket")
11
+ @msg_out = prom.counter(:syslogstash_messages_sent_total, "The number of logstash messages sent to each logstash server")
12
+ @lag = prom.gauge(:syslogstash_last_relayed_message_timestamp, "When the last message that was successfully relayed to logstash was originally received")
13
+ @queue = prom.gauge(:syslogstash_queue_size, "How many messages are currently in the queue to be sent")
14
+ @dropped = prom.counter(:syslogstash_messages_dropped, "How many messages have been dropped from the backlog queue")
15
+
14
16
  @q_mutex = Mutex.new
15
- @dropped = prom.counter(:syslogstash_messages_dropped, "How many syslog messages have been dropped from the backlog queue")
17
+
18
+ @lag.set({}, 0)
19
+ @queue.set({}, 0)
16
20
  end
17
21
 
18
- def received(socket, stamp)
22
+ def received(socket)
19
23
  @msg_in.increment(socket_path: socket)
20
- @q_mutex.synchronize { @queue.set({}, (@queue.get({}) || 0) + 1) }
21
-
22
- if @most_recent_received.nil? || @most_recent_received < stamp
23
- @most_recent_received = stamp
24
-
25
- refresh_lag
26
- end
24
+ @q_mutex.synchronize { @queue.set({}, @queue.get({}) + 1) }
27
25
  end
28
26
 
29
27
  def sent(server, stamp)
30
28
  @msg_out.increment(logstash_server: server)
31
29
  @q_mutex.synchronize { @queue.set({}, @queue.get({}) - 1) }
32
-
33
- if @most_recent_sent.nil? || @most_recent_sent < stamp
34
- @most_recent_sent = stamp
35
-
36
- refresh_lag
37
- end
30
+ @lag.set({}, stamp.to_f)
38
31
  end
39
32
 
40
33
  def dropped
@@ -43,28 +36,12 @@ class Syslogstash::PrometheusExporter
43
36
  end
44
37
 
45
38
  def run
46
- @thread = Thread.new do
47
- app = Rack::Builder.new
48
- app.use Prometheus::Middleware::Exporter
49
- app.run ->(env) { [404, {'Content-Type' => 'text/plain'}, ['Nope']] }
50
-
51
- logger = Logger.new($stderr)
52
- logger.level = Logger::INFO
53
- logger.formatter = proc { |s, t, p, m| "[Syslogstash::PrometheusExporter::WEBrick] #{m}\n" }
54
-
55
- Rack::Handler::WEBrick.run app, Host: '::', Port: 9159, Logger: logger, AccessLog: []
56
- end
39
+ @stats_server.run
57
40
  end
58
41
 
59
42
  private
60
43
 
61
44
  def prom
62
- Prometheus::Client.registry
63
- end
64
-
65
- def refresh_lag
66
- if @most_recent_received && @most_recent_sent
67
- @lag.set({}, ((@most_recent_received.to_f - @most_recent_sent.to_f) * 1000).to_i)
68
- end
45
+ @stats_server.registry
69
46
  end
70
47
  end
@@ -1,30 +1,15 @@
1
- require_relative 'worker'
2
-
3
1
  # A single socket reader.
4
2
  #
5
3
  class Syslogstash::SyslogReader
6
- include Syslogstash::Worker
7
-
8
- attr_reader :file
9
-
10
- def initialize(file, config, logstash, metrics)
11
- @file, @logstash, @metrics = file, logstash, metrics
12
- config ||= {}
4
+ attr_reader :thread
13
5
 
14
- @add_fields = config['add_fields'] || {}
15
- @relay_to = config['relay_to'] || []
16
-
17
- unless @add_fields.is_a? Hash
18
- raise ArgumentError,
19
- "add_fields parameter to socket #{file} must be a hash"
20
- end
6
+ def initialize(cfg, logstash, stats)
7
+ @file, @logstash, @stats = cfg.syslog_socket, logstash, stats
21
8
 
22
- unless @relay_to.is_a? Array
23
- raise ArgumentError,
24
- "relay_to parameter to socket #{file} must be an array"
25
- end
26
-
27
- log { "initialized syslog socket #{file} with config #{config.inspect}" }
9
+ @add_fields = cfg.add_fields
10
+ @relay_to = cfg.relay_sockets
11
+ @cfg = cfg
12
+ @logger = cfg.logger
28
13
  end
29
14
 
30
15
  # Start reading from the socket file, parsing entries, and flinging
@@ -32,33 +17,32 @@ class Syslogstash::SyslogReader
32
17
  # continuing in a separate thread.
33
18
  #
34
19
  def run
35
- debug { "#run called" }
20
+ @logger.debug("reader") { "#run called" }
36
21
 
37
22
  begin
38
23
  socket = Socket.new(Socket::AF_UNIX, Socket::SOCK_DGRAM, 0)
39
24
  socket.bind(Socket.pack_sockaddr_un(@file))
40
25
  File.chmod(0666, @file)
41
26
  rescue Errno::EEXIST, Errno::EADDRINUSE
42
- log { "socket file #{@file} already exists; deleting" }
27
+ @logger.info("reader") { "socket file #{@file} already exists; deleting" }
43
28
  File.unlink(@file) rescue nil
44
29
  retry
45
- rescue SystemCallError
46
- log { "Error while trying to bind to #{@file}" }
47
- raise
30
+ rescue StandardError => ex
31
+ raise ex.class, "Error while trying to bind to #{@file}: #{ex.message}", ex.backtrace
48
32
  end
49
33
 
50
- @worker = Thread.new do
34
+ @thread = Thread.new do
51
35
  begin
52
36
  loop do
53
37
  msg = socket.recvmsg
54
- debug { "Message received: #{msg.inspect}" }
55
- @metrics.received(@file, Time.now)
56
- process_message msg.first.chomp
38
+ @logger.debug("reader") { "Message received: #{msg.inspect}" }
39
+ @stats.received(@file)
57
40
  relay_message msg.first
41
+ process_message msg.first.chomp
58
42
  end
59
43
  ensure
60
44
  socket.close
61
- log { "removing socket file #{@file}" }
45
+ @logger.debug("reader") { "removing socket file #{@file}" }
62
46
  File.unlink(@file) rescue nil
63
47
  end
64
48
  end
@@ -103,7 +87,7 @@ class Syslogstash::SyslogReader
103
87
 
104
88
  @logstash.send_entry(log_entry)
105
89
  else
106
- log { "Unparseable message: #{msg}" }
90
+ @logger.warn("reader") { "Unparseable message: #{msg.inspect}" }
107
91
  end
108
92
  end
109
93
 
@@ -118,13 +102,19 @@ class Syslogstash::SyslogReader
118
102
  e.merge!(h.delete_if { |k,v| v.nil? })
119
103
  e.merge!(@add_fields)
120
104
 
121
- debug { "Log entry is: #{e.inspect}" }
105
+ @logger.debug("reader") { "Complete log entry is: #{e.inspect}" }
122
106
  end
123
107
  end
124
108
 
125
109
  def relay_message(msg)
126
110
  @currently_failed ||= {}
127
111
 
112
+ if @cfg.relay_to_stdout
113
+ # This one's easy
114
+ puts msg.sub(/\A<\d+>/, '')
115
+ $stdout.flush
116
+ end
117
+
128
118
  @relay_to.each do |f|
129
119
  s = Socket.new(Socket::AF_UNIX, Socket::SOCK_DGRAM, 0)
130
120
  begin
@@ -133,25 +123,34 @@ class Syslogstash::SyslogReader
133
123
  # Socket doesn't exist; we don't care enough about this to bother
134
124
  # reporting it. People will figure it out themselves soon enough.
135
125
  rescue StandardError => ex
136
- log { "Error while connecting to relay socket #{f}: #{ex.message} (#{ex.class})" }
126
+ unless @currently_failed[f]
127
+ @logger.warn("reader") { "Error while connecting to relay socket #{f}: #{ex.message} (#{ex.class})" }
128
+ @currently_failed[f] = true
129
+ end
137
130
  next
138
131
  end
139
132
 
140
133
  begin
134
+ # We really, *really* don't want to block the world just because
135
+ # whoever's on the other end of the relay socket can't process
136
+ # messages quick enough.
141
137
  s.sendmsg_nonblock(msg)
142
138
  if @currently_failed[f]
143
- log { "Backlog on socket #{f} has cleared; messages are being delivered again" }
139
+ @logger.info("reader") { "Error on socket #{f} has cleared; messages are being delivered again" }
144
140
  @currently_failed[f] = false
145
141
  end
146
142
  rescue Errno::ENOTCONN
147
- # Socket isn't being listened to. Not our problem.
143
+ unless @currently_failed[f]
144
+ @logger.debug("reader") { "Nothing is listening on socket #{f}" }
145
+ @currently_failed[f] = true
146
+ end
148
147
  rescue IO::EAGAINWaitWritable
149
148
  unless @currently_failed[f]
150
- log { "Socket #{f} is backlogged; messages to this socket from socket #{@file} are being discarded undelivered" }
149
+ @logger.warn("reader") { "Socket #{f} is currently backlogged; messages to this socket are now being discarded undelivered" }
151
150
  @currently_failed[f] = true
152
151
  end
153
152
  rescue StandardError => ex
154
- log { "Failed to relay message to socket #{f} from #{@file}: #{ex.message} (#{ex.class})" }
153
+ @logger.warn("reader") { (["Failed to relay message to socket #{f} from #{@file}: #{ex.message} (#{ex.class})"] + ex.backtrace).join("\n ") }
155
154
  end
156
155
  end
157
156
  end
data/syslogstash.gemspec CHANGED
@@ -23,7 +23,7 @@ Gem::Specification.new do |s|
23
23
 
24
24
  s.required_ruby_version = ">= 2.1.0"
25
25
 
26
- s.add_runtime_dependency 'prometheus-client', '>= 0.7'
26
+ s.add_runtime_dependency 'frankenstein'
27
27
  s.add_runtime_dependency 'rack'
28
28
 
29
29
  s.add_development_dependency 'bundler'
metadata CHANGED
@@ -1,29 +1,29 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: syslogstash
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.3.0
4
+ version: 2.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Matt Palmer
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2018-01-05 00:00:00.000000000 Z
11
+ date: 2018-04-21 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
- name: prometheus-client
14
+ name: frankenstein
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
17
  - - ">="
18
18
  - !ruby/object:Gem::Version
19
- version: '0.7'
19
+ version: '0'
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
24
  - - ">="
25
25
  - !ruby/object:Gem::Version
26
- version: '0.7'
26
+ version: '0'
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: rack
29
29
  requirement: !ruby/object:Gem::Requirement
@@ -193,14 +193,16 @@ extensions: []
193
193
  extra_rdoc_files: []
194
194
  files:
195
195
  - ".gitignore"
196
+ - Dockerfile
196
197
  - LICENCE
198
+ - Makefile
197
199
  - README.md
198
200
  - bin/syslogstash
199
201
  - lib/syslogstash.rb
202
+ - lib/syslogstash/config.rb
200
203
  - lib/syslogstash/logstash_writer.rb
201
204
  - lib/syslogstash/prometheus_exporter.rb
202
205
  - lib/syslogstash/syslog_reader.rb
203
- - lib/syslogstash/worker.rb
204
206
  - syslogstash.gemspec
205
207
  homepage: https://github.com/discourse/syslogstash
206
208
  licenses: []
@@ -1,34 +0,0 @@
1
- # Common code shared between both readers and writers.
2
- #
3
- module Syslogstash::Worker
4
- # If you ever want to stop a reader, here's how.
5
- def stop
6
- if @worker
7
- @worker.kill
8
- @worker.join
9
- @worker = nil
10
- end
11
- end
12
-
13
- def thread
14
- @worker
15
- end
16
-
17
- # If you want to wait for a reader to die, here's how.
18
- #
19
- def wait
20
- @worker.join
21
- end
22
-
23
- private
24
-
25
- def log
26
- $stderr.puts "[#{self.class}] #{yield.to_s}"
27
- end
28
-
29
- def debug
30
- if ENV['DEBUG_SYSLOGSTASH']
31
- $stderr.puts "[#{self.class}] #{yield.to_s}"
32
- end
33
- end
34
- end