fluent-plugin-postgres-flex 0.1.0.rc1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 45da1fcf4cb02a37294161935326a66098c6bfe4
+  data.tar.gz: e05ff3eb81b8015f544962e64f5de4f2339c19b0
+SHA512:
+  metadata.gz: 0f05aded3e2ca6d3b460996ac2c35f35e6f3c7ec2f4d385be31f8de047b169a5b30b8972c995b069dbc24e79f35c2fe7abf38520a52eab7188b4edae5261445e
+  data.tar.gz: c5513e6150d666d811b3db98feb9767b2e81d306bef9d8d7871866490415cf7eac768d6e254a2cae03aad103fac1ab6c00aabac383c088fa9f962dc48484a362

data/Changelog.md

@@ -0,0 +1,6 @@
+# Changelog
+
+### 0.1.1 
+2019-11-19
+
+Initial release.

data/Readme.md

@@ -0,0 +1,109 @@
+# Flexible Postgres output for fluentd
+
+An output plugin for [fluentd](https://www.fluentd.org/) for use with [Postgres](https://www.postgresql.org/) and [TimescaleDB](https://www.timescale.com/) that provides a great amount of flexibility in designing your log table structure.
+
+This plugin automatically reads your log table's schema and maps log record properties to dedicated columns if possible. All other properties will be stored in an _extra_ column of type `json` or `jsonb`. This plugin also handles Porstgres `enum` types, it will try to map string properties to enum columns.
+
+Consider following log table:
+
+```sql
+CREATE TYPE Severity AS ENUM (
+	'debug',
+	'info',
+	'notice',
+	'warning',
+	'error',
+	'alert',
+	'emergency'
+);
+
+CREATE TABLE public.logs (
+	time TIMESTAMPTZ NOT NULL,
+	severity Severity NOT NULL DEFAULT 'info',
+	message TEXT NULL,
+	extra JSONB NULL
+);
+```
+
+And a log event of the form:
+
+```
+time   2019-10-10 10:01:20.1234
+tag    'backend'
+record {"severity":"notice","message":"Starting up...","hostname":"node0123","meta":{"env":"production"}}
+```
+
+You will end up with this row inserted:
+
+| time | severity | message    | extra                                              |
+|------|----------|------------|----------------------------------------------------|
+| `2019-10-10 10:01:20.1234` | notice | Starting up... | `{"hostname":"node0123", "meta":{"env":"production"}}`
+
+The properties `severity` and `message` where mapped to their dedicated columns, all other properties landed in the `extra` column.
+
+__Note:__ The event's tag is not used in any way. I consider the tag an implementation detail of fluentd's routing system, that should not be used elsewhere. If the tag contains valuable data in your setup, you can include it as property with the `record_transformer` plugin.
+
+
+## Requirements
+
+- The `pg` Gem and it's native library.
+- A time column of type `timestamp with timezone` or `timezone without timezone` in your log table.
+- An _extra_ column of type `json` or `jsonb` to store all values without a dedicated column.
+
+
+## Configuration
+
+- __`host`__ (string, default: `localhost`)<br>
+    The database server's hostname.
+
+- __`port`__ (integer, default: `5432`)<br>
+    The database server's port.
+
+- __`database`__ (string)<br>
+    The database name.
+
+- __`username`__ (string)<br>
+    The database user name.
+
+- __`password`__ (string)<br>
+    The database user's password.
+
+- __`table`__ (string)<br>
+    The name of the log table.
+
+- __`time_column`__ (string, default: `time`)<br>
+    The column name to store the timestamp of log events. Must be of type `timestamp with timezone` or `timezone without timezone`.
+
+- __`extra_column`__ (string, default: `extra`)<br>
+    The column name to store excess properties without a dedicated column. Must be of type `json` or `jsonb`.
+
+
+## Value coercion
+
+This plugin tries to coerce all values in a meaningful way
+
+| column type  | value type | coercion rule                                                     |
+|-             |-           |-                                                                  |
+| timestamp    | `string`   | Parse as RFC3339 string                                           |
+|              | `number`   | Interpret as seconds since Unix epoch (with fractions)            |
+|              | _others_   | _undefined, place in extra columns_                               |
+| text         | _all_      | Convert to JSON string                                            |
+| boolean      | `string`   | Interpret `"t"`, `"true"` (any case) as `true`, `false` otherwise |
+|              | `number`   | Interpret `0` as `false`, other values as `true`                  |
+|              | _others_   | _undefined, place in extra columns_                               |
+| real numbers | `boolean`  | Interpret `true` as `1.0`, `false` as `0.0`                       |
+|              | `string`   | Parse as decimal (with fractions)                                 |
+|              | _others_   | _undefined, place in extra columns_                               |
+| integers     | `boolean`  | Interpret `true` as `1`, `false` as `0`                           |
+|              | `string`   | Parse as decimal (without fractions)                              |
+| json         | _all_      | Convert to JSON string                                            |
+
+
+## Log table design considerations
+
+- Since you want to avoid losing log events, your log table should be designed in a way that it is
+(almost) impossible to error at inserting data. This means that all columns should be either _nullable_ or provide a default value. The only exception is the time column, which is guaranteed to be filled with the event's time stamp.
+
+- You may or may not need a primary key. For general use, a primary key is not really necessary, since you `select` and `delete` events only in bulk, not individually.
+
+- Keep that in mind that the `timestamp` value type provides microsecond precision. This is good enough for many use cases but might not be enough for yours.

data/lib/fluent/plugin/out_postgres-flex.rb

@@ -0,0 +1,212 @@
+require 'fluent/plugin/output'
+require 'fluent/time'
+require 'pg'
+require 'oj'
+require 'date'
+
+module Fluent::Plugin
+	class PostgresFlexOutput < Output
+		Fluent::Plugin.register_output('postgres-flex', self)
+
+		config_param :host, :string, default: 'localhost'
+		config_param :port, :integer, default: 5432
+		config_param :database, :string
+		config_param :username, :string
+		config_param :password, :string
+		config_param :table, :string
+		config_param :time_column, :string, default: 'time'
+		config_param :extra_column, :string, default: 'extra'
+
+		config_section :buffer do
+			config_set_default :flush_mode, :immediate
+		end
+
+		TimestampFormat = '%Y-%m-%d %H:%M:%S.%N %z'
+		TimestampFormatter = Fluent::TimeFormatter.new(TimestampFormat)
+		OjOptions = { mode: :strict }
+
+		def start
+			super
+			reconnect()
+		end
+
+		def stop
+			super
+			@db.finish
+		end
+
+		def write(chunk)
+			values = []
+			chunk.each { |time, record| values << record_to_values(time, record) }
+
+			begin
+				@db.async_exec("INSERT INTO #{ @db.quote_ident(@table) } #{ @value_names } VALUES #{ values.join(',') }")
+			rescue PG::UnableToSend => err
+				reconnect()
+				throw err
+			end
+		end
+
+		private
+
+		def reconnect
+			@db = PG::Connection.new(
+				:host => @host,
+				:port => @port,
+				:dbname => @database,
+				:user => @username,
+				:password => @password
+			)
+			@schema, @value_names = parse_schema(@db)
+		end
+
+		# Convert a single record to a postgres value string
+		#
+		# All values that have a dedicated column will be coerced and stored there. If coercion fails,
+		# the value will be retained in the _extra_ column and the default value will be used.
+		def record_to_values(eventTime, record)
+			direct_fields = []
+
+			@schema.each_pair { |key, type|
+				value = coerce_value(record[key], type)
+
+				if value.nil?
+					log.warn "Could not coerce value #{record[key].inspect} to required type #{type.inspect}"
+				else
+					direct_fields << value
+					record.delete(key)
+				end
+			}
+
+			time = @db.escape_literal(TimestampFormatter.format_with_subsec(eventTime))
+			extras = @db.escape_literal(Oj.dump(record, OjOptions))
+
+			return "(#{ time },#{ direct_fields.join(',') },#{ extras })"
+		end
+
+		# Coerce a single value to the type requiered by the database column
+		#
+		# @return The coerced value or nil, if the value could not be coerced
+		def coerce_value(v, type)
+			if v.nil?
+				'DEFAULT'
+			else
+				case type
+				when :timestamp
+					case v
+					when String
+						# Parse as RFC3339
+						@db.escape_literal(DateTime.rfc3339(v).to_time.utc.strfrm(TimestampFormat))
+					when Numeric
+						# Interpret as Unix time: seconds (with fractions) since epoch
+						@db.escape_literal(Time.at(v).utc.strfrm(TimestampFormat))
+					else nil
+					end
+				when :string
+					@db.escape_literal(Oj.dump(v, OjOptions))
+				when :boolean
+					case v
+					when TrueClass; 'true'
+					when FalseClass; 'false'
+					when String
+						# Accept 't', 'T', 'true', 'TRUE', 'True'..., 1 as true, false otherwise
+						(v.downcase == 't' || v.downcase == 'true' || v == '1').to_s
+					when Numeric
+						v != 0
+					else nil
+					end
+				when :integer
+					case v
+					when TrueClass; '1' # Accept true as 1
+					when FalseClass; '0' # Accept false as 0
+					when String; v.to_i(10).to_s # Parse string as decimal
+					when Integer; v.to_s
+					else nil
+					end
+				when :float
+					case
+					when Float; v.to_s
+					when String; v.to_f.to_s # Parse string as float
+					else nil
+					end
+				when :json
+					begin
+						@db.escape_literal(Oj.dump(v, OjOptions))
+					rescue Oj::Error => e
+						# TODO log this
+						nil
+					end
+				when Array # enums
+					return @db.escape_literal(v) if type.include?(v)
+				else nil
+				end
+			end
+		end
+
+		# Parse postgres database schema and build a hash of column_name => type
+		def parse_schema(db)
+			# Map enum_name: String => enum_values: String[]
+			enums = db.async_exec(
+				'SELECT DISTINCT (n.nspname||\'.\'||t.typname) AS "name", e.enumlabel as "value"' +
+				'	FROM pg_type t' +
+				'	JOIN pg_enum e on t.oid = e.enumtypid' +
+				'	JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace'
+			).reduce({}) { |map, row|
+				name = row['name']
+				map[name] = [] unless map[name]
+				map[name] << row['value']
+
+				map
+			}
+
+			# Map column_name: String => type: Symbol|String[]
+			schema = db.async_exec(
+				'SELECT column_name, ' +
+				'	(CASE WHEN data_type != \'USER-DEFINED\' THEN data_type ELSE (udt_schema||\'.\'||udt_name) END) as "type"' +
+				"	FROM information_schema.columns WHERE table_name = #{ @db.escape_literal(@table) }"
+			).reduce({}) { |map, row|
+				name = row['column_name']
+				type = case row['type']
+					when 'timestamp with time zone'; :timestamp
+					when 'timestamp without time zone'; :timestamp
+					when 'text'; :string
+					when 'character varying'; :string
+					when 'character'; :string
+					when 'boolean'; :boolean
+					when 'smallint'; :integer
+					when 'integer'; :integer
+					when 'bigint'; :integer
+					when 'decimal'; :float
+					when 'numeric'; :float
+					when 'real'; :float
+					when 'double precision'; :float
+					when 'json'; :json
+					when 'jsonb'; :json
+					else enums[row['type']] # Is enum?
+				end
+
+				if type.nil?
+					log.warn "Unhandled column type '#{type}'"
+				else
+					if name == @time_column
+						if type != :timestamp
+							raise Fluent::ConfigError.new('time column must be of type "timestamp with/without timestamp"')
+						end
+					elsif name == @extra_column
+						if type != :json
+							raise Fluent::ConfigError.new('extra column must be of type "json/jsonb"')
+						end
+					else
+						map[name] = type
+					end
+				end
+
+				map
+			}
+
+			value_names = "(#{ @time_column },#{ schema.keys.join(',') },#{ @extra_column })"
+
+			return schema.freeze, value_names.freeze
+		end
+	end
+end

metadata

@@ -0,0 +1,47 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-postgres-flex
+version: !ruby/object:Gem::Version
+  version: 0.1.0.rc1
+platform: ruby
+authors:
+- André Wachter
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2019-11-19 00:00:00.000000000 Z
+dependencies: []
+description: Store fluentd structured log data in Postgres and TimescaleDB.
+email: rubygems@anfe.ma
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- Changelog.md
+- Readme.md
+- lib/fluent/plugin/out_postgres-flex.rb
+homepage: https://github.com/anfema/fluent-plugin-postgres-flex
+licenses:
+- Apache-2.0
+metadata:
+  source_code_uri: https://github.com/anfema/fluent-plugin-postgres-flex
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">"
+    - !ruby/object:Gem::Version
+      version: 1.3.1
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.2.3
+signing_key: 
+specification_version: 4
+summary: A fluentd plugin for storing logs in Postgres and TimescaleDB
+test_files: []