fluent-plugin-secure-forward-addproxy 0.3.3dev2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 2855715708739b31522d45e2bf0f68e299aafc97
+  data.tar.gz: fb0b48ccc7435b6da6e4add699d922bd23736fa3
+SHA512:
+  metadata.gz: 195d98a4b9e40bca34c986d3435844f2f7572ba37469a4be13e4181c79f880903094d91fde42c16d770f8ef850161a12c53543a4df0649d533bb9041384b5547
+  data.tar.gz: 3dd3e8f005e2051a4a54f3a9c0cc0a7cbba68736c150c711f594006ce49adb7fd3d197dfa184aa4ac6592eff679244bf6c7175722c8606e98f49eaaad44c821a

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.travis.yml

@@ -0,0 +1,4 @@
+language: ruby
+rvm:
+  - 2.2.2
+before_install: gem install bundler -v 1.10.3

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fluent-plugin-secure-forward.gemspec
+gemspec

data/README.md

@@ -0,0 +1,444 @@
+# fluent-plugin-secure-forward
+
+[Fluentd](http://fluentd.org) input/output plugin to forward fluentd messages over SSL with authentication.
+
+This plugin makes you to be able to:
+
+ * protect your data from others in transferring with SSL
+   * with certificate signed and registered correctly/publicly
+   * with private CA certificates generated by users
+   * with automatically generated and self-signed certificates **in vulnerable way**
+ * authenticate by shared\_key check from both of client(out\_secure\_forward) and server(in\_secure\_forward)
+ * authenticate with username / password pairs
+
+## Installation
+install with gem or fluent-gem command as:
+
+```
+ ### native gem
+$ gem install fluent-plugin-secure-forward
+ 
+ ### fluentd gem
+$ fluent-gem install fluent-plugin-secure-forward
+```
+
+### Using SSL certificates issued from trusted CA
+
+To communicate over SSL with valid certificate issued from public CA, configure params below for input plugin:
+
+* `secure`: set `yes` or `true`
+* `cert_path`: set path of certificate file issued from CA
+* `private_key_path`: set path of private key file
+* `private_key_passphrase`: set passphrase of private key
+
+```apache
+<source>
+  type secure_forward
+  
+  # bind 0.0.0.0 # default
+  # port 24284 # default
+  self_hostname server.fqdn.example.com
+  shared_key    secret_string
+  
+  secure yes
+  
+  cert_path        /path/for/certificate/cert.pem
+  private_key_path /path/for/certificate/key.pem
+  private_key_passphrase secret_foo_bar_baz
+</source>
+```
+
+For output plugin, specify just 2 options below:
+
+* `secure`: set `yes` or `true`
+* `enable_strict_verification`: specify `yes` or `true` to verify FQDN of servers (input plugin)
+
+```apache
+<match secret.data.**>
+  type secure_forward
+  
+  self_hostname client.fqdn.local
+  shared_key    secret_string
+  
+  secure yes
+  enable_strict_verification yes
+  
+  <server>
+    host server.fqdn.example.com  # or IP
+    # port 24284
+  </server>
+  <server>
+    host 203.0.113.8 # ip address to connect
+    hostlabel server.fqdn.example.com # specify hostlabel for FQDN verification if ipaddress is used for host
+  </server>
+</match>
+```
+
+### Using private CA file and key
+
+This plugin has a simple utility command to generate private CA cert/key files just for secure-forward.
+
+```
+$ secure-forward-ca-generate /path/for/dir/of/certs "passphrase for private CA secret key"
+```
+
+This command generates `ca_cert.pem` and `ca_key.pem` on `/path/for/dir/of/certs`. For SSL communication with private CA, users must deploy both files for input plugins, and also must deploy `ca_cert.pem` for output plugins.
+And then, configure Fluentd with these files and the passphrase. With this configuration, server certificates are automatically generated and issued by private CA.
+
+```apache
+<source>
+  type secure_forward
+  
+  # bind 0.0.0.0 # default
+  # port 24284 # default
+  self_hostname myserver.local
+  shared_key    secret_string
+  
+  secure yes
+  
+  ca_cert_path        /path/for/certificate/ca_cert.pem
+  ca_private_key_path /path/for/certificate/ca_key.pem
+  ca_private_key_passphrase passphrase for private CA secret key
+</source>
+```
+
+For output plugin, specify just 2 options below:
+
+* `secure`: set `yes` or `true`
+* `enable_strict_verification`: specify `yes` or `true`
+
+```apache
+<match secret.data.**>
+  type secure_forward
+  
+  self_hostname myclient.local
+  shared_key    secret_string
+  
+  secure yes
+  ca_cert_path /path/for/certificate/ca_cert.pem
+  # enable_strict_verification yes
+  
+  <server>
+    host server.fqdn.example.com  # or IP
+    # port 24284
+  </server>
+  <server>
+    host 203.0.113.8 # ip address to connect
+    hostlabel server.fqdn.example.com # specify hostlabel for FQDN verification if ipaddress is used for host
+  </server>
+</match>
+```
+
+### Using insecure self-signed certificates
+
+**This is very dangerous and vulnerable to man-in-the-middle attacks**
+
+For just testing or data center internal communications, this plugin has a feature to communicate without any verification of certificates. Turn `secure` option to `false` to use this feature.
+
+```apache
+<source>
+  type secure_forward
+  
+  self_hostname myserver.local
+  shared_key    secret_string
+  
+  secure no
+</source>
+```
+
+Configure output plugin just same way:
+
+```apache
+<match data.**>
+  type secure_forward
+  
+  self_hostname myclient.local
+  shared_key    secret_string
+  
+  secure no
+  
+  <server>
+    host server.fqdn.example.com  # or IP
+  </server>
+</match>
+```
+
+In this mode, output plugin cannot verify peer node of connections. Man-in-the-middle attackers can spoof messages from output plugins under many various situations.
+
+## Configuration
+
+### SecureForwardInput
+
+Default settings:
+  * listen 0.0.0.0:24284
+    * `bind 192.168.0.101`
+    * `port 24284`
+  * allow to accept from any sources
+  * allow to connect without authentications
+  * use certificate automatically generated
+    * `generate_private_key_length 2048`
+    * `generate_cert_country  US`
+    * `generate_cert_state    CA`
+    * `generate_cert_locality Mountain View`
+    * `generate_cert_common_name SAME_WITH_SELF_HOSTNAME_PARAMETER`
+  * use TLSv1.2
+  
+Minimal configurations like below:
+
+```apache
+<source>
+  type secure_forward
+  shared_key         secret_string
+  self_hostname      server.fqdn.local  # This fqdn is used as CN (Common Name) of certificates
+  
+  secure yes
+  # and configurations for certs
+</source>
+```
+
+To check username/password from clients, like this:
+
+```apache
+<source>
+  type secure_forward
+  shared_key         secret_string
+  self_hostname      server.fqdn.local
+  
+  secure yes
+  # and configurations for certs
+  
+  authentication     yes # Deny clients without valid username/password
+  <user>
+    username tagomoris
+    password foobar012
+  </user>
+  <user>
+    username frsyuki
+    password yakiniku
+  </user>
+</source>
+```
+
+To deny unknown source IP/hosts:
+
+```apache
+<source>
+  type secure_forward
+  shared_key         secret_string
+  self_hostname      server.fqdn.local
+  
+  secure yes
+  # and configurations for certs
+  
+  allow_anonymous_source no  # Allow to accept from nodes of <client>
+  <client>
+    host 192.168.10.30
+  </client>
+  <client>
+    host your.host.fqdn.local
+    # wildcard (ex: *.host.fqdn.local) NOT Supported now
+  </client>
+  <client>
+    network 192.168.16.0/24 # network address specification
+  </client>
+</source>
+```
+
+You can use both of username/password check and client check:
+
+```apache
+<source>
+  type secure_forward
+  shared_key         secret_string
+  self_hostname      server.fqdn.local
+  
+  secure yes
+  # and configurations for certs
+  
+  allow_anonymous_source no  # Allow to accept from nodes of <client>
+  authentication         yes # Deny clients without valid username/password
+  <user>
+    username tagomoris
+    password foobar012
+  </user>
+  <user>
+    username frsyuki
+    password sukiyaki
+  </user>
+  <user>
+    username repeatedly
+    password sushi
+  </user>
+  <client>
+    host 192.168.10.30      # allow all users to connect from 192.168.10.30
+  </client>
+  <client>
+    host  192.168.10.31
+    users tagomoris,frsyuki # deny repeatedly from 192.168.10.31
+  </client>
+  <client>
+    host 192.168.10.32
+    shared_key less_secret_string # limited shared_key for 192.168.10.32
+    users      repeatedly         # and repatedly only
+  </client>
+</source>
+```
+
+### SecureForwardOutput
+
+Minimal configurations like this:
+
+```apache
+<match secret.data.**>
+  type secure_forward
+  shared_key secret_string
+  self_hostname client.fqdn.local
+  
+  secure yes
+  # and configurations for certs/verification
+  
+  <server>
+    host server.fqdn.local  # or IP
+    # port 24284
+  </server>
+</match>
+```
+
+Without hostname ACL (and it's not implemented yet), `self_hostname` is not checked in any state. `${hostname}` placeholder is available for such cases.
+
+```apache
+<match secret.data.**>
+  type secure_forward
+  shared_key secret_string
+  self_hostname ${hostname}
+  
+  secure yes
+  # and configurations for certs/verification
+  
+  <server>
+    host server.fqdn.local  # or IP
+    # port 24284
+  </server>
+</match>
+```
+
+When specified 2 or more `<server>`, this plugin uses these nodes in simple round-robin order. And servers with `standby yes` will be selected until all of non-standby servers goes down.
+
+If server requires username/password, set `username` and `password` in `<server>` section:
+
+```apache
+<match secret.data.**>
+  type secure_forward
+  shared_key secret_string
+  self_hostname client.fqdn.local
+  
+  secure yes
+  # and configurations for certs/verification
+  
+  <server>
+    host      first.fqdn.local
+    hostlabel server.fqdn.local
+    username  repeatedly
+    password  sushi
+  </server>
+  <server>
+    host      second.fqdn.local
+    hostlabel server.fqdn.local
+    username  sasatatsu
+    password  karaage
+  </server>
+  <server>
+    host      standby.fqdn.local
+    hostlabel server.fqdn.local
+    username  kzk
+    password  hawaii
+    standby   yes
+  </server>
+</match>
+```
+
+Specify `hostlabel` if server (`in_forward`) have different hostname (`self_host` configuration of `in_forward`) from DNS name (`first.fqdn.local`, `second.fqdn.local` or `standby.fqdn.local`). This configuration variable will be used to check common name (CN) of certifications.
+
+To specify keepalive timeouts, use `keepalive` configuration with seconds. SSL connection will be disconnected and re-connected for each 1 hour with configuration below. In Default (and with `keepalive 0`), connections will not be disconnected without any communication troubles. (This feature is for dns name updates, and SSL common key refreshing.)
+
+```apache
+<match secret.data.**>
+  type secure_forward
+  shared_key secret_string
+  self_hostname client.fqdn.local
+  
+  secure yes
+  # and configurations for certs/verification
+  
+  keepalive 3600
+  <server>
+    host server.fqdn.local  # or IP
+    # port 24284
+  </server>
+</match>
+```
+
+## Senario (developer document)
+
+* server
+  * in\_secure\_forward
+* client
+  * out\_secure\_forward
+
+### Handshake
+
+1. (client) connect to server
+  * on SSL socket handshake, checks certificate and its significate (in client)
+2. (server)
+  * check network/domain acl (if enabled)
+  * check client dns reverse lookup result (if enabled)
+  * disconnect when failed
+3. (server) send HELO
+  * ['HELO', options(hash)]
+  * options:
+    * nonce: string as nonce: used for shared key digest (required, v0.3.2 or later)
+    * auth: string or blank\_string (string: authentication required, and its salt is this value)
+    * keepalive: bool (allowed or not)
+4. (client) send PING
+  * ['PING', selfhostname, sharedkey\_salt, sha512\_hex(sharedkey\_salt + selfhostname + nonce + sharedkey), username || '', sha512\_hex(auth\_salt + username + password) || '']
+5. (server) check PING
+  * check sharedkey
+  * check username / password (if required)
+  * send PONG FAILURE if failed
+  * ['PONG', false, 'reason of authentication failure', '', '']
+6. (server) send PONG
+  * ['PONG', bool(authentication result), 'reason if authentication failed', selfhostname, sha512\_hex(salt + selfhostname + nonce + sharedkey)]
+7. (client) check PONG
+  * check sharedkey
+  * disconnect when failed
+8. connection established
+  * send data from client (until keepalive expiration)
+
+### Data transferring
+
+CONSIDER RETURN ACK OR NOT
+
+ * Current version has no ACKs
+   * only supports burst transferring (same as ForwardInput/Output)
+ * ack for each message ?
+ * pipeline mode and one-by-one mode ?
+ * data sequence number in keepalive session ?
+
+## TODO
+
+* ACK mode (protocol)
+* support disabling keepalive (input/output)
+* access control (input plugin)
+  * network acl / domain acl
+  * check connecting source ip and its dns reverse lookup result (for domaian acl)
+  * access deny on accept (against DoS)
+* pluggable authentication database (input plugin)
+  * RDBMS, LDAP, or ...
+  * Authentication by clients certificate
+* TESTS!
+
+## Copyright
+
+* Copyright (c) 2013- TAGOMORI Satoshi (tagomoris)
+* License
+  * Apache License, Version 2.0

data/Rakefile

@@ -0,0 +1,11 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+task :default => :test

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "fluent/plugin/secure/forward/v033dev2/addproxy"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/secure-forward-ca-generate

@@ -0,0 +1,34 @@
+#!/usr/bin/env ruby
+
+require 'fileutils'
+require 'fluent/plugin/secure_forward/cert_util'
+
+ca_dir, passphrase = ARGV
+
+unless ca_dir && passphrase
+  puts 'USAGE: secure-forward-ca-generate  DIR_PATH  PRIVATE_KEY_PASSPHRASE'
+  puts ''
+  exit 0
+end
+
+FileUtils.mkdir_p(ca_dir)
+
+opt = {
+  private_key_length: 2048,
+  cert_country:  'US',
+  cert_state:    'CA',
+  cert_locality: 'Mountain View',
+  cert_common_name: 'SecureForward CA',
+}
+cert, key = Fluent::SecureForward::CertUtil.generate_ca_pair(opt)
+
+key_data = key.export(OpenSSL::Cipher::Cipher.new('aes256'), passphrase)
+File.open(File.join(ca_dir, 'ca_key.pem'), 'w') do |file|
+  file.write key_data
+end
+File.open(File.join(ca_dir, 'ca_cert.pem'), 'w') do |file|
+  file.write cert.to_pem
+end
+
+puts "successfully generated: ca_key.pem, ca_cert.pem"
+puts "copy and use ca_cert.pem to client(out_secure_forward)"

data/bin/setup

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -euo pipefail
+IFS=$'\n\t'
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/example/auth_client.conf

@@ -0,0 +1,19 @@
+<source>
+  type forward
+</source>
+
+<match test.**>
+  type secure_forward
+  self_hostname client
+  #shared_key hogeposxxx0
+  shared_key wrong_shared_key
+  <server>
+    host localhost
+    shared_key hogeposxxx1
+    username tagomoris
+    password 001122
+    # password XXYYZZ
+    # password wrong_pass
+  </server>
+  flush_interval 1s
+</match>

data/example/auth_server.conf

@@ -0,0 +1,30 @@
+<source>
+  type secure_forward
+  self_hostname server
+  shared_key hogeposxxx0
+  cert_auto_generate yes
+  allow_anonymous_source no
+  authentication yes
+  <user>
+    username tagomoris
+    password 001122
+  </user>
+  <user>
+    username sugomoris
+    password 012345
+  </user>
+  <user>
+    username tagomoris
+    password XXYYZZ
+  </user>
+  <client>
+    host localhost
+    users tagomoris
+    shared_key hogeposxxx1
+    # users sugomoris
+  </client> 
+</source>
+
+<match test.**>
+  type stdout
+</match>

data/example/cert_client.conf

@@ -0,0 +1,21 @@
+<source>
+  type forward
+</source>
+
+<match test.**>
+  type secure_forward
+  secure yes
+  self_hostname client
+  #shared_key hogeposxxx0
+  shared_key wrong_shared_key
+  <server>
+    host localhost
+    hostlabel tagomoris
+    shared_key hogeposxxx1
+    username tagomoris
+    password 001122
+    # password XXYYZZ
+    # password wrong_pass
+  </server>
+  flush_interval 1s
+</match>

data/example/cert_server.conf

@@ -0,0 +1,35 @@
+<source>
+  type secure_forward
+  secure yes
+  self_hostname server
+  # self_hostname tagomoris
+  shared_key hogeposxxx0
+  cert_path        /Users/tagomoris/Documents/fluent-plugin-secure-forward/example/certs/cert.pem
+  private_key_path /Users/tagomoris/Documents/fluent-plugin-secure-forward/example/certs/key.pem
+  # blank passphrase
+  private_key_passphrase 
+  allow_anonymous_source no
+  authentication yes
+  <user>
+    username tagomoris
+    password 001122
+  </user>
+  <user>
+    username sugomoris
+    password 012345
+  </user>
+  <user>
+    username tagomoris
+    password XXYYZZ
+  </user>
+  <client>
+    host localhost
+    users tagomoris
+    shared_key hogeposxxx1
+    # users sugomoris
+  </client> 
+</source>
+
+<match test.**>
+  type stdout
+</match>

data/example/certs/cert.pem

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC9TCCAl6gAwIBAgIJAPZkY4lTv8EcMA0GCSqGSIb3DQEBBQUAMFsxCzAJBgNV
+BAYTAkpQMQ4wDAYDVQQIEwVUb2t5bzEQMA4GA1UEBxMHU2hpYnV5YTEWMBQGA1UE
+ChMNRmx1ZW50ZCBKYXBhbjESMBAGA1UEAxMJdGFnb21vcmlzMB4XDTEzMDIxNDA4
+MzQ0OVoXDTIzMDIxMjA4MzQ0OVowWzELMAkGA1UEBhMCSlAxDjAMBgNVBAgTBVRv
+a3lvMRAwDgYDVQQHEwdTaGlidXlhMRYwFAYDVQQKEw1GbHVlbnRkIEphcGFuMRIw
+EAYDVQQDEwl0YWdvbW9yaXMwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAPli
+bZUddJEJDaPza0dQElKYefGcWyN5f6FHBrv0MU29PW4+9fape3/u6Kal2knXhz7c
+ujkyoQgK7pqCOuwpTCi0Fyg2peSLVJm4lw2TS5HP/7qRbKXhx2g3FaHrs/Ug/pbQ
+6xPSy894w2QaXgkeuDLb/bhu8MHulglm/iXg9wHrAgMBAAGjgcAwgb0wHQYDVR0O
+BBYEFNWgnetVbxQlGX6euMDea7WGgWO+MIGNBgNVHSMEgYUwgYKAFNWgnetVbxQl
+GX6euMDea7WGgWO+oV+kXTBbMQswCQYDVQQGEwJKUDEOMAwGA1UECBMFVG9reW8x
+EDAOBgNVBAcTB1NoaWJ1eWExFjAUBgNVBAoTDUZsdWVudGQgSmFwYW4xEjAQBgNV
+BAMTCXRhZ29tb3Jpc4IJAPZkY4lTv8EcMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcN
+AQEFBQADgYEAai2UAUa5WAahfUp/UV/7zX7+r/QdUP0fwrrmLzodk+FS3+yS6oqQ
+tBs0K81cD3XKfoYjAqzJ1Hul6orR63wD+yrPq3FApuWKd+CJDBxJmY8MtIA0xHHn
+nfotL/TzTAEIcFVLYb8yaBA27VMstBHvE4TsbL7mA0avF3FFzxG5GqE=
+-----END CERTIFICATE-----

data/example/certs/key.pem

@@ -0,0 +1,15 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIICXQIBAAKBgQD5Ym2VHXSRCQ2j82tHUBJSmHnxnFsjeX+hRwa79DFNvT1uPvX2
+qXt/7uimpdpJ14c+3Lo5MqEICu6agjrsKUwotBcoNqXki1SZuJcNk0uRz/+6kWyl
+4cdoNxWh67P1IP6W0OsT0svPeMNkGl4JHrgy2/24bvDB7pYJZv4l4PcB6wIDAQAB
+AoGBAIGvxu7Rl4nI3HgTIQm/wReExX144whKqa2UAxOBBJa5v5VyVnSEZH3+Hqxy
++VaHJ4TwQkN2abmF/dkJulyPiVNmsAEXeYKmNOOnOuvGVYlYgRHGJ0P13oszvtKC
+mIFsL4D01FYOHMeblxGhfPQgh4UTcQtIG9gB+yPJ/JJNH7whAkEA/XPV5rxkz/8i
+BMgUHxXxv1o4CJf0exJiMjqNViydgnWyOSEGpoABbbxsN/XV2pwaG0Sythz/4AcF
+phgCJssNUQJBAPvkIALt96XTB/mlcXap1LC+bleEdiwANpgBlwxp0HlxhBrgyDyJ
+iV65FGixi6xIOOjwQbFaLupDC383L8kW3HsCQEjHcX3PTVeY2Kjs1zJR99hNzNdS
+4yZQEhiATcOYDia/K01SWXmIOmDLgXvUQPOEbc60vGilDSjEe2/FZyDCn/ECQQCY
+pfLQU64UjAL1Q1Gze9AtG/p6hwemOqrbC3uiRi3UqvpH35j5NtBM2xSHLbFbQpla
+cN8ev2xXAzJgce0/i98pAkACvTTdRqRIp/7X24tzXJlageBxXX2vBQF8PZcjdx7C
+nVOmUTBuw5JrB34ehYnoWEwMqeyU3CNgUIIgslhcAsVl
+-----END RSA PRIVATE KEY-----