fluent-plugin-shodan 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: c5065dc8260c9015eebe3ab7e1095d050fdb64b3fa0d0168c414342e31f7960d
+  data.tar.gz: b26a7a5b53c56cc6469433eb5867749ec26c7298958294dcf7c6c5b2f8a335e0
+SHA512:
+  metadata.gz: 3244f0dd4e8c9d916a83220e2a534631f3c413a15b68ce985dc93a2632691a75b4147c838a2a871136a58c0e753b701304faf2cf86e6b18bd421302162836ac7
+  data.tar.gz: 14b471034818688f951fdb36d0102dccd63bd8f3a6437e6ee8e44de9d8a0d799190241250aaacd7aa631e5a4ab467f36d11ad4e59e2615cbfccfb33e92db174a

data/AUTHORS

@@ -0,0 +1 @@
+srilumpa <marcandre.doll@gmail.com>

data/README.md

@@ -0,0 +1,125 @@
+# fluent-plugin-shodan
+
+[![Unit tests](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml)[![Ruby Gem](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/rubygem-push.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/rubygem-push.yml)[![GPR](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/gpr-push.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/gpr-push.yml)
+
+The [Shodan](https://www.shodan.io/) plugin offers [Fluentd](https://fluentd.org/) capacities to gather data from shodan and send them to whatever system you want (on the condition Fluentd has an [output plugin](https://docs.fluentd.org/output) fitting your needs).
+
+The Shodan plugin can adress three ways of gathering data
+
+- by querying the [Search API](https://developer.shodan.io/api)
+- by consuming the [Stream API](https://developer.shodan.io/api/stream) (WIP)
+- or by consuming the [Alert API](https://developer.shodan.io/api/stream) (WIP)
+
+The outputed "logs" follow the Shodan [Banner specification](https://datapedia.shodan.io/).
+
+A valid API key will be necessary for this plugin to work. The Shodan Search plugin will work with a _Free_ account with limited functionnalities, but the Shodans Stream and the Shodan Alert plugins will need a subscription plan to work.
+
+This plugin relies on the
+
+## Installation
+
+### RubyGems
+
+```
+$ gem install fluent-plugin-shodan
+```
+
+### Bundler
+
+Add following line to your Gemfile:
+
+```ruby
+gem "fluent-plugin-shodan"
+```
+
+And then execute:
+
+```
+$ bundle
+```
+
+## Shodan Search
+
+### Example configuration
+
+```
+<source>
+  @type shodan_search
+  interval 15m
+  tag shodan.ssh
+  query ssh
+  api_key 1234567890AZERTYUIOP
+</source>
+```
+
+### How it works
+
+When Fluentd is started with `in_shodan_search`, it will create a Shodan client and passes to it the API key. It will then query the Shodan API to get the account information to check if the API key is valid. If it is not, an error will be logged and the plugin will stop.
+
+Once the client is ready, a timer will be set to query the Shodan API a the interval set up in the configuration. One line of "log" will be generated per element contained in the `matches` array from the query result. An other query will be submitted to gather data fro mthe next page if
+
+- the amount of read entries is lesser than the total available entries
+- the current read page is not greater than the `max_pages` parameter
+
+### Plugin helpers
+
+* [timer](https://docs.fluentd.org/v/1.0/plugin-helper-overview/api-plugin-helper-timer)
+* See also: [Input Plugin Overview](https://docs.fluentd.org/v/1.0/input#overview)
+
+### Configuration
+
+#### api_key (string) (required)
+
+The API key to connect to the Shodan API.
+
+#### interval (time) (optional)
+
+The interval time between running queries.
+
+Default value: `3600`.
+
+#### tag (string) (optional)
+
+The tag to apply to each shodan entries.
+
+#### query (string) (required)
+
+The Shodan query to execute.
+
+#### max_pages (integer) (optional)
+
+The maximum amount of pages to crawl. A 0 or negative value means to crawl all pages. Note that if you have a free account, querying a page other than the first one will result in a `HTTP 401` response.
+
+Default value: `1`.
+
+## Shodan Stream
+
+WIP
+
+## Shodan Alert
+
+WIP
+
+## Testing
+
+### Unit tests
+
+1. Clone this repository
+2. Install all dependencies with `bundle install`
+3. Set the `SHODAN_TEST_API_KEY` environment variable with your API key
+4. Run `rake` or `rake test`
+
+### Live tests
+
+On a system where fluentd is installed
+
+1. Clone this repository
+2. Build the gem with `gem build fluent-plugin-shodan.gemspec`
+3. Install the built gem with `fluent-gem install fluent-plugin-shodan-<version>.gem`
+4. Follow the [debugging guide from FluentD](https://docs.fluentd.org/plugin-development#debugging-plugins)
+
+## Copyright
+
+* Copyright(c) 2022 Marc-André Doll
+* License
+  * Apache License, Version 2.0

data/VERSION

@@ -0,0 +1 @@
+0.0.1

data/lib/fluent/plugin/in_shodan_search.rb

@@ -0,0 +1,65 @@
+require 'fluent/plugin/input'
+require 'shodanz'
+
+module Fluent::Plugin
+  class ShodanSearch < Input
+    Fluent::Plugin.register_input('shodan_search', self)
+
+    helpers :timer
+
+    desc "The API key to connect to the Shodan API."
+    config_param :api_key, :string, secret: true
+    desc "The interval time between running queries."
+    config_param :interval, :time, default: 3600
+    desc "The tag to apply to each shodan entries."
+    config_param :tag, :string, default: nil
+    desc "The Shodan query to execute."
+    config_param :query, :string
+    desc "The maximum amount of pages to crawl. A 0 or negative value means to crawl all pages."
+    config_param :max_pages, :integer, default: 1
+
+    def configure(conf)
+      super
+
+      @client = Shodanz.client.new(key: @api_key)
+      begin
+        log.info "Shodan client properly registered", client_info: @client.info
+      rescue RuntimeError => exception
+        raise Fluent::ConfigError.new "Invalid Shodan API key"
+      end
+    end
+
+    def multi_workers_ready?
+      false
+    end
+
+    def start
+      super
+
+      timer_execute("shodan_#{self.class.name}_#{@tag}".to_sym, @interval, repeat: true, &method(:run))
+    end
+
+    private
+
+    def run
+      log.debug "Starting Shodan search", query: @query, max_pages: @max_pages
+      es_time = Fluent::EventTime.now
+      current_page = 0
+      read_entries = 0
+      loop do
+        current_page += 1
+        result = @client.host_search(@query, page: current_page)
+        result['matches'].each do |rec|
+          router.emit(@tag, es_time, rec)
+        end
+        read_entries += result['matches'].length
+        break if (@max_pages >= 0 && current_page >= @max_pages) || read_entries >= result['total']
+      end
+      log.debug "Shodan search ending", query: @query, total_read: read_entries
+    rescue RuntimeError => re
+      log.error "Unable to execute Shodan query", query: @query, page: current_page, error: re
+    rescue => exception
+      log.error "Error executing Shodan query", error: exception
+    end
+  end
+end

data/test/test_in_shodan_search.rb

@@ -0,0 +1,59 @@
+require 'test/unit'
+require 'fluent/env'
+require 'fluent/test'
+require 'fluent/test/driver/input'
+require 'fluent/test/helpers'
+require "fluent/plugin/in_shodan_search"
+
+class ShodanSearchInputTest < Test::Unit::TestCase
+  include Fluent::Test::Helpers
+
+  API_KEY = ENV['SHODAN_TEST_API_KEY']
+
+  def setup
+    Fluent::Test.setup
+  end
+
+  sub_test_case 'Configuration' do
+    test 'is invalid without API key' do
+      assert_raise Fluent::ConfigError do
+        create_driver(%[
+          query 8.8.8.8
+        ])
+      end
+    end
+    test 'is invalid if the API key is invalid' do
+      assert_raise Fluent::ConfigError do
+        create_driver(%[
+          api_key 1234567890AZERTYUIOP
+          query 8.8.8.8
+        ])
+      end
+    end
+    test 'is invalid without a query' do
+      assert_raise Fluent::ConfigError do
+        create_driver(%[
+          api_key #{API_KEY}
+        ])
+      end
+    end
+    test 'query is set correctly' do
+      d = create_driver
+      assert_equal '8.8.8.8', d.instance.query
+    end
+  end
+
+  sub_test_case 'Plugin emission' do
+  end
+
+  private
+
+  CONFIG = %[
+    api_key #{API_KEY}
+    query 8.8.8.8
+  ]
+
+  def create_driver(conf = CONFIG)
+    Fluent::Test::Driver::Input.new(Fluent::Plugin::ShodanSearch).configure(conf)
+  end
+end

metadata

@@ -0,0 +1,139 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-shodan
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- srilumpa
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2022-01-28 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.12.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.12.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
+- !ruby/object:Gem::Dependency
+  name: shodanz
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: io-console
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13'
+- !ruby/object:Gem::Dependency
+  name: test-unit
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.5'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.5'
+description: 
+email: marcandre.doll@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files:
+- README.md
+files:
+- AUTHORS
+- README.md
+- VERSION
+- lib/fluent/plugin/in_shodan_search.rb
+- test/test_in_shodan_search.rb
+homepage: https://github.com/srilumpa/fluent-plugin-shodan
+licenses:
+- Apache-2.0
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.3.1
+signing_key: 
+specification_version: 3
+summary: Fluentd plugin to extract data from Shodan
+test_files:
+- test/test_in_shodan_search.rb