logstash-filter-ipinfo-db-zuin 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 2b36618d5861aa97840f2735fe419fd4303f858f553d883ee8f798c480153666
+  data.tar.gz: a298c76f7316d14c68d9efca5923797c7f6f231758ace241fa7a4c723344181d
+SHA512:
+  metadata.gz: 7e066fcca849ed60a80c95b745756185d92f5494492df4bd3a7792d011788c2243c3d91a91ee9dd5cb85ab46917b83f38d1b42125fed45f5f7e92d9c9c953040
+  data.tar.gz: 7e38e997a2a75444dfefc2630f1632ff8a1929c5d2e66942135601d6c991e26979eb16ff2f3f3a61d61d4a681fb753026978482da72e842af9085ebb7546a272

data/CHANGELOG.md

@@ -0,0 +1,30 @@
+## 0.2.0
+
+  - Forked as `logstash-filter-ipinfo-db-zuin` (filter name stays `ipinfo-db`)
+  - Add `token` setting; the ipinfo.io token is no longer hardcoded
+  - Add `database` setting. The database is now stored at
+    `<path.data>/plugins/filters/ipinfo-db/country_asn.mmdb` by default instead of the
+    Logstash working directory, which is `/` (not writable) under the deb/rpm systemd service
+  - Validate the downloaded database before atomically moving it into place, so an interrupted
+    or invalid download no longer leaves a broken file that blocks every later start
+  - Add `database_age_warning_days` (default 30): log a warning at startup and once a day
+    when the database is older than that
+  - Handle IPs not in the database and missing source fields without raising exceptions
+    (faster on JRuby); log unexpected lookup errors as warnings instead of swallowing them
+  - IPs that are valid but not in the database are no longer tagged with `tag_on_failure`;
+    the event is left unchanged
+  - Filters that use the same database file share one in-memory copy instead of loading one
+    each (several `ipinfo-db` blocks, pipelines, config reloads)
+  - Pin `down` to `~> 5.2` (compatible with `logstash-core`, which requires `down ~> 5.2.0`)
+    and `maxmind-db` to `~> 1.2`
+  - Docs: README now documents installation and settings; `source` docs no longer claim hostname
+    support; package `docs/` and no longer package `vendor/`
+
+## 0.1.0
+
+  - Plugin created with the logstash plugin generator
+  - Initial devlopment
+  - Test case for plugin
+  - Documentation
+  - Review Preparation
+

data/CONTRIBUTORS

@@ -0,0 +1,11 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Muhammad Awais Murad - awais@ipinfo.io
+* Vincent Kuo - vit@zuin.ai
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,10 @@
+source 'https://rubygems.org'
+gemspec
+
+logstash_path = ENV['LOGSTASH_PATH'] || '/usr/share/logstash'
+
+if Dir.exist?(logstash_path)
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end
+

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,120 @@
+# logstash-filter-ipinfo-db-zuin
+
+A [Logstash](https://github.com/elastic/logstash) filter plugin that looks up the country and ASN of an
+IP address in the [ipinfo.io](https://ipinfo.io) free country and ASN database (`.mmdb`), locally,
+without calling the ipinfo API per event.
+
+This is a fork of [`logstash-filter-ipinfo-db`](https://rubygems.org/gems/logstash-filter-ipinfo-db).
+The gem is renamed, but the filter is still called `ipinfo-db`, so existing pipeline configs keep working.
+Do not install both gems in the same Logstash.
+
+The license is Apache 2.0.
+
+## Installation
+
+```sh
+bin/logstash-plugin install logstash-filter-ipinfo-db-zuin
+```
+
+## Usage
+
+```
+filter {
+  ipinfo-db {
+    source => "ip"
+    token => "${IPINFO_TOKEN}"
+  }
+}
+```
+
+On first start, the plugin downloads the database with your ipinfo.io token to
+`<path.data>/plugins/filters/ipinfo-db/country_asn.mmdb` (for example `/var/lib/logstash/...` for
+deb/rpm installs). Store the token in an environment variable or the Logstash keystore:
+
+```sh
+bin/logstash-keystore add IPINFO_TOKEN
+```
+
+The lookup result is written to the `ipinfo-db` field. For `8.8.8.8` it looks like this (illustrative):
+
+```json
+"ipinfo-db": {
+  "country": "US",
+  "country_name": "United States",
+  "continent": "NA",
+  "continent_name": "North America",
+  "asn": "AS15169",
+  "as_name": "Google LLC",
+  "as_domain": "google.com"
+}
+```
+
+| Setting | Default | Description |
+|---|---|---|
+| `source` | (required) | Field containing the IP address (IPv4 or IPv6). Hostnames are not resolved. |
+| `target` | `ipinfo-db` | Field to write the result into. |
+| `token` | none | ipinfo.io token, used only to download the database when the file does not exist. |
+| `database` | `<path.data>/plugins/filters/ipinfo-db/country_asn.mmdb` | Path to the `.mmdb` file. Downloaded there if missing. |
+| `database_age_warning_days` | `30` | Log a warning when the database is older than this. `0` disables it. |
+| `tag_on_failure` | `["_ipinfo_db_lookup_failure"]` | Tags added when the field is missing, the value is not a valid IP, or the lookup fails. IPs that are simply not in the database are not tagged. |
+
+The plugin does not update the database automatically. To refresh it, delete the file and restart
+Logstash, or replace the file with a newer copy. See [docs/index.asciidoc](docs/index.asciidoc) for the
+full reference.
+
+## Development
+
+You need JRuby with Bundler, and a local Logstash for `logstash-core`. Set `LOGSTASH_PATH` if it is not
+in `/usr/share/logstash` (see `Gemfile`).
+
+```sh
+bundle install
+bundle exec rspec
+```
+
+The default suite runs offline against `spec/fixtures/ipinfo-test.mmdb`, a tiny synthetic database
+(documentation IP ranges only) generated by `spec/fixtures/generate_ipinfo_test_mmdb.py`. The live test,
+which downloads the real database from ipinfo.io, is tagged `integration` and excluded by default:
+
+```sh
+IPINFO_TOKEN=... bundle exec rspec --tag integration
+```
+
+### Verify on a real Logstash
+
+`verify/verify.sh` builds the gem, installs it into the official Logstash Docker image, runs real
+pipelines (download as a non-root user with working directory `/`, lookups, restart without token,
+missing-token error) and the RSpec suite on Logstash's JRuby. It needs Docker, Ruby and Python 3:
+
+```sh
+IPINFO_TOKEN=... verify/verify.sh 9.5.5
+```
+
+Without `IPINFO_TOKEN`, the checks that download the live database (first start, restart and the
+integration spec) are reported as `SKIP`; the install check, the missing-token error and the offline
+specs still run.
+
+Logs are written to `verify/out/`. GitHub Actions (`.github/workflows/verify.yml`) runs the same
+script against Logstash 8.19.24 and 9.5.5 on pushes and pull requests to `main`, reading the token
+from the `IPINFO_TOKEN` repository secret. Pull requests from forks don't get the secret, so they run
+the offline checks only.
+
+### Run the plugin in Logstash
+
+Build and install the gem:
+
+```sh
+gem build logstash-filter-ipinfo-db-zuin.gemspec
+bin/logstash-plugin install /path/to/logstash-filter-ipinfo-db-zuin-<version>.gem
+```
+
+Or add it to the Logstash `Gemfile` from your local checkout, then run `bin/logstash-plugin install --no-verify`:
+
+```ruby
+gem "logstash-filter-ipinfo-db-zuin", :path => "/path/to/logstash-filter-ipinfo-db"
+```
+
+## Help
+
+Open an issue at https://github.com/reflective-technology/logstash-filter-ipinfo-db/issues, or ask in the
+[Logstash discussion forum](https://discuss.elastic.co/c/logstash).

data/docs/index.asciidoc

@@ -0,0 +1,127 @@
+:plugin: logstash-filter-ipinfo-db-zuin
+:type: filter
+
+///////////////////////////////////////////
+START - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+:version: %VERSION%
+:release_date: %RELEASE_DATE%
+:changelog_url: %CHANGELOG_URL%
+:include_path: ../../../../logstash/docs/include
+///////////////////////////////////////////
+END - GENERATED VARIABLES, DO NOT EDIT!
+///////////////////////////////////////////
+
+[id="plugins-{type}s-{plugin}"]
+
+=== logstash-filter-ipinfo-db-zuin filter plugin
+
+include::{include_path}/plugin_header.asciidoc[]
+
+==== Description
+
+The ipinfo filter adds information about the geographical location  and autonomous system number (ASN) of IP addresses,
+based on data from the ipinfo.io free country and ASN database.
+
+For example, if you have a field named `foo` which you want to lookup for location and ASN in ipinfo.io free database, and you want to store the
+result of lookup in `bar`, do this:
+[source,ruby]
+    filter {
+      ipinfo-db {
+        source => "foo"
+        target => "bar"
+        token => "${IPINFO_TOKEN}"
+      }
+    }
+
+[id="plugins-{type}s-{plugin}-options"]
+==== ipinfo-db Filter Configuration Options
+
+This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+[cols="<,<,<",options="header",]
+|=======================================================================
+|Setting |Input type|Required
+| <<plugins-{type}s-{plugin}-database>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-database_age_warning_days>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-source>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-tag_on_failure>> |<<array,array>>|No
+| <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-token>> |<<password,password>>|No
+|=======================================================================
+
+Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
+filter plugins.
+
+&nbsp;
+
+[id="plugins-{type}s-{plugin}-database"]
+===== `database`
+
+  * Value type is <<string,string>>
+  * Default value is `<path.data>/plugins/filters/ipinfo-db/country_asn.mmdb`
+
+Path to the ipinfo `.mmdb` database file. If the file exists, the plugin uses it as is. If it does not exist,
+the plugin downloads it to this path on startup using <<plugins-{type}s-{plugin}-token>>, creating parent
+directories as needed. `path.data` is the Logstash data directory (for example `/var/lib/logstash` for
+deb/rpm installs and `/usr/share/logstash/data` in Docker). Use an absolute path; a relative path is resolved
+against the Logstash working directory.
+
+The database is loaded into memory once per file: all `ipinfo-db` filters in the Logstash process that use
+the same path (including across pipelines and config reloads) share one copy. If the file is replaced, the
+next filter that starts, for example after a config reload, loads the new file. When no running filter uses a
+database any more (its pipelines stopped, or were reloaded with a different `database` path), it is released
+from memory.
+
+[id="plugins-{type}s-{plugin}-database_age_warning_days"]
+===== `database_age_warning_days`
+
+  * Value type is <<number,number>>
+  * Default value is `30`
+
+The plugin does not update the database by itself. It logs a warning when the database was built more than this
+many days ago, using the build time stored in the `.mmdb` file. The check runs at startup and then once a day.
+To refresh the database, delete the file so it is downloaded again on the next start, or replace it with a newer
+copy. Set to `0` to disable the warning.
+
+[id="plugins-{type}s-{plugin}-source"]
+===== `source` 
+
+  * This is a required setting.
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The field containing the IP address (IPv4 or IPv6) to look up in the ipinfo database. Hostnames are not resolved;
+a value that is not a valid IP address tags the event with <<plugins-{type}s-{plugin}-tag_on_failure>>.
+
+[id="plugins-{type}s-{plugin}-tag_on_failure"]
+===== `tag_on_failure`
+
+  * Value type is <<array,array>>
+  * Default value is `["_ipinfo_db_lookup_failure"]`
+
+Tags the event when the lookup cannot be done: the source field is missing or empty, the value is not a valid
+IP address, or the lookup raises an error. This can be used in later analysis. An IP address that is valid but
+not in the database (for example a private address) is not a failure: the event is left unchanged and not tagged.
+
+[id="plugins-{type}s-{plugin}-target"]
+===== `target` 
+
+  * Value type is <<string,string>>
+  * Default value is `ipinfo-db`
+
+The field to write the lookup results into. 
+
+
+[id="plugins-{type}s-{plugin}-token"]
+===== `token`
+
+  * Value type is <<password,password>>
+  * There is no default value for this setting.
+
+Your ipinfo.io access token, used to download the database when the file at
+<<plugins-{type}s-{plugin}-database>> does not exist. If the file is missing and no token is set, the plugin fails to start.
+To keep the token out of the pipeline config, use `${IPINFO_TOKEN}` with an environment variable or the Logstash keystore.
+
+[id="plugins-{type}s-{plugin}-common-options"]
+include::{include_path}/{type}.asciidoc[]

data/lib/logstash/filters/ipinfo-db.rb

@@ -0,0 +1,268 @@
+# encoding: utf-8
+require "logstash/filters/base"
+require 'down'
+require 'maxmind/db'
+require 'fileutils'
+require 'securerandom'
+
+# Looks up the country and ASN of an IP address in the ipinfo.io free country
+# and ASN database (.mmdb) and stores the result in the event.
+#
+# filter {
+#   ipinfo-db {
+#     source => "ip"
+#     token => "${IPINFO_TOKEN}"
+#   }
+# }
+class LogStash::Filters::IpinfoDb < LogStash::Filters::Base
+
+  DOWNLOAD_URL = "https://ipinfo.io/data/free/country_asn.mmdb"
+  DEFAULT_FILE_NAME = "country_asn.mmdb"
+  AGE_CHECK_INTERVAL = 24 * 60 * 60 # seconds
+
+  config_name "ipinfo-db"
+
+  # The field containing the IP address to look up. Hostnames are not resolved.
+  config :source, :validate => :string, :required => true
+
+  # The field to write the lookup results into. If not specified, the default
+  # value will be used.
+  config :target, :validate => :string, :default => "ipinfo-db"
+
+  # Tags the event on failure to look up ipinfo information. This can be used in later analysis.
+  # If not specified, the default value will be used.
+  config :tag_on_failure, :validate => :array, :default => ["_ipinfo_db_lookup_failure"]
+
+  # ipinfo.io access token used to download the database. Only needed when
+  # the file at `database` does not exist yet. Use ${IPINFO_TOKEN} to read it
+  # from an environment variable or the Logstash keystore.
+  config :token, :validate => :password
+
+  # Path to the ipinfo .mmdb database file. If the file does not exist, it is
+  # downloaded to this path using `token`. Defaults to
+  # <path.data>/plugins/filters/ipinfo-db/country_asn.mmdb.
+  config :database, :validate => :string
+
+  # Log a warning when the database was built more than this many days ago.
+  # Checked at startup and then once a day. Set to 0 to disable.
+  config :database_age_warning_days, :validate => :number, :default => 30
+
+  # One in-memory reader per database file, shared by every filter instance in
+  # the process (several ipinfo-db blocks, pipelines, config reloads), so the
+  # database is held in the heap once. Readers are safe to use from multiple
+  # threads.
+  #
+  # - Entries record the identity (inode, size, mtime) of the file they were
+  #   loaded from, verified before and after loading, so a replaced file is
+  #   loaded again by the next instance that registers.
+  # - Entries are reference counted: register acquires, close releases, and the
+  #   last release evicts the entry, so readers for files no filter uses any
+  #   more (stopped pipelines, reloads to another database path) can be
+  #   garbage collected.
+  # - Setup for a given file (existence check, download, load, cache insert)
+  #   runs under a per-path lock, so pipelines starting together download and
+  #   load a missing database only once, while unrelated database files still
+  #   set up in parallel. @registry_mutex only guards the hashes.
+  @shared_readers = {}
+  @path_locks = {}
+  @registry_mutex = Mutex.new
+
+  LOAD_ATTEMPTS = 3
+
+  class << self
+    def with_path_lock(path)
+      lock = @registry_mutex.synchronize { @path_locks[path] ||= Mutex.new }
+      lock.synchronize { yield }
+    end
+
+    # Acquire the cached reader for path if it was loaded from the file
+    # currently there; nil otherwise.
+    def acquire_reader(path)
+      identity = file_identity(path)
+      @registry_mutex.synchronize do
+        entry = @shared_readers[path]
+        next nil unless entry && entry[:identity] == identity
+        entry[:refs] += 1
+        entry[:reader]
+      end
+    end
+
+    # Load path and cache the reader under the identity the file had for the
+    # whole load. If the file is replaced while loading, load it again.
+    def load_reader(path)
+      LOAD_ATTEMPTS.times do
+        before = file_identity(path)
+        reader = MaxMind::DB.new(path, mode: MaxMind::DB::MODE_MEMORY)
+        return store_reader(path, before, reader) if file_identity(path) == before
+      end
+      raise LogStash::ConfigurationError, "ipinfo-db: database file #{path} kept changing while it was being loaded"
+    end
+
+    # Cache reader (loaded from the file with identity) with one reference.
+    def store_reader(path, identity, reader)
+      @registry_mutex.synchronize { @shared_readers[path] = { :identity => identity, :reader => reader, :refs => 1 } }
+      reader
+    end
+
+    # Drop one reference; evict the entry when no filter uses it any more. A
+    # reader that was already superseded by a newer file is not in the cache.
+    def release_reader(path, reader)
+      @registry_mutex.synchronize do
+        entry = @shared_readers[path]
+        next unless entry && entry[:reader].equal?(reader)
+        entry[:refs] -= 1
+        @shared_readers.delete(path) if entry[:refs] <= 0
+      end
+    end
+
+    # The cached reader for path, if any (no reference taken). For specs.
+    def cached_reader(path)
+      @registry_mutex.synchronize { @shared_readers[File.expand_path(path)] }&.fetch(:reader)
+    end
+
+    # For specs.
+    def clear_shared_readers
+      @registry_mutex.synchronize { @shared_readers.clear }
+    end
+
+    def file_identity(path)
+      stat = File.stat(path)
+      [stat.ino, stat.size, stat.mtime.to_r]
+    end
+  end
+
+  def register
+    setup_database
+    check_database_age
+  end
+
+  def close
+    self.class.release_reader(@database, @db) if @db
+    @db = nil
+  end
+
+  def filter(event)
+    next_age_check = @next_age_check
+    check_database_age if next_age_check && Time.now >= next_age_check
+    @logger.debug? && @logger.debug("Running ipinfo-db lookup", :event => event)
+    ip = event.get(@source)
+
+    # Misses are common (private IPs, missing fields), so handle them without
+    # raising: exceptions are expensive in JRuby.
+    if ip.nil? || ip == ""
+      tag_unsuccessful_lookup(event, "source field is empty")
+      return
+    end
+
+    begin
+      record = @db.get(ip)
+    rescue ArgumentError => e
+      # Not a valid IP address (IPAddr errors are ArgumentErrors), or an IPv6
+      # address looked up in an IPv4-only database.
+      tag_unsuccessful_lookup(event, e.message)
+      return
+    rescue => e
+      @logger.warn("Unexpected error during ipinfo lookup", :ip => ip, :exception => e.class.name, :message => e.message)
+      tag_unsuccessful_lookup(event, e.message)
+      return
+    end
+
+    # Not in the database (e.g. private ranges): not an error, leave the event as is.
+    if record.nil?
+      @logger.debug? && @logger.debug("IP not found in the ipinfo database", :ip => ip)
+      return
+    end
+
+    begin
+      event.set(@target, record)
+      filter_matched(event)
+    rescue => e
+      @logger.warn("Unexpected error during ipinfo enrichment", :ip => ip, :exception => e.class.name, :message => e.message)
+      tag_unsuccessful_lookup(event, e.message)
+    end
+  end
+
+  def tag_unsuccessful_lookup(event, reason)
+    @logger.debug? && @logger.debug("ipinfo lookup failed", :ip => event.get(@source), :reason => reason)
+    @tag_on_failure.each{|tag| event.tag(tag)}
+  end
+
+  # Download .mmdb file and load it into mmdb reader
+  def setup_database
+    # Normalize once (expands ~ and makes relative paths absolute) so the existence
+    # check, download, rename and reader cache all use the same path.
+    @database = File.expand_path(@database || default_database_path)
+    path = @database
+
+    @db = self.class.with_path_lock(path) do
+      if File.exist?(path)
+        self.class.acquire_reader(path) || self.class.load_reader(path)
+      else
+        if @token.nil? || @token.value.to_s.empty?
+          raise LogStash::ConfigurationError, "ipinfo-db: database file #{@database} not found. Set `token` to download it, or place the file there."
+        end
+        reader, identity = download_database
+        # A rename keeps inode, size and mtime, so a different identity here means
+        # another process replaced the file after our rename: load what is there.
+        if File.exist?(path) && self.class.file_identity(path) == identity
+          self.class.store_reader(path, identity, reader)
+        else
+          self.class.load_reader(path)
+        end
+      end
+    end
+  end
+
+  private
+
+  # Warn when the loaded database is older than database_age_warning_days,
+  # based on the build time stored in the database itself.
+  def check_database_age
+    @next_age_check = nil
+    return if @database_age_warning_days <= 0
+
+    build_epoch = @db.metadata.build_epoch
+    return if build_epoch.nil?
+
+    @next_age_check = Time.now + AGE_CHECK_INTERVAL
+    age_days = ((Time.now.to_i - build_epoch) / 86400.0).floor
+    if age_days > @database_age_warning_days
+      @logger.warn("ipinfo database is outdated; lookups may be inaccurate. " \
+                   "Delete the file to download a fresh copy on the next start (requires `token`), or replace it",
+                   :path => @database, :age_days => age_days, :built_at => Time.at(build_epoch).utc.to_s,
+                   :database_age_warning_days => @database_age_warning_days)
+    end
+  end
+
+  # Download into a temp file next to @database, check that it is a readable
+  # database, then rename it into place. The rename is atomic, so @database is
+  # never a partial or invalid file, even if the download is interrupted or
+  # several pipelines download at the same time. Returns the reader and the
+  # identity of the file it was loaded from.
+  def download_database
+    @logger.info("Downloading ipinfo database", :path => @database)
+    dir = File.dirname(@database)
+    FileUtils.mkdir_p(dir)
+    tmp_path = File.join(dir, ".#{File.basename(@database)}.#{SecureRandom.hex(8)}.tmp")
+
+    tempfile = Down.download("#{DOWNLOAD_URL}?token=#{@token.value}")
+    FileUtils.mv(tempfile.path, tmp_path)
+    begin
+      db = MaxMind::DB.new(tmp_path, mode: MaxMind::DB::MODE_MEMORY)
+    rescue MaxMind::DB::InvalidDatabaseError => e
+      raise LogStash::ConfigurationError, "ipinfo-db: downloaded database is not a valid .mmdb file: #{e.message}"
+    end
+    identity = self.class.file_identity(tmp_path) # only this process writes tmp_path
+    File.rename(tmp_path, @database)
+    [db, identity]
+  ensure
+    FileUtils.rm_f(tmp_path) if tmp_path
+  end
+
+  # Keep the downloaded database under Logstash's data directory, which is
+  # writable by the logstash user and persists across restarts.
+  def default_database_path
+    File.join(LogStash::SETTINGS.get_value("path.data"), "plugins", "filters", "ipinfo-db", DEFAULT_FILE_NAME)
+  end
+
+end # class LogStash::Filters::IpinfoDb

data/logstash-filter-ipinfo-db-zuin.gemspec

@@ -0,0 +1,28 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-filter-ipinfo-db-zuin'
+  s.version       = '0.2.0'
+  s.licenses      = ['Apache-2.0']
+  s.summary       = 'Logstash Filter Plugin for IPinfoDb'
+  s.description   = 'This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline ' \
+                    'using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program. ' \
+                    'It looks up the country and ASN of IP addresses in the ipinfo.io free country and ASN database.'
+  s.homepage      = 'https://github.com/reflective-technology/logstash-filter-ipinfo-db'
+  s.authors       = ['ZUIN RD']
+  s.email         = 'rd@zuin.ai'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','docs/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
+  # logstash-core itself depends on down ~> 5.2.0, so this must allow 5.2.x.
+  s.add_runtime_dependency "down", "~> 5.2"
+  s.add_runtime_dependency "maxmind-db", "~> 1.2"
+  s.add_development_dependency 'logstash-devutils', '~> 2.6'
+end

data/spec/filters/ipinfo-db_spec.rb

@@ -0,0 +1,379 @@
+# encoding: utf-8
+require_relative '../spec_helper'
+require "logstash/filters/ipinfo-db"
+require "fileutils"
+require "tmpdir"
+require "tempfile"
+require "ipaddr"
+
+# Synthetic database with documentation networks only; see spec/fixtures/generate_ipinfo_test_mmdb.py.
+FIXTURE_DATABASE = File.expand_path("../fixtures/ipinfo-test.mmdb", __dir__)
+
+describe LogStash::Filters::IpinfoDb do
+  after { described_class.clear_shared_readers }
+
+  describe "shared database reader" do
+    let(:data_dir) { Dir.mktmpdir }
+    let(:database) { File.join(data_dir, "ipinfo.mmdb").tap { |path| FileUtils.cp(FIXTURE_DATABASE, path) } }
+    after { FileUtils.rm_rf(data_dir) }
+
+    def registered(path)
+      described_class.new("source" => "ip", "database" => path, "database_age_warning_days" => 0).tap(&:register)
+    end
+
+    def reader(plugin)
+      plugin.instance_variable_get(:@db)
+    end
+
+    it "loads a database file once for all filter instances" do
+      expect(MaxMind::DB).to receive(:new).once.and_call_original
+      first = registered(database)
+      second = registered(database)
+      expect(reader(second)).to equal(reader(first))
+
+      event = LogStash::Event.new("ip" => "192.0.2.10")
+      second.filter(event)
+      expect(event.get("[ipinfo-db][country]")).to eq("US")
+    end
+
+    it "loads the file again after it is replaced" do
+      first = registered(database)
+      replacement = File.join(data_dir, "replacement.mmdb")
+      FileUtils.cp(FIXTURE_DATABASE, replacement)
+      File.rename(replacement, database)
+      second = registered(database)
+      expect(reader(second)).not_to equal(reader(first))
+    end
+
+    it "downloads a missing database once when filters start at the same time" do
+      missing = File.join(data_dir, "downloaded.mmdb")
+      allow(Down).to receive(:download) do
+        sleep 0.5 # keep the first download in progress while the second filter registers
+        Tempfile.new("ipinfo-download").tap do |file|
+          file.binmode
+          file.write(File.binread(FIXTURE_DATABASE))
+          file.flush
+        end
+      end
+      plugins = Array.new(2) do
+        described_class.new("source" => "ip", "database" => missing, "token" => "t", "database_age_warning_days" => 0)
+      end
+
+      plugins.map { |plugin| Thread.new { plugin.register } }.each(&:join)
+
+      expect(Down).to have_received(:download).once
+      expect(reader(plugins[1])).to equal(reader(plugins[0]))
+      expect(Dir.children(data_dir)).to eq(["downloaded.mmdb"])
+    end
+
+    it "evicts the reader when the last filter using it closes" do
+      first = registered(database)
+      second = registered(database)
+      first.close
+      expect(described_class.cached_reader(database)).to equal(reader(second))
+      second.close
+      expect(described_class.cached_reader(database)).to be_nil
+    end
+
+    it "evicts the old reader when a reload switches to another database path" do
+      old = registered(database)
+      versioned = File.join(data_dir, "ipinfo-v2.mmdb")
+      FileUtils.cp(FIXTURE_DATABASE, versioned)
+      current = registered(versioned)
+      old.close
+      expect(described_class.cached_reader(database)).to be_nil
+      expect(described_class.cached_reader(versioned)).to equal(reader(current))
+    end
+
+    it "loads the file again if it is replaced while it is being loaded" do
+      path = database
+      loads = 0
+      allow(MaxMind::DB).to receive(:new).and_wrap_original do |original, *args, **kwargs|
+        loads += 1
+        loaded = original.call(*args, **kwargs)
+        if loads == 1 # an external updater replaces the file mid-load
+          replacement = File.join(data_dir, "replacement.mmdb")
+          FileUtils.cp(FIXTURE_DATABASE, replacement)
+          File.rename(replacement, path)
+        end
+        loaded
+      end
+
+      registered(path)
+      expect(loads).to eq(2)
+      registered(path) # the cached identity matches the file now there, so no further load
+      expect(loads).to eq(2)
+    end
+
+    it "loads the file again if it is replaced right after a download is moved into place" do
+      target = File.join(data_dir, "downloaded.mmdb")
+      allow(Down).to receive(:download) do
+        Tempfile.new("ipinfo-download").tap do |file|
+          file.binmode
+          file.write(File.binread(FIXTURE_DATABASE))
+          file.flush
+        end
+      end
+      allow(File).to receive(:rename).and_wrap_original do |original, from, to|
+        result = original.call(from, to)
+        if to == target # an external updater rewrites it immediately
+          File.binwrite(to, File.binread(FIXTURE_DATABASE))
+          File.utime(Time.now, Time.now + 60, to)
+        end
+        result
+      end
+      expect(MaxMind::DB).to receive(:new).twice.and_call_original
+
+      plugin = described_class.new("source" => "ip", "database" => target, "token" => "t", "database_age_warning_days" => 0)
+      plugin.register
+      expect(described_class.cached_reader(target)).to equal(reader(plugin))
+    end
+
+    it "expands ~ in the database path before downloading and loading" do
+      home = File.join(data_dir, "home")
+      cwd = File.join(data_dir, "cwd")
+      FileUtils.mkdir_p([home, cwd])
+      allow(Down).to receive(:download) do
+        Tempfile.new("ipinfo-download").tap do |file|
+          file.binmode
+          file.write(File.binread(FIXTURE_DATABASE))
+          file.flush
+        end
+      end
+      original_home = ENV["HOME"]
+      begin
+        ENV["HOME"] = home
+        plugin = described_class.new("source" => "ip", "database" => "~/ipinfo/country_asn.mmdb",
+                                     "token" => "t", "database_age_warning_days" => 0)
+        Dir.chdir(cwd) { plugin.register }
+      ensure
+        ENV["HOME"] = original_home
+      end
+
+      expanded = File.join(home, "ipinfo", "country_asn.mmdb")
+      expect(File.file?(expanded)).to be true
+      expect(Dir.children(cwd)).to be_empty # no literal "~" directory
+      expect(described_class.cached_reader(expanded)).to equal(reader(plugin))
+    end
+
+    it "keeps separate readers for different files" do
+      other = File.join(data_dir, "other.mmdb")
+      FileUtils.cp(FIXTURE_DATABASE, other)
+      expect(reader(registered(other))).not_to equal(reader(registered(database)))
+    end
+  end
+
+  describe "pipeline lookups against the fixture database" do
+    config <<-CONFIG
+      filter {
+        ipinfo-db {
+          source => "ip"
+          database => "#{FIXTURE_DATABASE}"
+          database_age_warning_days => 0
+        }
+      }
+    CONFIG
+
+    sample({ "ip" => "192.0.2.10" }) do
+      expect(subject.get("ipinfo-db")).to eq(
+        "country" => "US", "country_name" => "United States",
+        "continent" => "NA", "continent_name" => "North America",
+        "asn" => "AS64496", "as_name" => "Example Networks US", "as_domain" => "example.com"
+      )
+      expect(subject.get("tags")).to be_nil
+    end
+
+    sample({ "ip" => "198.51.100.7" }) do
+      expect(subject.get("[ipinfo-db][country]")).to eq("JP")
+      expect(subject.get("[ipinfo-db][asn]")).to eq("AS64497")
+    end
+
+    sample({ "ip" => "2001:db8::1" }) do
+      expect(subject.get("[ipinfo-db][country]")).to eq("TW")
+      expect(subject.get("[ipinfo-db][asn]")).to eq("AS64498")
+    end
+
+    # Valid but not in the database: left unchanged, not tagged.
+    sample({ "ip" => "10.0.0.1" }) do
+      expect(subject.get("ipinfo-db")).to be_nil
+      expect(subject.get("tags")).to be_nil
+    end
+
+    sample({ "ip" => "not-an-ip" }) do
+      expect(subject.get("ipinfo-db")).to be_nil
+      expect(subject.get("tags")).to eq(["_ipinfo_db_lookup_failure"])
+    end
+
+    sample({ "other" => "no ip field" }) do
+      expect(subject.get("ipinfo-db")).to be_nil
+      expect(subject.get("tags")).to eq(["_ipinfo_db_lookup_failure"])
+    end
+  end
+
+  describe "pipeline with custom target and tag_on_failure" do
+    config <<-CONFIG
+      filter {
+        ipinfo-db {
+          source => "[client][ip]"
+          target => "[client][ipinfo]"
+          tag_on_failure => ["no_ipinfo"]
+          database => "#{FIXTURE_DATABASE}"
+          database_age_warning_days => 0
+        }
+      }
+    CONFIG
+
+    sample({ "client" => { "ip" => "192.0.2.10" } }) do
+      expect(subject.get("[client][ipinfo][country]")).to eq("US")
+      expect(subject.get("ipinfo-db")).to be_nil
+    end
+
+    sample({ "client" => { "ip" => "bogus" } }) do
+      expect(subject.get("tags")).to eq(["no_ipinfo"])
+    end
+  end
+
+  describe "age warning with a real database" do
+    let(:build_epoch) { MaxMind::DB.new(FIXTURE_DATABASE, mode: MaxMind::DB::MODE_MEMORY).metadata.build_epoch }
+
+    it "uses the build time stored in the database" do
+      plugin = described_class.new("source" => "ip", "database" => FIXTURE_DATABASE)
+      allow(Time).to receive(:now).and_return(Time.at(build_epoch + 40 * 86400))
+      expect(plugin.logger).to receive(:warn).with(/outdated/, hash_including(:age_days => 40))
+      plugin.register
+    end
+  end
+
+  # Downloads the real database from ipinfo.io. Excluded by default; run with
+  #   IPINFO_TOKEN=... bundle exec rspec --tag integration
+  describe "live ipinfo.io database", :integration => true do
+    let(:data_dir) { Dir.mktmpdir }
+    after { FileUtils.rm_rf(data_dir) }
+
+    it "downloads the database and looks up a well-known IP" do
+      skip("set IPINFO_TOKEN to run") if ENV["IPINFO_TOKEN"].to_s.empty?
+      database = File.join(data_dir, "country_asn.mmdb")
+      plugin = described_class.new("source" => "ip", "database" => database, "token" => ENV["IPINFO_TOKEN"])
+      plugin.register
+      expect(File.size(database)).to be > 0
+      expect(Dir.children(data_dir)).to eq(["country_asn.mmdb"])
+
+      event = LogStash::Event.new("ip" => "8.8.8.8")
+      plugin.filter(event)
+      expect(event.get("[ipinfo-db][country]")).to eq("US")
+      expect(event.get("[ipinfo-db][asn]")).to eq("AS15169")
+    end
+  end
+
+  describe "database location" do
+    let(:data_dir) { Dir.mktmpdir }
+    after { FileUtils.rm_rf(data_dir) }
+
+    it "defaults to a file under path.data" do
+      allow(LogStash::SETTINGS).to receive(:get_value).and_call_original
+      allow(LogStash::SETTINGS).to receive(:get_value).with("path.data").and_return(data_dir)
+      expected = File.join(data_dir, "plugins", "filters", "ipinfo-db", "country_asn.mmdb")
+      plugin = described_class.new("source" => "ip")
+      expect { plugin.register }.to raise_error(LogStash::ConfigurationError, /#{Regexp.escape(expected)}/)
+    end
+
+    it "uses the configured database path" do
+      missing = File.join(data_dir, "missing.mmdb")
+      plugin = described_class.new("source" => "ip", "database" => missing)
+      expect { plugin.register }.to raise_error(LogStash::ConfigurationError, /#{Regexp.escape(missing)}/)
+    end
+
+    it "does not keep an invalid download" do
+      bad = Tempfile.new("bad-mmdb")
+      bad.write("not a database")
+      bad.flush
+      allow(Down).to receive(:download).and_return(bad)
+      database = File.join(data_dir, "ipinfo.mmdb")
+      plugin = described_class.new("source" => "ip", "database" => database, "token" => "t")
+      expect { plugin.register }.to raise_error(LogStash::ConfigurationError, /not a valid \.mmdb/)
+      expect(Dir.children(data_dir)).to be_empty
+    end
+  end
+
+  describe "database age warning" do
+    let(:data_dir) { Dir.mktmpdir }
+    let(:database) { File.join(data_dir, "ipinfo.mmdb").tap { |path| File.write(path, "x") } }
+    after { FileUtils.rm_rf(data_dir) }
+
+    def stub_db_built_days_ago(days)
+      metadata = double("metadata", :build_epoch => Time.now.to_i - days * 86400)
+      allow(MaxMind::DB).to receive(:new).and_return(double("db", :metadata => metadata))
+    end
+
+    it "warns when the database is older than database_age_warning_days" do
+      stub_db_built_days_ago(45)
+      plugin = described_class.new("source" => "ip", "database" => database)
+      expect(plugin.logger).to receive(:warn).with(/outdated/, hash_including(:age_days => 45))
+      plugin.register
+    end
+
+    it "does not warn for a recent database" do
+      stub_db_built_days_ago(3)
+      plugin = described_class.new("source" => "ip", "database" => database)
+      expect(plugin.logger).not_to receive(:warn).with(/outdated/, anything)
+      plugin.register
+    end
+
+    it "does not warn when disabled" do
+      stub_db_built_days_ago(400)
+      plugin = described_class.new("source" => "ip", "database" => database, "database_age_warning_days" => 0)
+      expect(plugin.logger).not_to receive(:warn).with(/outdated/, anything)
+      plugin.register
+    end
+  end
+
+  describe "lookup results" do
+    let(:data_dir) { Dir.mktmpdir }
+    let(:database) { File.join(data_dir, "ipinfo.mmdb").tap { |path| File.write(path, "x") } }
+    let(:db) { double("db", :metadata => double("metadata", :build_epoch => Time.now.to_i)) }
+    let(:plugin) { described_class.new("source" => "ip", "database" => database) }
+    after { FileUtils.rm_rf(data_dir) }
+
+    before do
+      allow(MaxMind::DB).to receive(:new).and_return(db)
+      plugin.register
+    end
+
+    def run(fields)
+      LogStash::Event.new(fields).tap { |event| plugin.filter(event) }
+    end
+
+    it "sets the target field when the IP is found" do
+      allow(db).to receive(:get).with("8.8.8.8").and_return("country" => "US")
+      event = run("ip" => "8.8.8.8")
+      expect(event.get("[ipinfo-db][country]")).to eq("US")
+      expect(event.get("tags")).to be_nil
+    end
+
+    it "leaves the event untouched when the IP is not in the database" do
+      allow(db).to receive(:get).with("10.0.0.1").and_return(nil)
+      event = run("ip" => "10.0.0.1")
+      expect(event.get("ipinfo-db")).to be_nil
+      expect(event.get("tags")).to be_nil
+    end
+
+    it "tags the event without a lookup when the source field is missing" do
+      expect(db).not_to receive(:get)
+      event = run("other" => "x")
+      expect(event.get("tags")).to eq(["_ipinfo_db_lookup_failure"])
+    end
+
+    it "tags the event when the value is not a valid IP" do
+      allow(db).to receive(:get).and_raise(IPAddr::InvalidAddressError, "invalid address: nope")
+      event = run("ip" => "nope")
+      expect(event.get("tags")).to eq(["_ipinfo_db_lookup_failure"])
+    end
+
+    it "logs unexpected errors and tags the event" do
+      allow(db).to receive(:get).and_raise(RuntimeError, "boom")
+      expect(plugin.logger).to receive(:warn).with(/Unexpected error/, hash_including(:message => "boom"))
+      event = run("ip" => "1.2.3.4")
+      expect(event.get("tags")).to eq(["_ipinfo_db_lookup_failure"])
+    end
+  end
+end

data/spec/fixtures/generate_ipinfo_test_mmdb.py

@@ -0,0 +1,56 @@
+"""Generate spec/fixtures/ipinfo-test.mmdb, a tiny database for offline specs.
+
+It uses the field layout of the ipinfo.io free country_asn database, but only documentation
+networks (RFC 5737, RFC 3849) and documentation ASNs (RFC 5398), so its data is synthetic.
+
+Regenerate from the repository root with:
+
+  docker run --rm -v "$PWD/spec/fixtures":/fixtures python:3.12-slim sh -c \
+    'pip install -q mmdb-writer==0.2.7 netaddr==1.3.0 && python /fixtures/generate_ipinfo_test_mmdb.py'
+
+Keep this data in sync with spec/filters/ipinfo-db_spec.rb.
+"""
+from netaddr import IPSet
+from mmdb_writer import MMDBWriter
+
+RECORDS = {
+    "192.0.2.0/24": {
+        "country": "US",
+        "country_name": "United States",
+        "continent": "NA",
+        "continent_name": "North America",
+        "asn": "AS64496",
+        "as_name": "Example Networks US",
+        "as_domain": "example.com",
+    },
+    "198.51.100.0/24": {
+        "country": "JP",
+        "country_name": "Japan",
+        "continent": "AS",
+        "continent_name": "Asia",
+        "asn": "AS64497",
+        "as_name": "Example Networks JP",
+        "as_domain": "example.jp",
+    },
+    "2001:db8::/32": {
+        "country": "TW",
+        "country_name": "Taiwan",
+        "continent": "AS",
+        "continent_name": "Asia",
+        "asn": "AS64498",
+        "as_name": "Example Networks TW",
+        "as_domain": "example.tw",
+    },
+}
+
+writer = MMDBWriter(
+    ip_version=6,
+    ipv4_compatible=True,
+    database_type="ipinfo country_asn.mmdb",
+    languages=["en"],
+    description="Synthetic test database for logstash-filter-ipinfo-db-zuin specs",
+)
+for network, record in RECORDS.items():
+    writer.insert_network(IPSet([network]), record)
+writer.to_db_file("/fixtures/ipinfo-test.mmdb")
+print("wrote /fixtures/ipinfo-test.mmdb")

data/spec/spec_helper.rb

@@ -0,0 +1,18 @@
+# Licensed to Elasticsearch B.V. under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Elasticsearch B.V. licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+require "logstash/devutils/rspec/spec_helper"

metadata

@@ -0,0 +1,119 @@
+--- !ruby/object:Gem::Specification
+name: logstash-filter-ipinfo-db-zuin
+version: !ruby/object:Gem::Version
+  version: 0.2.0
+platform: ruby
+authors:
+- ZUIN RD
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2026-11-05 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logstash-core-plugin-api
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: down
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.2'
+- !ruby/object:Gem::Dependency
+  name: maxmind-db
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
+- !ruby/object:Gem::Dependency
+  name: logstash-devutils
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.6'
+description: This gem is a Logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This
+  gem is not a stand-alone program. It looks up the country and ASN of IP addresses
+  in the ipinfo.io free country and ASN database.
+email: rd@zuin.ai
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- README.md
+- docs/index.asciidoc
+- lib/logstash/filters/ipinfo-db.rb
+- logstash-filter-ipinfo-db-zuin.gemspec
+- spec/filters/ipinfo-db_spec.rb
+- spec/fixtures/generate_ipinfo_test_mmdb.py
+- spec/fixtures/ipinfo-test.mmdb
+- spec/spec_helper.rb
+homepage: https://github.com/reflective-technology/logstash-filter-ipinfo-db
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: filter
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.22
+signing_key:
+specification_version: 4
+summary: Logstash Filter Plugin for IPinfoDb
+test_files:
+- spec/filters/ipinfo-db_spec.rb
+- spec/fixtures/generate_ipinfo_test_mmdb.py
+- spec/fixtures/ipinfo-test.mmdb
+- spec/spec_helper.rb