fluent-plugin-ip2location 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5b7b81c42ee55b5a20fcbf953dcfb6edd6076ae22af638e917bce7a779441ad8
+  data.tar.gz: 778f82f161d577d1a43b0b729c0f273730cbfc4a51fcb938666cd2695fd01743
+SHA512:
+  metadata.gz: cc6b58c35506a9862123f96bcb4f1f2c138d4d13e349b548ee0eb40114419789ba0c9c73ef9a4d0e9db0766ce53dd1ec648e11902aa366f1b53981b3a5a21d59
+  data.tar.gz: 4eb502dae00e7f66f6a872e2baf37f37b034bc867705150b91ac752eae78261a6e27ca611c19271e8a220f64ac5eef9e4d4144f210572258e57203f3c746a6bc

data/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Changelog
+
+## 0.1.0 - 2026-06-24
+
+- Initial release.
+- Adds Fluentd filter plugin `@type ip2location`.
+- Supports nested input and output fields through Fluentd record accessor syntax.
+- Supports IP2Location BIN lookup, field selection, private IP skipping, error handling and in-memory lookup cache.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 IP2Location.com
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,185 @@
+# IP2Location Fluentd Filter Plugin
+
+`fluent-plugin-ip2location` is a Fluentd filter plugin that enriches Fluentd events with IP geolocation data from an IP2Location BIN database.
+
+It reads an IP address from a configured record field, queries the IP2Location Ruby library against an IP2Location BIN database, and writes selected lookup fields back into the event record.
+
+## Features
+
+- Reads IP address from a top-level or nested record field, for example `ip`, `remote_addr`, or `$.client.ip`.
+- Supports nested output fields, for example `ip2location` or `$.enrichment.ip2location`.
+- Supports IP2Location geolocation fields including country, region, city, latitude, longitude, ISP, domain, ZIP code, time zone, usage type, ASN and AS name.
+- Skips invalid IP addresses and private/local IP addresses by default.
+- Supports configurable field selection.
+- Supports an in-memory lookup cache.
+
+## Installation
+
+Install it into the Ruby environment used by Fluentd:
+
+```bash
+fluent-gem install fluent-plugin-ip2location
+```
+
+Or add it to your Gemfile:
+
+```ruby
+gem "fluent-plugin-ip2location"
+```
+
+Then run:
+
+```bash
+bundle install
+```
+
+## IP2Location database
+
+This plugin requires an IP2Location BIN database file. For DB26, use a file such as:
+
+```text
+IPV6-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE-ZIPCODE-TIMEZONE-ISP-DOMAIN-NETSPEED-AREACODE-WEATHER-MOBILE-ELEVATION-USAGETYPE-ADDRESSTYPE-CATEGORY-DISTRICT-ASN.BIN
+```
+
+Place the BIN file somewhere readable by the Fluentd process, for example:
+
+```text
+/opt/ip2location/IPV6-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE-ZIPCODE-TIMEZONE-ISP-DOMAIN-NETSPEED-AREACODE-WEATHER-MOBILE-ELEVATION-USAGETYPE-ADDRESSTYPE-CATEGORY-DISTRICT-ASN.BIN
+```
+
+## Fluentd configuration
+
+```apache
+<filter app.**>
+  @type ip2location
+  database /opt/ip2location/IPV6-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE-ZIPCODE-TIMEZONE-ISP-DOMAIN-NETSPEED-AREACODE-WEATHER-MOBILE-ELEVATION-USAGETYPE-ADDRESSTYPE-CATEGORY-DISTRICT-ASN.BIN
+  ip_field $.client.ip
+  output_field $.ip2location
+  fields country_short,country_long,region,city,latitude,longitude,isp,domain,zipcode,timezone,usagetype,asn,as
+  skip_private_ip true
+  skip_invalid_ip true
+  cache_size 10000
+  on_error warn
+</filter>
+```
+
+### Input record
+
+```json
+{
+  "client": {
+    "ip": "8.8.8.8"
+  },
+  "message": "hello"
+}
+```
+
+### Output record
+
+```json
+{
+  "client": {
+    "ip": "8.8.8.8"
+  },
+  "message": "hello",
+  "ip2location": {
+    "country_short": "US",
+    "country_long": "United States of America",
+    "region": "California",
+    "city": "Mountain View",
+    "latitude": 37.40599,
+    "longitude": -122.078514,
+    "isp": "Google LLC",
+    "zipcode": "94043",
+    "timezone": "-08:00",
+    "asn": "15169",
+    "as": "Google LLC"
+  }
+}
+```
+
+## Configuration parameters
+
+| Parameter | Required | Default | Description |
+|---|---:|---|---|
+| `database` | Yes | - | Absolute path to the IP2Location BIN database file. |
+| `ip_field` | Yes | - | Record field containing the IP address. Supports Fluentd record accessor syntax, for example `ip`, `remote_addr`, or `$.client.ip`. |
+| `output_field` | No | `ip2location` | Destination field for enrichment output. Supports record accessor syntax. Ignored when `merge_record true`. |
+| `fields` | No | `all` | Comma-separated list of IP2Location fields to include. Use `all` to include every supported field. |
+| `merge_record` | No | `false` | When `true`, writes enrichment fields into the root record instead of a nested object. |
+| `prefix` | No | `ip2location_` | Prefix used when `merge_record true`. |
+| `skip_invalid_ip` | No | `true` | When `true`, invalid IP values are ignored. When `false`, invalid IP values raise an error. |
+| `skip_private_ip` | No | `true` | When `true`, private, loopback and link-local IPs are not looked up. |
+| `include_unknown` | No | `false` | When `false`, empty values and `-` values returned by IP2Location are omitted. |
+| `cache_size` | No | `10000` | Maximum number of IP lookup results kept in memory. Set `0` to disable caching. |
+| `on_error` | No | `warn` | Lookup error behavior. One of `ignore`, `warn`, or `raise`. |
+
+## Supported fields
+
+```text
+country_short,country_long,region,city,isp,latitude,longitude,domain,zipcode,timezone,netspeed,iddcode,areacode,weatherstationcode,weatherstationname,mcc,mnc,mobilebrand,elevation,usagetype,addresstype,category,district,asn,as,as_domain,as_usagetype,as_cidr
+```
+
+## Merge fields into the root record
+
+```apache
+<filter app.**>
+  @type ip2location
+  database /opt/ip2location/IPV6-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE-ZIPCODE-TIMEZONE-ISP-DOMAIN-NETSPEED-AREACODE-WEATHER-MOBILE-ELEVATION-USAGETYPE-ADDRESSTYPE-CATEGORY-DISTRICT-ASN.BIN
+  ip_field remote_addr
+  merge_record true
+  prefix geoip_
+  fields country_short,country_long,city,asn,as
+</filter>
+```
+
+Output example:
+
+```json
+{
+  "remote_addr": "8.8.8.8",
+  "geoip_country_short": "US",
+  "geoip_country_long": "United States of America",
+  "geoip_city": "Mountain View",
+  "geoip_asn": "15169",
+  "geoip_as": "Google LLC"
+}
+```
+
+## Development
+
+Install dependencies:
+
+```bash
+bundle install
+```
+
+Run tests:
+
+```bash
+bundle exec rake
+```
+
+Build the gem locally:
+
+```bash
+gem build fluent-plugin-ip2location.gemspec
+```
+
+Install the local gem:
+
+```bash
+gem install fluent-plugin-ip2location-0.1.0.gem
+```
+
+Run the example configuration from the project directory:
+
+```bash
+bundle exec fluentd -c example/fluent.conf -p ./lib
+```
+
+Update the `database` path in `example/fluent.conf` before running it.
+
+## License
+
+LICENSE.txt

data/lib/fluent/plugin/filter_ip2location.rb

@@ -0,0 +1,228 @@
+# frozen_string_literal: true
+
+require "fluent/plugin/filter"
+require "ip2location_ruby"
+require "ipaddr"
+require "thread"
+
+module Fluent
+  module Plugin
+    class IP2LocationFilter < Filter
+      Fluent::Plugin.register_filter("ip2location", self)
+
+      helpers :record_accessor
+
+      KNOWN_FIELDS = %w[
+        country_short
+        country_long
+        region
+        city
+        isp
+        latitude
+        longitude
+        domain
+        zipcode
+        timezone
+        netspeed
+        iddcode
+        areacode
+        weatherstationcode
+        weatherstationname
+        mcc
+        mnc
+        mobilebrand
+        elevation
+        usagetype
+        addresstype
+        category
+        district
+        asn
+        as
+        as_domain
+        as_usagetype
+        as_cidr
+      ].freeze
+
+      VALID_ERROR_MODES = %w[ignore warn raise].freeze
+
+      config_param :database, :string
+      config_param :ip_field, :string
+      config_param :output_field, :string, default: "ip2location"
+      config_param :fields, :string, default: "all"
+      config_param :merge_record, :bool, default: false
+      config_param :prefix, :string, default: "ip2location_"
+      config_param :skip_invalid_ip, :bool, default: true
+      config_param :skip_private_ip, :bool, default: true
+      config_param :include_unknown, :bool, default: false
+      config_param :cache_size, :integer, default: 10_000
+      config_param :on_error, :string, default: "warn"
+
+      def configure(conf)
+        super
+
+        raise Fluent::ConfigError, "database does not exist: #{@database}" unless File.file?(@database)
+        raise Fluent::ConfigError, "cache_size must be zero or greater" if @cache_size.negative?
+        raise Fluent::ConfigError, "on_error must be one of: #{VALID_ERROR_MODES.join(', ')}" unless VALID_ERROR_MODES.include?(@on_error)
+        raise Fluent::ConfigError, "prefix cannot be empty when merge_record is true" if @merge_record && @prefix.empty?
+
+        @selected_fields = parse_fields(@fields)
+        @ip_accessor = record_accessor_create(@ip_field)
+        @output_accessor = record_accessor_create(@output_field) unless @merge_record
+        @cache = {}
+        @cache_order = []
+        @cache_mutex = Mutex.new
+      end
+
+      def start
+        super
+        @ip2location = Ip2location.new.open(@database)
+      rescue StandardError => e
+        raise Fluent::ConfigError, "failed to open IP2Location database #{@database}: #{e.message}"
+      end
+
+      def shutdown
+        @ip2location.close if @ip2location&.respond_to?(:close)
+      ensure
+        super
+      end
+
+      def filter(_tag, _time, record)
+        ip = normalize_ip(@ip_accessor.call(record))
+        return record if ip.nil?
+
+        case ip_status(ip)
+        when :invalid
+          return handle_invalid_ip(record, ip)
+        when :private
+          return record
+        end
+
+        enriched = lookup(ip)
+        return record if enriched.empty?
+
+        if @merge_record
+          enriched.each { |key, value| record["#{@prefix}#{key}"] = value }
+        else
+          @output_accessor.set(record, enriched)
+        end
+
+        record
+      rescue StandardError => e
+        handle_lookup_error(e, ip)
+        record
+      end
+
+      private
+
+      def parse_fields(value)
+        raw = value.to_s.strip
+        return KNOWN_FIELDS if raw.empty? || raw.casecmp("all").zero?
+
+        fields = raw.split(",").map(&:strip).reject(&:empty?)
+        invalid = fields - KNOWN_FIELDS
+        raise Fluent::ConfigError, "unknown fields: #{invalid.join(', ')}. Valid fields: #{KNOWN_FIELDS.join(', ')}" unless invalid.empty?
+
+        fields
+      end
+
+      def normalize_ip(value)
+        return nil if value.nil?
+
+        ip = value.to_s.strip
+        ip.empty? ? nil : ip
+      end
+
+      def ip_status(ip)
+        addr = IPAddr.new(ip)
+        return :private if @skip_private_ip && private_or_local_address?(addr)
+
+        :valid
+      rescue IPAddr::InvalidAddressError
+        :invalid
+      end
+
+      def private_or_local_address?(addr)
+        %i[private? loopback? link_local?].any? do |method_name|
+          addr.respond_to?(method_name) && addr.public_send(method_name)
+        end
+      end
+
+      def handle_invalid_ip(record, ip)
+        if @skip_invalid_ip
+          log.debug "Skipping invalid IP address: #{ip}"
+          return record
+        end
+
+        raise ArgumentError, "invalid IP address: #{ip}"
+      end
+
+      def lookup(ip)
+        cached = cache_read(ip)
+        return cached unless cached.nil?
+
+        raw = @ip2location.get_all(ip)
+        enriched = normalize_record(raw)
+        cache_write(ip, enriched)
+        enriched
+      end
+
+      def normalize_record(raw)
+        hash = raw.respond_to?(:to_h) ? raw.to_h : raw
+        return {} unless hash.respond_to?(:key?)
+
+        @selected_fields.each_with_object({}) do |field, output|
+          next unless hash.key?(field)
+
+          value = hash[field]
+          next if unknown_value?(value)
+
+          output[field] = value
+        end
+      end
+
+      def unknown_value?(value)
+        return false if @include_unknown
+        return true if value.nil?
+
+        string_value = value.to_s.strip
+        string_value.empty? || string_value == "-"
+      end
+
+      def cache_read(ip)
+        return nil if @cache_size.zero?
+
+        @cache_mutex.synchronize { @cache[ip] }
+      end
+
+      def cache_write(ip, value)
+        return if @cache_size.zero?
+
+        @cache_mutex.synchronize do
+          unless @cache.key?(ip)
+            @cache_order << ip
+            if @cache_order.length > @cache_size
+              oldest = @cache_order.shift
+              @cache.delete(oldest)
+            end
+          end
+          @cache[ip] = value
+        end
+      end
+
+      def handle_lookup_error(error, ip)
+        message = "IP2Location lookup failed"
+        message = "#{message} for #{ip}" unless ip.nil?
+        message = "#{message}: #{error.message}"
+
+        case @on_error
+        when "ignore"
+          log.debug message
+        when "warn"
+          log.warn message
+        when "raise"
+          raise error
+        end
+      end
+    end
+  end
+end

data/lib/fluent/plugin/ip2location/version.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module Fluent
+  module Plugin
+    module IP2Location
+      VERSION = "0.1.0"
+    end
+  end
+end

metadata

@@ -0,0 +1,119 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-ip2location
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- IP2Location
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.16'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.16'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '2'
+- !ruby/object:Gem::Dependency
+  name: ip2location_ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.8'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.8'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+- !ruby/object:Gem::Dependency
+  name: test-unit
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.6'
+description: Reads an IP address from a Fluentd record, queries an IP2Location BIN
+  file with ip2location_ruby, and enriches the event record with geolocation and network
+  fields.
+email:
+- support@ip2location.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- lib/fluent/plugin/filter_ip2location.rb
+- lib/fluent/plugin/ip2location/version.rb
+homepage: https://github.com/ip2location/fluent-plugin-ip2location
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/ip2location/fluent-plugin-ip2location
+  source_code_uri: https://github.com/ip2location/fluent-plugin-ip2location
+  changelog_uri: https://github.com/ip2location/fluent-plugin-ip2location/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.15
+specification_version: 4
+summary: Fluentd filter plugin for IP2Location BIN database enrichment.
+test_files: []