fluent-plugin-geoip-kk 1.0.3 → 1.0.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a09c35276c49497e5ce32dfdb4be3ceb9281a4c74061c057ef63e3f2429c597e
-  data.tar.gz: d61d017ac27cd4ec7d7d3b448df7e34713dde36b36792cc7f43ee71dc88dc583
+  metadata.gz: ca6935a0106345ce2856c8f459b8b51b479f2058c07d06a054e65098b656ce8d
+  data.tar.gz: aa54bc540919b711a437803a1ee68c57d1457d315dc51448d3acaabf24197c8f
 SHA512:
-  metadata.gz: '06163815ea4dcd37e4e42bad34d00991f7b38d21a1b05fb9b99f73821c45f570fa1706986b495d9e9225342ec0ea4cba73f4c5d26c92b161a1df19a5fdf77e1d'
-  data.tar.gz: fc25e0ab574149dcf1cc1bcd92e8d2ea6924b9ad2c9e955caca1b871a29aa002f5e802b4920e139b07b407c9587ca36a6ff8cc3b250e27823ac0e97460dd9fcc
+  metadata.gz: be6b1caf1cdb676909b21b19cd5827a232163844f8610f7f4490c40f442e2076ef9e4b40d3ac953d6f734cb88654223265eccf9a16def980342280599c1d4f17
+  data.tar.gz: 1503fab319f62797e5d1fede06acbdbb932b0003ddd6722628f259f391d0bc428d31d1662a5deb6d56c8720eb77df25c240803f3906891069633a427aaaf4760

data/README.md

@@ -9,9 +9,17 @@
 
 | fluent-plugin-geoip-kk | fluentd    | ruby   |
 |----------------------------|------------|--------|
-| >= 1.0.2                   | >= v0.14.0 | >= 2.1 |
+| >= 1.0.5                   | >= v0.14.0 | >= 2.1 |
 | < 1.0.0                    | >= v0.12.0 | >= 1.9 |
 
+## Features
+
+- Automatic database handling (compressed/uncompressed)
+- Memory-optimized database loading
+- LRU caching for high-performance lookups
+- Support for both legacy GeoIP and GeoIP2 databases
+- Private IP address filtering
+- Flexible output formatting (nested/flattened)
 
 ## Installation
 
@@ -29,11 +37,12 @@ $ sudo td-agent-gem install fluent-plugin-geoip-kk
 |-----------|------|---------|-------------|
 | key_name | string | client_ip | Field name that contains the IP address |
 | out_key | string | geo | Output field name to store GeoIP data |
-| database_path | string | data/GeoLite2-City.mmdb | Path to the GeoIP database file |
+| database_path | string | auto-detect | Path to the GeoIP database file (supports .mmdb and .mmdb.gz) |
 | flatten | bool | false | Flatten the GeoIP data structure |
 | cache_size | integer | 8192 | Size of the LRU cache |
 | cache_ttl | integer | 3600 | TTL for cached items in seconds |
 | skip_private_ip | bool | true | Skip adding GeoIP data for private IP addresses |
+| memory_cache | bool | true | Keep database in memory for better performance |
 
 ## Usage Examples
 
@@ -58,11 +67,12 @@ $ sudo td-agent-gem install fluent-plugin-geoip-kk
   out_key geo
 
   # Database configuration
-  database_path /path/to/GeoLite2-City.mmdb
+  database_path /path/to/your/GeoLite2-City.mmdb.gz  # Optional: supports both .mmdb and .mmdb.gz
 
-  # Cache configuration
-  cache_size 10000    # Cache up to 10000 IP addresses
-  cache_ttl 3600      # Cache TTL: 1 hour
+  # Performance optimization
+  memory_cache true     # Keep database in memory (recommended)
+  cache_size 10000     # LRU cache size
+  cache_ttl 3600      # Cache TTL in seconds
 
   # Output configuration
   flatten false       # Keep nested structure
@@ -72,34 +82,12 @@ $ sudo td-agent-gem install fluent-plugin-geoip-kk
 </filter>
 ```
 
-### Input Example
-
-```json
-{
-  "client_ip": "93.184.216.34",
-  "scheme": "http",
-  "method": "GET",
-  "host": "example.com",
-  "path": "/",
-  "query": "-",
-  "req_bytes": 200,
-  "referer": "-",
-  "status": 200,
-  "res_bytes": 800,
-  "res_body_bytes": 600,
-  "taken_time": 0.001,
-  "user_agent": "Mozilla/5.0"
-}
-```
-
-### Output Example (Default Structure)
+### Output Examples
 
+#### Default Structure (flatten: false)
 ```json
 {
   "client_ip": "93.184.216.34",
-  "scheme": "http",
-  "method": "GET",
-  // ... other original fields ...
   "geo": {
     "coordinates": {
       "latitude": 42.150800000000004,
@@ -127,14 +115,10 @@ $ sudo td-agent-gem install fluent-plugin-geoip-kk
 }
 ```
 
-### Output Example (Flattened Structure)
-
-When `flatten true` is specified:
-
+#### Flattened Structure (flatten: true)
 ```json
 {
   "client_ip": "93.184.216.34",
-  // ... other original fields ...
   "geo_coordinates_latitude": 42.150800000000004,
   "geo_coordinates_longitude": -70.8228,
   "geo_coordinates_accuracy_radius": 100,
@@ -152,31 +136,37 @@ When `flatten true` is specified:
 
 ## Performance Optimization
 
-The plugin includes several performance optimizations:
+### Memory Usage vs Performance
+
+The plugin offers two modes for database handling:
 
-1. LRU Cache with TTL
-   - Caches GeoIP lookups to reduce database access
-   - Configurable cache size and TTL
-   - Automatic cache cleanup for expired entries
+1. Memory Mode (Default, Recommended)
+   - Loads entire database into memory
+   - Fastest lookup performance
+   - Higher memory usage (~56MB for GeoLite2-City)
+   - Best for production environments
 
-2. Skip Private IPs
-   - Optionally skip processing private IP addresses
-   - Reduces unnecessary database lookups
+2. File Mode
+   - Keeps database on disk
+   - Lower memory usage
+   - Slightly slower lookups due to disk I/O
+   - Suitable for memory-constrained environments
 
-3. Efficient Record Access
-   - Uses Fluentd's record accessor for optimized field access
-   - Reduces memory allocations
+### Caching Strategy
 
-## VS.
-[fluent-plugin-geoip](https://github.com/y-ken/fluent-plugin-geoip)
-Fluentd output plugin to geolocate with geoip.
-It is able to customize fields with placeholder.
+- LRU (Least Recently Used) cache with TTL
+- Default cache size: 8192 entries
+- Default TTL: 3600 seconds (1 hour)
+- Adjust based on your traffic patterns:
+  - High unique IPs: Increase cache_size
+  - Stable IP patterns: Increase cache_ttl
 
-* Easy to install.
-    * Not require to install Development Tools and geoip-dev library.
-    * ( fluent-plugin-geoip use geoip-c gem but our plugin use geoip. It's conflict. )
-* 5-10 times faster by the LRU cache.
-    * See [benchmark](test/bench_geoip_filter.rb).
+### Database Compression
+
+- Database is distributed in compressed format (.mmdb.gz)
+- Automatic handling of compressed/uncompressed files
+- ~52% size reduction (56MB → 27MB)
+- No performance impact when using memory_cache
 
 ## Development
 
@@ -209,10 +199,6 @@ This gem uses GitHub Actions for automated publishing. To publish a new version:
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create a new Pull Request
 
-## Copyright
-
-Copyright (c) 2015 Yuri Umezaki
-
 ## License
 
 [Apache License, Version 2.0.](http://www.apache.org/licenses/LICENSE-2.0)

data/Rakefile

@@ -12,6 +12,7 @@ Rake::TestTask.new(:test) do |test|
 end
 
 task :push do
+  system 'rubocop -A'
   system 'git add .'
   system "git commit -m 'Update #{Time.now}.'"
   system 'git pull'

data/fluent-plugin-geoip-kk.gemspec

@@ -5,7 +5,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
   spec.name          = 'fluent-plugin-geoip-kk'
-  spec.version       = '1.0.3'
+  spec.version       = '1.0.5'
   spec.authors       = ['kevin197011']
   spec.email         = ['kevin197011@outlook.com']
   spec.homepage      = 'https://github.com/kevin197011/fluent-plugin-geoip-kk'
@@ -13,11 +13,27 @@ Gem::Specification.new do |spec|
   spec.description   = 'A Fluentd filter plugin that adds GeoIP information to records. Supports both legacy GeoIP and GeoIP2 databases, with LRU caching for improved performance.'
   spec.license       = 'Apache-2.0'
 
-  spec.files         = `git ls-files -z`.split("\x0")
+  # Include all git tracked files and vendor/data directory
+  spec.files = Dir[
+    'lib/**/*',
+    'vendor/**/*.mmdb.gz',
+    'Gemfile',
+    'LICENSE',
+    'README.md',
+    'Rakefile',
+    '*.gemspec'
+  ]
+
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ['lib']
 
+  # Ensure vendor/data directory is included
+  spec.metadata = {
+    'data_dir' => 'vendor/data',
+    'database_file' => 'GeoLite2-City.mmdb.gz'
+  }
+
   spec.add_runtime_dependency 'fluentd', ['>= 0.14.0', '< 2']
   spec.add_runtime_dependency 'geoip', '~> 0.1', '>= 0.1.22'
   spec.add_runtime_dependency 'lru_redux', '~> 1.0', '>= 1.0.0'

data/lib/fluent/plugin/filter_geoip.rb

@@ -4,6 +4,8 @@ require 'maxminddb'
 require 'lru_redux'
 require 'fluent/plugin/filter'
 require 'ipaddr'
+require 'zlib'
+require 'stringio'
 
 module Fluent
   module Plugin
@@ -15,9 +17,10 @@ module Fluent
       # Default cache size (number of IP addresses to cache)
       DEFAULT_CACHE_SIZE = 8192
       DEFAULT_CACHE_TTL = 3600 # 1 hour in seconds
+      DEFAULT_DB_FILENAME = 'GeoLite2-City.mmdb'
 
       desc 'Path to the GeoIP database file'
-      config_param :database_path, :string, default: "#{File.dirname(__FILE__)}/../../../data/GeoLite2-City.mmdb"
+      config_param :database_path, :string, default: nil
 
       desc 'Field name that contains the IP address'
       config_param :key_name, :string, default: 'client_ip'
@@ -37,22 +40,37 @@ module Fluent
       desc 'Skip adding GeoIP data for private IP addresses'
       config_param :skip_private_ip, :bool, default: true
 
+      desc 'Keep database in memory'
+      config_param :memory_cache, :bool, default: true
+
       def initialize
         super
         @geoip = nil
         @ip_accessor = nil
+        @database_data = nil
       end
 
       def configure(conf)
         super
 
-        unless File.exist?(@database_path)
-          raise Fluent::ConfigError, "GeoIP database file '#{@database_path}' does not exist"
-        end
+        start_time = Time.now
+        @database_path = resolve_database_path
 
         # Initialize MaxMindDB
         begin
-          @geoip = MaxMindDB.new(@database_path)
+          if @memory_cache
+            @database_data = load_database(@database_path)
+            @geoip = MaxMindDB.new(StringIO.new(@database_data))
+          else
+            @geoip = MaxMindDB.new(decompress_if_needed(@database_path))
+          end
+
+          load_time = Time.now - start_time
+          log.info 'Loaded GeoIP database',
+                   database: @database_path,
+                   size: format_size(@database_data ? @database_data.size : File.size(@database_path)),
+                   compressed_size: format_size(File.size(@database_path)),
+                   load_time: format_time(load_time)
         rescue StandardError => e
           raise Fluent::ConfigError, "Failed to load GeoIP database: #{e.message}"
         end
@@ -63,7 +81,16 @@ module Fluent
         # Initialize cache with TTL support
         @geoip_cache = LruRedux::TTL::Cache.new(@cache_size, @cache_ttl)
 
-        log.info 'Initialized GeoIP filter', database: @database_path, cache_size: @cache_size, cache_ttl: @cache_ttl
+        log.info 'Initialized GeoIP filter',
+                 database: @database_path,
+                 cache_size: @cache_size,
+                 cache_ttl: @cache_ttl,
+                 memory_cache: @memory_cache
+      end
+
+      def shutdown
+        super
+        @temp_db.unlink if @temp_db && File.exist?(@temp_db.path)
       end
 
       def filter(tag, time, record)
@@ -95,29 +122,100 @@ module Fluent
 
       private
 
+      def resolve_database_path
+        return database_path if database_path && !database_path.empty?
+
+        # Search paths in order of preference
+        search_paths = [
+          # 1. Current directory
+          File.join(Dir.pwd, 'vendor/data', DEFAULT_DB_FILENAME),
+          File.join(Dir.pwd, 'vendor/data', "#{DEFAULT_DB_FILENAME}.gz"),
+          # 2. Gem vendor directory
+          File.expand_path("../../../vendor/data/#{DEFAULT_DB_FILENAME}", __FILE__),
+          File.expand_path("../../../vendor/data/#{DEFAULT_DB_FILENAME}.gz", __FILE__),
+          # 3. System-wide locations
+          '/usr/share/GeoIP/GeoLite2-City.mmdb',
+          '/usr/local/share/GeoIP/GeoLite2-City.mmdb'
+        ]
+
+        # Find first existing database file
+        found_path = search_paths.find { |path| File.exist?(path) }
+
+        unless found_path
+          available_paths = search_paths.join("\n  ")
+          raise Fluent::ConfigError, "Could not find GeoIP database file. Searched in:\n  #{available_paths}"
+        end
+
+        log.info 'Found GeoIP database', path: found_path
+        found_path
+      end
+
+      def load_database(path)
+        if path.end_with?('.gz')
+          Zlib::GzipReader.open(path, &:read)
+        else
+          File.binread(path)
+        end
+      end
+
+      def decompress_if_needed(path)
+        return path unless path.end_with?('.gz')
+
+        decompressed_path = path.chomp('.gz')
+        unless File.exist?(decompressed_path)
+          File.open(decompressed_path, 'wb') do |file|
+            Zlib::GzipReader.open(path) do |gz|
+              file.write(gz.read)
+            end
+          end
+        end
+        decompressed_path
+      end
+
+      def format_size(bytes)
+        units = %w[B KB MB GB]
+        size = bytes.to_f
+        unit_index = 0
+
+        while size > 1024 && unit_index < units.length - 1
+          size /= 1024
+          unit_index += 1
+        end
+
+        format('%.2f %s', size, units[unit_index])
+      end
+
+      def format_time(seconds)
+        if seconds < 1
+          format('%.2f ms', seconds * 1000)
+        else
+          format('%.2f s', seconds)
+        end
+      end
+
       def get_geoip(ip_addr)
         geo_ip = @geoip.lookup(ip_addr)
         return {} if geo_ip.nil?
 
         result = {}
 
-        if coordinates = get_coordinates(geo_ip)
+        if (coordinates = get_coordinates(geo_ip))
           result['coordinates'] = coordinates
         end
 
-        if country = get_country_info(geo_ip)
+        if (country = get_country_info(geo_ip))
           result['country'] = country
         end
 
-        if city = get_city_info(geo_ip)
+        if (city = get_city_info(geo_ip))
           result['city'] = city
         end
 
-        if region = get_region_info(geo_ip)
+        if (region = get_region_info(geo_ip))
           result['region'] = region
         end
 
-        if postal = get_postal_info(geo_ip)
+        if (postal = get_postal_info(geo_ip))
           result['postal'] = postal
         end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-geoip-kk
 version: !ruby/object:Gem::Version
-  version: 1.0.3
+  version: 1.0.5
 platform: ruby
 authors:
 - kevin197011
@@ -140,23 +140,19 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- ".github/workflows/gem-push.yml"
-- ".gitignore"
 - Gemfile
 - LICENSE
 - README.md
 - Rakefile
-- data/GeoLite2-City.mmdb
 - fluent-plugin-geoip-kk.gemspec
 - lib/fluent/plugin/filter_geoip.rb
-- push.rb
-- test.rb
-- test/bench_geoip_filter.rb
-- test/helper.rb
+- vendor/data/GeoLite2-City.mmdb.gz
 homepage: https://github.com/kevin197011/fluent-plugin-geoip-kk
 licenses:
 - Apache-2.0
-metadata: {}
+metadata:
+  data_dir: vendor/data
+  database_file: GeoLite2-City.mmdb.gz
 post_install_message:
 rdoc_options: []
 require_paths:
@@ -176,6 +172,4 @@ rubygems_version: 3.4.19
 signing_key:
 specification_version: 4
 summary: Fluentd filter plugin to add geoip
-test_files:
-- test/bench_geoip_filter.rb
-- test/helper.rb
+test_files: []

data/.github/workflows/gem-push.yml

@@ -1,34 +0,0 @@
-name: Ruby Gem
-
-on:
-  push:
-    branches: [ "main" ]
-  pull_request:
-    branches: [ "main" ]
-
-jobs:
-  build:
-    name: Build + Publish
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      packages: write
-
-    steps:
-      - uses: actions/checkout@v4
-
-      - name: Set up Ruby
-        uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: '3.2'   # 你可以指定 3.1、3.2、3.3 等
-
-      - name: Publish to RubyGems
-        run: |
-          mkdir -p $HOME/.gem
-          touch $HOME/.gem/credentials
-          chmod 0600 $HOME/.gem/credentials
-          printf -- "---\n:rubygems_api_key: ${GEM_HOST_API_KEY}\n" > $HOME/.gem/credentials
-          gem build *.gemspec
-          gem push *.gem
-        env:
-          GEM_HOST_API_KEY: "${{secrets.RUBYGEMS_API_KEY}}"

data/.gitignore

@@ -1,22 +0,0 @@
-*.gem
-*.rbc
-.bundle
-.config
-.yardoc
-Gemfile.lock
-InstalledFiles
-_yardoc
-coverage
-doc/
-lib/bundler/man
-pkg
-rdoc
-spec/reports
-test/tmp
-test/version_tmp
-tmp
-*.bundle
-*.so
-*.o
-*.a
-mkmf.log

data/push.rb

@@ -1,3 +0,0 @@
-# frozen_string_literal: true
-
-system 'rake'

data/test/bench_geoip_filter.rb

@@ -1,50 +0,0 @@
-# frozen_string_literal: true
-
-require_relative 'helper'
-require 'fluent/plugin/filter_geoip'
-
-# setup
-Fluent::Test.setup
-config = %(
-  @type geoip
-  out_key geo
-)
-time = Time.now.to_i
-tag = 'foo.bar'
-driver = Fluent::Test::FilterTestDriver.new(Fluent::GeoipFilter, tag).configure(config, true)
-
-# bench
-require 'benchmark'
-require 'ipaddr'
-n = 100_000
-Benchmark.bm(7) do |x|
-  # 8192 random IP
-  x.report do
-    driver.run do
-      n.times do
-        driver.emit({ 'client_ip' => IPAddr.new(rand(2**13) * 2**19, Socket::AF_INET).to_s }, time)
-      end
-    end
-  end
-end
-
-# Without LRU cache
-#              user     system      total        real
-#         11.410000   2.730000  14.140000 ( 15.431248)
-# With LRU cache(8192) & random  1024 IP
-#              user     system      total        real
-#          1.250000   0.070000   1.320000 (  1.322339)
-# With LRU cache(8192) & random  8192 IP
-#              user     system      total        real
-#          1.890000   0.210000   2.100000 (  2.102445)
-# With LRU cache(8192) & random 16384 IP
-#              user     system      total        real
-#          8.450000   2.070000  10.520000 ( 12.170379)
-# With LRU cache(8192) & random 65536 IP
-#              user     system      total        real
-#         11.890000   2.820000  14.710000 ( 16.051674)
-
-# fluent-plugin-geoip
-# (https://github.com/y-ken/fluent-plugin-geoip)
-#              user     system      total        real
-#         11.540000   0.270000  11.810000 ( 12.685000)

data/test/helper.rb

@@ -1,36 +0,0 @@
-# frozen_string_literal: true
-
-require 'rubygems'
-require 'bundler'
-begin
-  Bundler.setup(:default, :development)
-rescue Bundler::BundlerError => e
-  warn e.message
-  warn 'Run `bundle install` to install missing gems'
-  exit e.status_code
-end
-require 'test/unit'
-
-$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
-$LOAD_PATH.unshift(File.dirname(__FILE__))
-require 'fluent/test'
-unless ENV.key?('VERBOSE')
-  nulllogger = Object.new
-  nulllogger.instance_eval do |_obj|
-    def method_missing(method, *args)
-      # pass
-    end
-  end
-  $log = nulllogger
-end
-require 'fluent/test/helpers'
-require 'fluent/test/driver/filter'
-require 'fluent/plugin/filter_geoip'
-
-module Test
-  module Unit
-    class TestCase
-      include Fluent::Test::Helpers
-    end
-  end
-end

data/test.rb

@@ -1,20 +0,0 @@
-# frozen_string_literal: true
-
-require 'maxminddb'
-
-db = MaxMindDB.new('data/GeoLite2-City.mmdb')
-
-ip = '128.101.101.101'
-
-begin
-  result = db.lookup(ip)
-
-  puts "IP: #{ip}"
-  puts "Country: #{result.country.name}"
-  puts "City: #{result.city.name}" if result.city.name
-  puts "Latitude: #{result.location.latitude}" if result.location.latitude
-  puts "Longitude: #{result.location.longitude}" if result.location.longitude
-  puts result.inspect
-rescue MaxMindDB::AddressNotFoundError
-  puts 'IP address not found in the database.'
-end