logstash-filter-ipinfo-db-zuin 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2b36618d5861aa97840f2735fe419fd4303f858f553d883ee8f798c480153666
-  data.tar.gz: a298c76f7316d14c68d9efca5923797c7f6f231758ace241fa7a4c723344181d
+  metadata.gz: 1962e6ef9deb4779538f53751b6a1dc5c923cdefa9f6f55cbbc49ca975508f22
+  data.tar.gz: 03b6b14aaa3f41fd6e79f036c394edc471d8c27f3a295be1e649be9dc52ca6ad
 SHA512:
-  metadata.gz: 7e066fcca849ed60a80c95b745756185d92f5494492df4bd3a7792d011788c2243c3d91a91ee9dd5cb85ab46917b83f38d1b42125fed45f5f7e92d9c9c953040
-  data.tar.gz: 7e38e997a2a75444dfefc2630f1632ff8a1929c5d2e66942135601d6c991e26979eb16ff2f3f3a61d61d4a681fb753026978482da72e842af9085ebb7546a272
+  metadata.gz: dc2dcf4bec7885b0922dd9fc2d3edd58162f239178e98491095bca67771e6480c69a06aac6297eb90e02406c162b1aa9112e29eaea8479242642050bd8435c6d
+  data.tar.gz: b2c07c3692ac8e86c61c8705c1c27d11a8b30b999a4db7630d331b9a18258451918cb0eb3a5192c086e717c33e1b0480df6006e3868b492512984c8a0ab2bc25

data/CHANGELOG.md

@@ -1,3 +1,14 @@
+## 0.4.0
+
+  - Recover from an existing invalid database file (for example truncated or empty): with `token`
+    set, log a warning and download a fresh copy; without it, fail with a message saying to delete or
+    replace the file (#3)
+
+## 0.3.0
+
+  - Declare `required_ruby_version >= 3.1` in the gemspec. Logstash 8.19 (JRuby 9.4, Ruby 3.1
+    compatible) is the oldest runtime verified in CI; Logstash 9.x ships JRuby 10.
+
 ## 0.2.0
 
   - Forked as `logstash-filter-ipinfo-db-zuin` (filter name stays `ipinfo-db`)

data/README.md

@@ -53,13 +53,14 @@ The lookup result is written to the `ipinfo-db` field. For `8.8.8.8` it looks li
 |---|---|---|
 | `source` | (required) | Field containing the IP address (IPv4 or IPv6). Hostnames are not resolved. |
 | `target` | `ipinfo-db` | Field to write the result into. |
-| `token` | none | ipinfo.io token, used only to download the database when the file does not exist. |
-| `database` | `<path.data>/plugins/filters/ipinfo-db/country_asn.mmdb` | Path to the `.mmdb` file. Downloaded there if missing. |
+| `token` | none | ipinfo.io token, used only to download the database: when the file does not exist, or is not a valid database. |
+| `database` | `<path.data>/plugins/filters/ipinfo-db/country_asn.mmdb` | Path to the `.mmdb` file. Downloaded there if missing, or if the file there is not a valid database (with `token`). |
 | `database_age_warning_days` | `30` | Log a warning when the database is older than this. `0` disables it. |
 | `tag_on_failure` | `["_ipinfo_db_lookup_failure"]` | Tags added when the field is missing, the value is not a valid IP, or the lookup fails. IPs that are simply not in the database are not tagged. |
 
 The plugin does not update the database automatically. To refresh it, delete the file and restart
-Logstash, or replace the file with a newer copy. See [docs/index.asciidoc](docs/index.asciidoc) for the
+Logstash (needs `token`, to download it again), or replace the file with a newer copy and restart or
+reload the pipeline. See [docs/index.asciidoc](docs/index.asciidoc) for the
 full reference.
 
 ## Development
@@ -72,6 +73,11 @@ bundle install
 bundle exec rspec
 ```
 
+Run them with Logstash's JRuby and JDK, and pass the JVM flags that `bin/logstash` normally adds itself
+(the `JVM_FLAGS` list in `verify/verify.sh`) through `JAVA_OPTS`. Without them the pipeline specs fail
+with `IllegalAccessError` on `jdk.compiler`. The easiest way to get this right is `verify/verify.sh`
+(below), which runs the suite inside the official Logstash image.
+
 The default suite runs offline against `spec/fixtures/ipinfo-test.mmdb`, a tiny synthetic database
 (documentation IP ranges only) generated by `spec/fixtures/generate_ipinfo_test_mmdb.py`. The live test,
 which downloads the real database from ipinfo.io, is tagged `integration` and excluded by default:
@@ -114,6 +120,17 @@ Or add it to the Logstash `Gemfile` from your local checkout, then run `bin/logs
 gem "logstash-filter-ipinfo-db-zuin", :path => "/path/to/logstash-filter-ipinfo-db"
 ```
 
+## Releasing
+
+1. Set `s.version` in `logstash-filter-ipinfo-db-zuin.gemspec`, and add a `## <version>` section at the
+   top of `CHANGELOG.md` (rename `## Unreleased` if there is one).
+2. Merge to `main`.
+
+`.github/workflows/tag.yml` sees the new gemspec version and pushes `v<version>`. That tag starts
+`.github/workflows/release.yml`, which checks that the tag, gemspec and newest CHANGELOG section agree,
+runs the verification above, publishes the gem to RubyGems via trusted publishing, and creates the
+GitHub Release. Versions with a letter (e.g. `0.4.0.rc1`) are published as pre-releases.
+
 ## Help
 
 Open an issue at https://github.com/reflective-technology/logstash-filter-ipinfo-db/issues, or ask in the

data/docs/index.asciidoc

@@ -63,7 +63,11 @@ filter plugins.
 
 Path to the ipinfo `.mmdb` database file. If the file exists, the plugin uses it as is. If it does not exist,
 the plugin downloads it to this path on startup using <<plugins-{type}s-{plugin}-token>>, creating parent
-directories as needed. `path.data` is the Logstash data directory (for example `/var/lib/logstash` for
+directories as needed. If the file exists but is not a valid `.mmdb` database (for example truncated or
+empty), the plugin logs a warning and downloads a fresh copy when `token` is set; without a token it fails to
+start with a message saying to delete or replace the file.
+
+`path.data` is the Logstash data directory (for example `/var/lib/logstash` for
 deb/rpm installs and `/usr/share/logstash/data` in Docker). Use an absolute path; a relative path is resolved
 against the Logstash working directory.
 

data/lib/logstash/filters/ipinfo-db.rb

@@ -196,25 +196,48 @@ class LogStash::Filters::IpinfoDb < LogStash::Filters::Base
 
     @db = self.class.with_path_lock(path) do
       if File.exist?(path)
-        self.class.acquire_reader(path) || self.class.load_reader(path)
+        self.class.acquire_reader(path) || load_or_replace_invalid(path)
       else
-        if @token.nil? || @token.value.to_s.empty?
-          raise LogStash::ConfigurationError, "ipinfo-db: database file #{@database} not found. Set `token` to download it, or place the file there."
-        end
-        reader, identity = download_database
-        # A rename keeps inode, size and mtime, so a different identity here means
-        # another process replaced the file after our rename: load what is there.
-        if File.exist?(path) && self.class.file_identity(path) == identity
-          self.class.store_reader(path, identity, reader)
-        else
-          self.class.load_reader(path)
+        unless token_set?
+          raise LogStash::ConfigurationError, "ipinfo-db: database file #{path} not found. Set `token` to download it, or place the file there."
         end
+        download_and_cache(path)
       end
     end
   end
 
   private
 
+  def token_set?
+    !(@token.nil? || @token.value.to_s.empty?)
+  end
+
+  # Load an existing database file. If it is not a valid database (for example
+  # truncated by an older version or edited by hand), download a fresh copy when
+  # a token is set; otherwise fail with a message saying how to fix it.
+  def load_or_replace_invalid(path)
+    self.class.load_reader(path)
+  rescue MaxMind::DB::InvalidDatabaseError => e
+    unless token_set?
+      raise LogStash::ConfigurationError, "ipinfo-db: database file #{path} is not a valid .mmdb file (#{e.message}). " \
+                                          "Delete or replace it, or set `token` to download a fresh copy."
+    end
+    @logger.warn("ipinfo database file is not a valid .mmdb file; downloading a fresh copy", :path => path, :error => e.message)
+    download_and_cache(path)
+  end
+
+  # Download the database to path (replacing any file there) and cache its reader.
+  def download_and_cache(path)
+    reader, identity = download_database
+    # A rename keeps inode, size and mtime, so a different identity here means
+    # another process replaced the file after our rename: load what is there.
+    if File.exist?(path) && self.class.file_identity(path) == identity
+      self.class.store_reader(path, identity, reader)
+    else
+      self.class.load_reader(path)
+    end
+  end
+
   # Warn when the loaded database is older than database_age_warning_days,
   # based on the build time stored in the database itself.
   def check_database_age

data/logstash-filter-ipinfo-db-zuin.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-filter-ipinfo-db-zuin'
-  s.version       = '0.2.0'
+  s.version       = '0.4.0'
   s.licenses      = ['Apache-2.0']
   s.summary       = 'Logstash Filter Plugin for IPinfoDb'
   s.description   = 'This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline ' \
@@ -10,6 +10,8 @@ Gem::Specification.new do |s|
   s.authors       = ['ZUIN RD']
   s.email         = 'rd@zuin.ai'
   s.require_paths = ['lib']
+  # Oldest runtime verified in CI: Logstash 8.19 ships JRuby 9.4, which is Ruby 3.1 compatible.
+  s.required_ruby_version = '>= 3.1'
 
   # Files
   s.files = Dir['lib/**/*','spec/**/*','docs/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE']

data/spec/filters/ipinfo-db_spec.rb

@@ -269,6 +269,46 @@ describe LogStash::Filters::IpinfoDb do
     let(:data_dir) { Dir.mktmpdir }
     after { FileUtils.rm_rf(data_dir) }
 
+    def stub_download_of_fixture
+      allow(Down).to receive(:download) do
+        Tempfile.new("ipinfo-download").tap do |file|
+          file.binmode
+          file.write(File.binread(FIXTURE_DATABASE))
+          file.flush
+        end
+      end
+    end
+
+    { "corrupt" => "not a database", "empty" => "" }.each do |kind, content|
+      it "replaces an existing #{kind} database file with a fresh download when a token is set" do
+        database = File.join(data_dir, "country_asn.mmdb")
+        File.write(database, content)
+        stub_download_of_fixture
+        plugin = described_class.new("source" => "ip", "database" => database, "token" => "t", "database_age_warning_days" => 0)
+        expect(plugin.logger).to receive(:warn).with(/not a valid \.mmdb file; downloading a fresh copy/, hash_including(:path => database))
+
+        plugin.register
+
+        expect(Down).to have_received(:download).once
+        expect(File.binread(database)).to eq(File.binread(FIXTURE_DATABASE))
+        expect(Dir.children(data_dir)).to eq(["country_asn.mmdb"])
+        event = LogStash::Event.new("ip" => "192.0.2.10")
+        plugin.filter(event)
+        expect(event.get("[ipinfo-db][country]")).to eq("US")
+      end
+    end
+
+    it "fails with instructions when an existing database file is invalid and no token is set" do
+      database = File.join(data_dir, "country_asn.mmdb")
+      File.write(database, "not a database")
+      expect(Down).not_to receive(:download)
+      plugin = described_class.new("source" => "ip", "database" => database)
+
+      expect { plugin.register }.to raise_error(LogStash::ConfigurationError,
+        /#{Regexp.escape(database)} is not a valid \.mmdb file .*Delete or replace it, or set `token`/)
+      expect(File.read(database)).to eq("not a database") # left for the user to inspect
+    end
+
     it "defaults to a file under path.data" do
       allow(LogStash::SETTINGS).to receive(:get_value).and_call_original
       allow(LogStash::SETTINGS).to receive(:get_value).with("path.data").and_return(data_dir)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-ipinfo-db-zuin
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.4.0
 platform: ruby
 authors:
 - ZUIN RD
@@ -101,7 +101,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '3.1'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="