groonga-query-log 1.7.7 → 1.7.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: af3ac522ec27ea395b44209b8ab56521de14b828d67bf4b8d4dad52bb0df2d1c
-  data.tar.gz: c570a4a49aad267b71047d992e0593afac1290135711a7f45b8114ded4232dbd
+  metadata.gz: d37a2e35f6eb1b0002f81d5d3e5e3f1cbb3115816da95d0d2fba95e40d4602ac
+  data.tar.gz: f790ad37a5cb6748181588534a8ce6fc06979921e0c5eb68e016c56b2df8f906
 SHA512:
-  metadata.gz: 7f0dc1247dc380bd50f8b249fcc7976a800bb67de75f10e4cd600741508f4236d89aa750277043336dc4abd561641804cf53056373338ef9ae2dd2b0f9cc5c0e
-  data.tar.gz: ddcce99bb10e8b8876fbbce95d9c0c49f5d1b8ca7e6ce8beb4a9178624ab169475ce49173c33f1e1a5796909ae20c1fa1976126033c944757859f18e9a05689f
+  metadata.gz: 61c8927fe0bc65eeeb763694af65e2988047264dd1f05a3daf6b2c7fa702f7793e1a9466e5c7764a9f2f48f804a5927093fd62e09414f7ed88184bcee659dbf7
+  data.tar.gz: a7aee9d119a6e1569320bf3e4b2abf2b78e8eccb69c694fcc3850ac37fb3848d8fc01e62c2bbd8e845bb2435f13e7e10c2104f1b01bc2275095ee9f9a8ae134d

data/README.md

@@ -1,6 +1,6 @@
 # README
 
-[![Build Status](https://travis-ci.org/groonga/groonga-query-log.png?branch=master)](https://travis-ci.org/groonga/groonga-query-log)
+[![Test](https://github.com/groonga/groonga-query-log/actions/workflows/test.yml/badge.svg)](https://github.com/groonga/groonga-query-log/actions/workflows/test.yml)
 
 ## Name
 
@@ -24,6 +24,10 @@ by using groonga-query-log as a tool.
 
 TODO...
 
+### groonga-query-log-check-crash
+
+  * [doc/text/check-crash.md](doc/text/check-crash.md)
+
 ### groonga-query-log-detect-memory-leak
 
 TODO...
@@ -54,10 +58,9 @@ TODO...
   * [groonga-command-parser](http://rubygems.org/gems/groonga-command-parser)
   * [groonga-client](http://rubygems.org/gems/groonga-client)
 
-## Mailing list
+## Community
 
-  * English: [groonga-talk@lists.sourceforge.net](https://lists.sourceforge.net/lists/listinfo/groonga-talk)
-  * Japanese: [groonga-dev@lists.sourceforge.jp](http://lists.sourceforge.jp/mailman/listinfo/groonga-dev)
+See [the community documentation](https://groonga.org/docs/community.html) for details.
 
 ## Thanks
 

data/doc/text/check-crash.md

@@ -0,0 +1,152 @@
+# `groonga-query-log-check-crash`
+
+`groonga-query-log-check-crash` is a tool that supports recovery when Groonga crashes.
+
+If Groonga crashes while updating a database, the database may become corrupted.
+It shows tables, columns, and indexes that require recovery, so you can use it when checking what needs to be recovered.
+
+## Usage
+
+This section describes how to use `groonga-query-log-check-crash`.
+
+You need both the process log and the query log to run `groonga-query-log-check-crash`.
+For log output settings and content, see the Groonga documentation: https://groonga.org/docs/reference/log.html
+
+Specify the process log and query log. There is no fixed order for specifying them, so specify all logs.
+
+Command example:
+
+```bash
+groonga-query-log-check-crash \
+  path/to/process-log-2025-* \
+  path/to/query-log-2025-* \
+  path/to/process-log-2026-* \
+  path/to/query-log-2026-*
+```
+
+### Options
+
+#### `--command-format`
+
+This script displays any Groonga commands that terminated abnormally.
+Specify the format of that command.
+
+You can specify `command` or `uri`, and output will be as follows.
+
+`--command-format=command`:
+
+```
+===
+[unflushed] 2000-01-01T00:00:01+09:00: 
+load \
+  --table "Data"
+===
+```
+
+`--command-format=uri`:
+
+```
+===
+[unflushed] 2000-01-01T00:00:01+09:00: /d/load?table=Data
+===
+```
+
+The default is `command`.
+When using it as HTTP server, specifying `uri` makes it easier to use when re-running.
+
+#### `--pretty-print`
+
+This is only available when `command` is specified in `--command-format`.
+
+By default, `--pretty-print` is enabled. Disabling it will output Groonga commands on a single line.
+
+`--no-pretty-print`:
+
+```
+[unflushed] 2000-01-01T00:00:01+09:00: load --table "Data"
+```
+
+#### `--output-level`
+
+You can specify `info` or `debug`. The default value is `info`.
+Please use `info` as the default. `debug` provides more detailed output for developers.
+
+## Reading the Output
+
+### Summary
+
+The `Summary` includes the following items.
+
+* `crashed`
+  * In the case of `yes`, Groonga has crashed.
+  * Please report it with the logs.
+* `unflushed`
+  * In the case of `yes`, data in memory (non-persistent data) may not have been written to disk (persistent data).
+  * Please re-run the target command.
+* `unfinished`
+  * In the case of `yes`, the target table, column, or index may be broken.
+  * Please rebuild the target table, column, or index.
+* `leak`
+  * In the case of `yes`, Groonga is leaking memory.
+  * Please report it with the logs.
+
+Example:
+
+```
+Summary:
+crashed:yes, unflushed:yes, unfinished:no, leak:no
+```
+
+### Unflushed
+
+If the target command exists, the following output is displayed.
+
+```
+!!!
+!!! [unflushed] Recovery information
+!!!
+There may be commands that were not flushed between 2000-01-01T00:00:00+09:00 and 2000-01-01T12:00:00+09:00.
+These commands may not have been written to the database files, so please re-run them.
+===
+[unflushed] 2000-01-01T00:00:01+09:00: load --table "Data"
+===
+
+Summary:
+crashed:yes, unflushed:yes, unfinished:no, leak:no
+NG: Please check the display and logs.
+```
+
+Lines beginning with `[unflushed]` are commands that may not have been flushed.
+Since it may not have been flushed, please re-run that command.
+
+### Unfinished
+
+If the target command exists, the following output is displayed.
+
+```
+!!!
+!!! [unfinished] Recovery information
+!!!
+Unfinished commands were found due to abnormal termination or other issues.
+It is safer to rebuild the target tables, columns, and indexes because the data may be corrupted.
+===
+[unfinished] 2000-01-01T00:00:01+09:00: load --table "Data"
+===
+
+Summary:
+crashed:yes, unflushed:no, unfinished:yes, leak:no
+NG: Please check the display and logs.
+```
+
+Lines beginning with `[unfinished]` are commands that did not complete normally due to abnormal termination or other issues.
+In this case, `Data` and any dependent columns or indexes may be broken and need to be rebuilt.
+
+### Others
+
+If any output other than `unflushed` or `unfinished` was displayed, there might have been an issue with Groonga.
+Please report the problem to the Groonga team using the links below, and include the logs used for the check if possible.
+
+* [Groonga issues](https://github.com/groonga/groonga/issues)
+* [Groonga discussions](https://github.com/groonga/groonga/discussions)
+
+This will help with investigating the issue.

data/doc/text/news.md

@@ -1,5 +1,34 @@
 # News
 
+## 1.7.9: 2026-01-27
+
+### Improvements
+
+  * `check-crash`: Improved the output. Made it easier to use for recovery.
+
+    * Check the [documentation](check-crash.md) as some options have been added.
+
+  * `check-crash`: Improved the accuracy of unflushed checks.
+
+    * Improved detection accuracy when no `target_name` is specified for `io_flush`.
+
+    * Added a `select` command with `--load_table` option specified as the check target for unflushed.
+
+  * `check-crash`: Showed PID and thread ID
+
+  * Added support for Ruby 3.4
+
+### Fixes
+
+  * `show-running-queries`: Fixed running query condition
+
+## 1.7.8: 2024-04-10
+
+### Improvements
+
+  * `run-regression-test`: Improved processing time for `request_cancel` command verification.
+    * Eliminated unnecessary waiting time.
+
 ## 1.7.7: 2024-04-08
 
 ### Improvements

data/groonga-query-log.gemspec

@@ -1,6 +1,6 @@
 # -*- ruby -*-
 #
-# Copyright (C) 2012-2019  Sutou Kouhei <kou@clear-code.com>
+# Copyright (C) 2012-2025  Sutou Kouhei <kou@clear-code.com>
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -52,11 +52,13 @@ Gem::Specification.new do |spec|
   spec.licenses = ["LGPLv2.1+"]
   spec.require_paths = ["lib"]
 
+  spec.add_runtime_dependency("base64")
   spec.add_runtime_dependency("charty")
   spec.add_runtime_dependency("diff-lcs")
-  spec.add_runtime_dependency("net-smtp")
   spec.add_runtime_dependency("groonga-client", ">= 0.6.2")
   spec.add_runtime_dependency("groonga-log", ">= 0.1.2")
+  spec.add_runtime_dependency("net-smtp")
+  spec.add_runtime_dependency("ostruct")
 
   spec.add_development_dependency("test-unit")
   spec.add_development_dependency("test-unit-rr")

data/lib/groonga-query-log/command/check-crash.rb

@@ -1,4 +1,4 @@
-# Copyright (C) 2018  Kouhei Sutou <kou@clear-code.com>
+# Copyright (C) 2018-2025  Sutou Kouhei <kou@clear-code.com>
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -34,6 +34,11 @@ module GroongaQueryLog
           return false
         end
 
+        if log_paths.empty?
+          puts(@option_parser.help)
+          return false
+        end
+
         begin
           check(log_paths)
         rescue Interrupt
@@ -47,11 +52,35 @@ module GroongaQueryLog
 
       private
       def setup_options
-        @options = {}
+        available_command_format = [:command, :uri]
+        available_output_levels = [:info, :debug]
+        @options = {
+          command_format: :command,
+          output_level: :info,
+          pretty_print: true
+        }
 
         @option_parser = OptionParser.new do |parser|
           parser.version = VERSION
           parser.banner += " LOG1 ..."
+          parser.on("--command-format=FORMAT",
+                    available_command_format,
+                    "Specify the output format of the Groonga command that had a problem. [#{@options[:command_format]}]",
+                    "(#{available_command_format.join(", ")})") do |format|
+            @options[:command_format] = format
+          end
+          parser.on("--[no-]pretty-print",
+                    "Specify to make command output easier to read. [#{@options[:pretty_print]}]",
+                    "Only available when `--command-format=command` is specified.") do |boolean|
+            @options[:pretty_print] = boolean
+          end
+          parser.on("--output-level=LEVEL",
+                    available_output_levels,
+                    "Specify the output level. [#{@options[:output_level]}]",
+                    "Specifying 'debug' displays detailed information.",
+                    "(#{available_output_levels.join(", ")})") do |output_level|
+            @options[:output_level] = output_level
+          end
         end
       end
 
@@ -66,7 +95,7 @@ module GroongaQueryLog
       end
 
       def check(log_paths)
-        checker = Checker.new(log_paths)
+        checker = Checker.new(log_paths, @options)
         checker.check
       end
 
@@ -111,58 +140,62 @@ module GroongaQueryLog
       end
 
       class Checker
-        def initialize(log_paths)
+        def initialize(log_paths, options)
           split_log_paths(log_paths)
+          @options = options
         end
 
         def check
+          summary = {
+            crashed: false,
+            unflushed: false,
+            unfinished: false,
+            leak: false,
+          }
           processes = ProcessEnumerator.new(@general_log_paths)
           processes.each do |process|
             need_query_log_parsing = true
             if process.successfully_finished?
               need_query_log_parsing = false
-              p [:process,
-                 :success,
-                 process.version,
-                 process.start_time.iso8601,
-                 process.end_time.iso8601,
-                 process.pid,
-                 process.start_log_path,
-                 process.end_log_path]
+              debug([:process,
+                     :success,
+                     process.version,
+                     process.start_time.iso8601,
+                     process.end_time.iso8601,
+                     process.pid,
+                     process.start_log_path,
+                     process.end_log_path].inspect)
             elsif process.crashed?
-              p [:process,
-                 :crashed,
-                 process.version,
-                 process.start_time.iso8601,
-                 process.end_time.iso8601,
-                 process.pid,
-                 process.start_log_path,
-                 process.end_log_path]
+              debug([:process,
+                     :crashed,
+                     process.version,
+                     process.start_time.iso8601,
+                     process.end_time.iso8601,
+                     process.pid,
+                     process.start_log_path,
+                     process.end_log_path].inspect)
+              summary[:crashed] = true
             else
-              p [:process,
-                 :unfinished,
-                 process.version,
-                 process.start_time.iso8601,
-                 process.pid,
-                 process.start_log_path]
+              debug([:process,
+                     :unfinished,
+                     process.version,
+                     process.start_time.iso8601,
+                     process.pid,
+                     process.start_log_path].inspect)
             end
 
             unless process.n_leaks.zero?
-              p [:leak,
-                 process.version,
-                 process.n_leaks,
-                 process.end_time.iso8601,
-                 process.pid,
-                 process.end_log_path]
+              debug([:leak,
+                     process.version,
+                     process.n_leaks,
+                     process.end_time.iso8601,
+                     process.pid,
+                     process.end_log_path].inspect)
+              summary[:leak] = true
             end
 
             unless process.important_entries.empty?
-              puts("Important entries:")
-              process.important_entries.each_with_index do |entry, i|
-                puts("#{entry.timestamp.iso8601}: " +
-                     "#{entry.log_level}: " +
-                     "#{entry.message}")
-              end
+              output_important_entries_info(process.important_entries)
             end
 
             next unless need_query_log_parsing
@@ -183,20 +216,22 @@ module GroongaQueryLog
               statistic.start_time < start_time
             end
             unless target_parsing_statistics.empty?
-              puts("Running queries:")
-              target_parsing_statistics.each do |statistic|
-                puts("#{statistic.start_time.iso8601}:")
-                puts(statistic.command.to_command_format(pretty_print: true))
-              end
+              summary[:unfinished] = true
+              output_unfinished_info(target_parsing_statistics)
             end
             unless @unflushed_statistics.empty?
-              puts("Unflushed commands in " +
-                   "#{start_time.iso8601}/#{end_time.iso8601}")
-              @unflushed_statistics.each do |statistic|
-                puts("#{statistic.start_time.iso8601}: #{statistic.raw_command}")
-              end
+              summary[:unflushed] = true
+              output_unflushed_info(start_time, end_time)
             end
           end
+          info("",
+               "Summary:",
+               summary.map {|k, v| "#{k}:#{v ? "yes" : "no"}" }.join(", "))
+          if summary.value?(true)
+            info("NG: Please check the display and logs.")
+          else
+            info("OK: no problems.")
+          end
         end
 
         private
@@ -215,6 +250,21 @@ module GroongaQueryLog
           end
         end
 
+        def formated_command(command)
+          if @options[:command_format] == :uri
+            command.to_uri_format
+          else
+            (@options[:pretty_print] ? "\n" : "") +
+              command.to_command_format(pretty_print: @options[:pretty_print])
+          end
+        end
+
+        def with_load?(statistic)
+          statistic.operations.any? do |operation|
+            operation[:name] == "load"
+          end
+        end
+
         def check_query_log_statistic(path, statistic)
           command = statistic.command
           return if command.nil?
@@ -230,7 +280,7 @@ module GroongaQueryLog
             @flushed = false
             @unflushed_statistics << statistic
           when "io_flush"
-            check_io_flush(command)
+            check_io_flush(statistic)
           when "database_unmap"
             @unflushed_statistics.reject! do |statistic|
               command.name == "load"
@@ -246,33 +296,65 @@ module GroongaQueryLog
           when "plugin_register", "plugin_unregister"
             @flushed = false
             @unflushed_statistics << statistic
+          when "select"
+            if with_load?(statistic)
+              @flushed = false
+              @unflushed_statistics << statistic
+            end
+          end
+        end
+
+        def flushed_objects(statistic)
+          objects = {}
+          statistic.operations.each do |operation|
+            object = operation[:name][/\Aflush\[(.+)\]/, 1]
+            objects[object] = true unless object.nil?
+          end
+          objects
+        end
+
+        def load_check_keys(statistic)
+          case statistic.command.command_name
+          when "load"
+            [:table, :columns]
+          when "select"
+            [:load_table, :load_columns]
+          else
+            raise "Unsupported command name"
           end
         end
 
-        def check_io_flush(io_flush)
-          # TODO: Improve flushed target detection.
+        def flushed_load?(statistic, flushed)
+          table_key, columns_key = load_check_keys(statistic)
+
+          table_name = statistic.command.arguments[table_key]
+          return false unless flushed.key?(table_name)
+          columns = statistic.command.arguments[columns_key]
+          return true unless columns
+          columns.split(",").each do |name|
+            name.strip!
+            next if name == "_key"
+            return false unless flushed["#{table_name}.#{name}"]
+          end
+          true
+        end
+
+        def check_io_flush(io_flush_statistic)
+          io_flush = io_flush_statistic.command
           if io_flush.target_name
-            if io_flush.recursive?
+            if io_flush.recursive_dependent?
               @unflushed_statistics.reject! do |statistic|
                 case statistic.command.command_name
-                when "load"
-                  # TODO: Not enough
-                  statistic.command.table == io_flush.target_name
+                when "load", "select"
+                  flushed_load?(statistic, flushed_objects(io_flush_statistic))
                 when "delete"
-                  # TODO: Not enough
                   statistic.command.table == io_flush.target_name
                 when "truncate"
-                  # TODO: Not enough
                   statistic.command.target_name == io_flush.target_name
-                else
-                  false
-                end
-              end
-            else
-              @unflushed_statistics.reject! do |statistic|
-                case statistic.command.command_name
-                when /_create/
-                  true # TODO: Need io_flush for database
+                when "table_create"
+                  statistic.command.name == io_flush.target_name
+                when "column_create"
+                  "#{statistic.command.table}.#{statistic.command.name}" == io_flush.target_name
                 else
                   false
                 end
@@ -280,24 +362,101 @@ module GroongaQueryLog
             end
           else
             if io_flush.recursive?
-              @unflushed_statistics.clear
-            else
+              flushed = flushed_objects(io_flush_statistic)
               @unflushed_statistics.reject! do |statistic|
                 case statistic.command.command_name
-                when /_create\z/
-                  true # TODO: Need io_flush for the target
-                when /_remove\z/, /_rename\z/
-                  true
-                when "plugin_register", "plugin_unregister"
-                  true
+                when "load", "select"
+                  # TODO: Not enough
+                  flushed_load?(statistic, flushed_objects(io_flush_statistic))
+                when "delete"
+                  # TODO: Not enough
+                  flushed.key?(statistic.command.table)
+                when "truncate"
+                  # TODO: Not enough
+                  flushed.key?(statistic.command.target_name)
+                when "table_create"
+                  # TODO: Not enough
+                  flushed.key?(statistic.command.name)
+                when "column_create"
+                  # TODO: Not enough
+                  flushed.key?("#{statistic.command.table}.#{statistic.command.name}")
                 else
                   false
                 end
               end
             end
+            @unflushed_statistics.reject! do |statistic|
+              case statistic.command.command_name
+              when /_remove\z/, /_rename\z/
+                true
+              when "plugin_register", "plugin_unregister"
+                true
+              else
+                false
+              end
+            end
           end
           @flushed = @unflushed_statistics.empty?
         end
+
+        def debug(*messages)
+          return unless @options[:output_level] == :debug
+          puts(*messages)
+        end
+
+        def info(*messages)
+          puts(*messages)
+        end
+
+        def output_important_entries_info(entries)
+          info("",
+               "!!!",
+               "!!! Important entries",
+               "!!!",
+               "It contained logs that require checking.",
+               "If you need help, please feel free to contact the community: https://groonga.org/docs/community.html",
+               "===")
+          entries.each do |entry|
+            info("#{entry.timestamp.iso8601}: " +
+                 "#{entry.pid}: " +
+                 "#{entry.thread_id}: " +
+                 "#{entry.log_level}: " +
+                 "#{entry.message}")
+          end
+          info("===")
+        end
+
+        def output_statistics_info(message, tag, statistics)
+          info(message, "===")
+          statistics.each do |statistic|
+            info("[#{tag}] #{statistic.start_time.iso8601}: #{formated_command(statistic.command)}")
+          end
+          info("===")
+        end
+
+        def output_unfinished_info(statistics)
+          message = <<-MESSAGE
+
+!!!
+!!! [unfinished] Recovery information
+!!!
+Unfinished commands were found due to abnormal termination or other issues.
+It is safer to rebuild the target tables, columns, and indexes because the data may be corrupted.
+          MESSAGE
+          output_statistics_info(message, "unfinished", statistics)
+        end
+
+        def output_unflushed_info(start_time, end_time)
+          message = <<-MESSAGE
+
+!!!
+!!! [unflushed] Recovery information
+!!!
+There may be commands that were not flushed between #{start_time.iso8601} and #{end_time.iso8601}.
+These commands may not have been written to the database files, so please re-run them.
+          MESSAGE
+          output_statistics_info(message, "unflushed", @unflushed_statistics)
+        end
       end
 
       class ProcessEnumerator

data/lib/groonga-query-log/command/show-running-queries.rb

@@ -61,7 +61,7 @@ module GroongaQueryLog
                 parser.parse(input) do |statistic|
                   next if @base_time.nil?
                   next if statistic.start_time < @base_time
-                  if statistic.start_time == @base_time
+                  if statistic.end_time > @base_time
                     yield(statistic)
                   end
                   throw(tag)