neo4j_bolt 0.4.0 → 0.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a0582f714d9d1c8793edb70481db6a2ff5548200b14ad55c1f44b0859270b7d9
-  data.tar.gz: 73e300f77c941909738411fdbd9ce6b72ea6ef73a72a10184395e2228ce6a52c
+  metadata.gz: 66da85ac04ffd0ef43fea4fba9a676ea2168770f85825573652a58dbc0a48f2b
+  data.tar.gz: 132911b813e88d5a3efb590fe5cd29da0c8adb6663b06b9931bb0a03f4fcae9c
 SHA512:
-  metadata.gz: 8e5debab26d8b655b1fa3fc1e4c877308e8221345aec76844d2b423ef4fe3061bc535241566f9af63ba889274b944c81efbad9daeadf1cc9b0ed166c82a18417
-  data.tar.gz: 65ed32647603bdeaf9bbca19335a0ca6fdccf5b17f3aaa11e19ef48868b242731a34b3cba74ff9475dbd5db94e6bf15f44f60110dca5adb8fe83d9709ef45a3d
+  metadata.gz: 37e9205800f8661683bc246c6a962a15254c2d1e4a33d5b948f85d5e59d9ac38911e67c4a2e18bb318bc9f3ceeb3eedb7199004e98b0ce1694ae2e0f16f9ce8e
+  data.tar.gz: 3f7066485f251b5939b9a448b441290f16714311ebd7baa2b2baa27d785ae2463902c63b6c14b532df5adfe574242c8e415305967677110d92943a431e4360d3

data/README.md

@@ -4,14 +4,14 @@ Neo4jBolt 0.4 is a small compatibility and convenience layer for Ruby applicatio
 
 Neo4jBolt no longer implements the Bolt wire protocol itself.
 
-`0.4.0.pre1` requires Ruby 3.4 or newer and pins `neo4j-ruby-driver` to `6.2.1.beta.4`. Applications on older Rubies can remain on the Neo4jBolt 0.3.x line; this prerelease is intentionally not an automatic upgrade for them.
+`0.4.0` requires Ruby 3.4 or newer and pins `neo4j-ruby-driver` to `6.2.1.beta.4`. Applications on older Rubies can remain on the Neo4jBolt 0.3.x line; this prerelease is intentionally not an automatic upgrade for them.
 
 ## Installation
 
 For this prerelease, specify the version explicitly:
 
 ```ruby
-gem "neo4j_bolt", "0.4.0.pre1"
+gem "neo4j_bolt", "0.4.0"
 ```
 
 Then run `bundle install`. A running Neo4j database is required.
@@ -151,13 +151,21 @@ r {"from":0,"to":1,"type":"KNOWS","properties":{"since":2024}}
 ```
 
 ```ruby
-File.open("database.dump", "w") { |io| dump_database(io) }
-File.open("database.dump", "r") { |io| load_database_dump(io) }
+File.open("database.dump", "w") { |io| dump_database(io, progress_io: $stderr) }
+File.open("database.dump", "r") do |io|
+  load_database_dump(io, progress_io: $stderr)
+end
 ```
 
-The `0, 1, 2, ...` IDs are synthetic dump-local IDs. Database-internal IDs and element IDs are never written to the persistent format. The adapter uses driver-provided entity identity while dumping and a random temporary property (removed in `ensure`) while loading, so relationship reconstruction does not depend on deprecated persistent integer IDs. To keep dump numbering deterministic, ordering uses `elementId()` on modern Neo4j and an internal `id()` fallback only on Neo4j 4.4, where `elementId()` does not exist. Old Neo4jBolt dumps remain loadable.
+Passing `progress_io:` is optional for the Ruby API. The CLI always reports dump/load progress on stderr, so dump data written to stdout remains safe to redirect or pipe. In `auto` mode a terminal gets a colored, in-place progress bar while redirected stderr receives periodic plain-text progress lines. Set `NEO4J_BOLT_PROGRESS=pretty` to force the terminal display through a container or other wrapper that hides the TTY, or `NEO4J_BOLT_PROGRESS=plain` to force log-friendly output. `NO_COLOR` disables colors without disabling the in-place display.
+
+The `0, 1, 2, ...` IDs are synthetic dump-local IDs. Database-internal IDs and element IDs are never written to the persistent format. Dumping keeps the count, node, and relationship reads in one transaction, so the driver-provided entity identity used to connect the two streamed result sets stays within Neo4j's transaction-scoped identity guarantee. To keep dump numbering deterministic, ordering uses `elementId()` on modern Neo4j and an internal `id()` fallback only on Neo4j 4.4, where `elementId()` does not exist. Old Neo4jBolt dumps remain loadable.
+
+For relational loads, the adapter assigns a random temporary label and dump-ID property to imported nodes. After the nodes are committed it builds a temporary `neo4j_bolt_` index on that property, uses indexed lookups while creating relationships, drops the index, and removes the temporary metadata in batches. No database-internal identity is carried from one load transaction to another. Loading a relational dump therefore requires permission to create and drop an index.
 
-Loading requires an empty database unless `force_append: true` is passed.
+Loads start with batches of 5,000 records by default. This is an initial ceiling rather than a claimed optimum: if Neo4j returns a transaction-memory/resource error whose server semantics guarantee rollback, the loader halves the failed batch and retries it, then keeps the smaller size for the rest of that phase. Node and relationship phases adapt independently. Other errors are not retried, because a generic connection failure cannot safely prove that a `CREATE` transaction did not commit. Callers can change the initial ceiling with `initial_batch_size:`; the CLI exposes the same setting as `--batch-size`.
+
+Loading requires an empty database unless `force_append: true` is passed. `load_database_dump` owns its batch transactions and therefore rejects being called from inside `transaction`.
 
 ## CLI
 
@@ -168,7 +176,7 @@ The `neo4j_bolt` executable retains these commands:
 | `neo4j_bolt console` | Open an IRB console with Neo4jBolt loaded |
 | `neo4j_bolt clear --srsly` | Delete all nodes and relationships |
 | `neo4j_bolt dump` | Write the textual database dump |
-| `neo4j_bolt load [--force] PATH` | Load a textual dump |
+| `neo4j_bolt load [--force] [--batch-size N] PATH` | Load a textual dump |
 | `neo4j_bolt index ls` | List constraints and indexes |
 | `neo4j_bolt index rm --force` | Remove all constraints and indexes |
 | `neo4j_bolt visualize` | Generate a GraphViz document |
@@ -183,7 +191,7 @@ The same complete integration suite is run against these exact Community images:
 - `neo4j:5.26.28-community`
 - `neo4j:2026.06.0-community`
 
-No compatibility beyond this matrix is claimed for `0.4.0.pre1`.
+No compatibility beyond this matrix is claimed for `0.4.0`.
 
 Run one modern LTS target:
 
@@ -220,7 +228,7 @@ To point RSpec itself at an already disposable database, set `NEO4J_BOLT_TEST_HO
 | CLI commands | Preserved |
 | `BoltSocket`, `BoltBuffer`, protocol markers/state/parser/packer | Intentionally removed private implementation details |
 
-## 0.4.0.pre1 migration notes
+## 0.4.0 migration notes
 
 - Ruby 3.4 or newer is required; Ruby 2.x/3.0–3.3 applications should stay on 0.3.x until upgraded.
 - The exact prerelease upstream dependency is pinned while no stable `neo4j-ruby-driver` 6.2.x exists.
@@ -233,3 +241,4 @@ To point RSpec itself at an already disposable database, set `NEO4J_BOLT_TEST_HO
 Run `bin/setup`, then `bundle exec rake spec` or `bundle exec rake spec:matrix`. Build without publishing with `bundle exec rake build`.
 
 Bug reports and pull requests are welcome at <https://github.com/specht/neo4j_bolt>.
+

data/bin/neo4j_bolt

@@ -49,7 +49,7 @@ class App
         c.flag [:o, 'out-file'.to_sym], :default_value => '/dev/stdout'
         c.action do |global_options, options|
             File.open(options['out-file'.to_sym], 'w') do |f|
-                dump_database(f)
+                dump_database(f, progress_io: $stderr)
             end
         end
     end
@@ -61,11 +61,18 @@ class App
     command :load do |c|
         # c.flag [:i, :in_file], :desc => 'input path', :required => true
         c.switch [:f, :force], :default_value => false, :desc => 'force appending nodes even if the database is not empty'
+        c.flag [:b, 'batch-size'.to_sym], :default_value => Neo4jBolt::LOAD_INITIAL_BATCH_SIZE,
+               :desc => 'initial batch size; automatically reduced on transaction-memory errors'
         c.action do |global_options, options, args|
             help_now!('input path is required') if args.empty?
             path = args.shift
             File.open(path, 'r') do |f|
-                load_database_dump(f, force_append: options[:force])
+                load_database_dump(
+                    f,
+                    force_append: options[:force],
+                    progress_io: $stderr,
+                    initial_batch_size: options['batch-size'.to_sym]
+                )
             end
         end
     end

data/lib/neo4j_bolt/version.rb

@@ -1,3 +1,3 @@
 module Neo4jBolt
-    VERSION = "0.4.0"
+    VERSION = "0.4.2"
 end

data/lib/neo4j_bolt.rb

@@ -9,6 +9,201 @@ require "neo4j_bolt/version"
 
 module Neo4jBolt
   CONSTRAINT_INDEX_PREFIX = "neo4j_bolt_"
+  LOAD_INITIAL_BATCH_SIZE = 5_000
+  LOAD_MEMORY_ERROR_CODES = %w[
+    Neo.ClientError.General.TransactionOutOfMemoryError
+    Neo.TransientError.General.MemoryPoolOutOfMemoryError
+    Neo.TransientError.General.TransactionMemoryLimit
+  ].freeze
+  LOAD_MEMORY_GQL_STATUSES = %w[51N72 51N73].freeze
+  LOAD_PROGRESS_STEP = 5_000
+  DUMP_PROGRESS_STEP = 1_000
+  private_constant :LOAD_MEMORY_ERROR_CODES, :LOAD_MEMORY_GQL_STATUSES,
+                   :LOAD_PROGRESS_STEP, :DUMP_PROGRESS_STEP
+
+  class ProgressReporter
+    UPDATE_INTERVAL = 0.5
+    BAR_WIDTH = 24
+    SPINNER = %w[⠋ ⠙ ⠹ ⠸ ⠼ ⠴ ⠦ ⠧ ⠇ ⠏].freeze
+    ANSI = {
+      reset: "\e[0m",
+      bold: "\e[1m",
+      dim: "\e[2m",
+      cyan: "\e[36m",
+      green: "\e[32m",
+      yellow: "\e[33m"
+    }.freeze
+
+    def initialize(io)
+      @io = io
+      mode = ENV.fetch("NEO4J_BOLT_PROGRESS", "auto")
+      unless %w[auto pretty plain].include?(mode)
+        raise ArgumentError, "NEO4J_BOLT_PROGRESS must be auto, pretty, or plain"
+      end
+
+      detected_tty = io && io.respond_to?(:tty?) && io.tty? && Neo4jBolt.bolt_verbosity.to_i.zero?
+      @pretty = io && (mode == "pretty" || (mode == "auto" && detected_tty))
+      @color = @pretty && !ENV.key?("NO_COLOR") && (mode == "pretty" || ENV["TERM"] != "dumb")
+      @last_update_at = nil
+      @line_open = false
+      @spinner_index = 0
+    end
+
+    def update(message, force: false, percent: nil)
+      return unless @io
+
+      now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      return if !force && @last_update_at && now - @last_update_at < UPDATE_INTERVAL
+
+      if @pretty
+        percent ||= extract_percent(message)
+        render_line(activity_line(message, percent))
+      else
+        @io.puts message
+      end
+      @last_update_at = now
+    end
+
+    def note(message, level: :info)
+      return unless @io
+
+      if @pretty && level == :info
+        update(message, force: true)
+      else
+        clear_line
+        if @pretty
+          icon = level == :warning ? color(:yellow, "!") : color(:cyan, "•")
+          @io.puts "#{icon} #{terminal_text(message)}"
+        else
+          @io.puts message
+        end
+        @last_update_at = nil
+      end
+    end
+
+    def finish(message)
+      return unless @io
+
+      if @pretty
+        clear_line
+        label, detail = terminal_parts(message)
+        line = "#{color(:green, '✓')} #{color(:bold, label)}"
+        line += "  #{detail}" unless detail.empty?
+        @io.puts line
+        @line_open = false
+      else
+        @io.puts message
+      end
+      @last_update_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
+
+    def close
+      return unless @io && @pretty && @line_open
+
+      clear_line
+    end
+
+    private
+
+    def activity_line(message, percent)
+      label, detail = terminal_parts(message, strip_percent: !percent.nil?)
+      spinner = SPINNER[@spinner_index % SPINNER.length]
+      @spinner_index += 1
+
+      parts = [color(:cyan, spinner), color(:bold, label)]
+      if percent
+        percent = [[percent.to_i, 0].max, 100].min
+        parts << progress_bar(percent)
+        parts << color(:bold, format("%3d%%", percent))
+      end
+      parts << detail unless detail.empty?
+      parts.join("  ")
+    end
+
+    def progress_bar(percent)
+      filled = (percent * BAR_WIDTH / 100.0).round
+      empty = BAR_WIDTH - filled
+      "#{color(:cyan, '━' * filled)}#{color(:dim, '─' * empty)}"
+    end
+
+    def terminal_parts(message, strip_percent: false)
+      text = message.sub(/\.\.\.\z/, "")
+      label, detail = text.split(": ", 2)
+      label = pretty_label(label)
+      detail ||= ""
+      if strip_percent
+        detail = detail.sub(/\s+\(\d+%\)/, "")
+        detail = "" if detail.match?(/\A\d+%\z/)
+      end
+      detail = detail.gsub(/ \[batch (\d+)\]/) do
+        " #{color(:dim, '·')} batch #{format_count(Regexp.last_match(1))}"
+      end
+      detail = format_counts(detail)
+      detail = detail.gsub(", ", " #{color(:dim, '·')} ")
+      [label, detail]
+    end
+
+    def terminal_text(message)
+      label, detail = terminal_parts(message)
+      detail.empty? ? format_counts(label) : "#{label}: #{detail}"
+    end
+
+    def pretty_label(label)
+      case label
+      when "Building temporary relationship lookup index"
+        "Building lookup index"
+      when "Temporary relationship lookup index ready."
+        "Lookup index ready"
+      when "Cleaning temporary load metadata"
+        "Cleaning load metadata"
+      when "Temporary load metadata removed."
+        "Load metadata removed"
+      else
+        label
+      end
+    end
+
+    def extract_percent(message)
+      if (match = message.match(/\((\d+)%\)/)) || (match = message.match(/:\s*(\d+)%\z/))
+        return match[1].to_i
+      end
+
+      match = message.match(/:\s*(\d+)\/(\d+) nodes\z/)
+      return nil unless match
+
+      total = match[2].to_i
+      total.zero? ? 100 : match[1].to_i * 100 / total
+    end
+
+    def format_counts(text)
+      text.gsub(/\b\d{4,}\b/) { |number| format_count(number) }
+    end
+
+    def format_count(number)
+      number.to_s.reverse.scan(/.{1,3}/).join(",").reverse
+    end
+
+    def render_line(line)
+      @io.print "\r\e[2K#{line}"
+      @io.flush
+      @line_open = true
+    end
+
+    def clear_line
+      return unless @pretty && @line_open
+
+      @io.print "\r\e[2K"
+      @io.flush
+      @line_open = false
+    end
+
+    def color(style, text)
+      return text unless @color
+
+      "#{ANSI.fetch(style)}#{text}#{ANSI[:reset]}"
+    end
+  end
+  private_constant :ProgressReporter
   MIN_INTEGER = -(2**63)
   MAX_INTEGER = (2**63) - 1
   TRANSACTION_CONTEXT_KEY = :neo4j_bolt_transaction_contexts
@@ -319,95 +514,201 @@ module Neo4jBolt
     nil
   end
 
-  def dump_database(io)
-    dump_ids = {}
-    next_dump_id = 0
-    identity_function = dump_identity_function
+  def dump_database(io, progress_io: nil)
+    progress = ProgressReporter.new(progress_io)
+    dumped_nodes = 0
+    dumped_relationships = 0
 
-    neo4j_query("MATCH (n) RETURN n ORDER BY #{identity_function}(n)") do |row|
-      node = row["n"]
-      dump_ids[node.element_id || node.id.to_s] = next_dump_id
-      io.puts "n #{JSON.generate(id: next_dump_id, labels: node.labels, properties: node)}"
-      next_dump_id += 1
-    end
+    transaction do
+      identity_function = dump_identity_function
+      total_nodes = neo4j_query_expect_one("MATCH (n) RETURN count(n) AS count")["count"]
+      total_relationships = neo4j_query_expect_one("MATCH ()-[r]->() RETURN count(r) AS count")["count"]
+      dump_ids = total_relationships.positive? ? {} : nil
+
+      progress.update(
+        progress_message("Dumping", dumped_nodes, dumped_relationships, total_nodes, total_relationships),
+        force: true
+      )
+
+      neo4j_query(
+        "MATCH (n) " \
+        "RETURN #{identity_function}(n) AS identity, labels(n) AS labels, properties(n) AS properties " \
+        "ORDER BY #{identity_function}(n)"
+      ) do |row|
+        dump_ids[row["identity"].to_s] = dumped_nodes if dump_ids
+        io.puts "n #{JSON.generate(id: dumped_nodes, labels: row["labels"], properties: row["properties"])}"
+        dumped_nodes += 1
+        if (dumped_nodes % DUMP_PROGRESS_STEP).zero?
+          progress.update(progress_message(
+            "Dumping", dumped_nodes, dumped_relationships, total_nodes, total_relationships
+          ))
+        end
+      end
 
-    neo4j_query("MATCH ()-[r]->() RETURN r ORDER BY #{identity_function}(r)") do |row|
-      relationship = row["r"]
-      from_identity = relationship.start_node_element_id || relationship.start_node_id.to_s
-      to_identity = relationship.end_node_element_id || relationship.end_node_id.to_s
-      io.puts "r #{JSON.generate(
-        from: dump_ids.fetch(from_identity),
-        to: dump_ids.fetch(to_identity),
-        type: relationship.type,
-        properties: relationship
-      )}"
+      if total_relationships.positive?
+        neo4j_query(
+          "MATCH (from)-[r]->(to) " \
+          "RETURN #{identity_function}(from) AS from_identity, " \
+          "#{identity_function}(to) AS to_identity, type(r) AS type, properties(r) AS properties " \
+          "ORDER BY #{identity_function}(r)"
+        ) do |row|
+          io.puts "r #{JSON.generate(
+            from: dump_ids.fetch(row["from_identity"].to_s),
+            to: dump_ids.fetch(row["to_identity"].to_s),
+            type: row["type"],
+            properties: row["properties"]
+          )}"
+          dumped_relationships += 1
+          if (dumped_relationships % DUMP_PROGRESS_STEP).zero?
+            progress.update(progress_message(
+              "Dumping", dumped_nodes, dumped_relationships, total_nodes, total_relationships
+            ))
+          end
+        end
+      end
     end
+
+    progress.finish("Dumped: #{dumped_nodes} nodes, #{dumped_relationships} relationships")
     nil
+  ensure
+    progress&.close
   end
 
-  def load_database_dump(io, force_append: false)
+  def load_database_dump(io, force_append: false, progress_io: nil,
+                         initial_batch_size: LOAD_INITIAL_BATCH_SIZE)
+    raise Error, "load_database_dump cannot run inside a transaction" if transaction_context
+
+    initial_batch_size = Integer(initial_batch_size)
+    raise ArgumentError, "initial_batch_size must be positive" unless initial_batch_size.positive?
+
     unless force_append
       count = neo4j_query_expect_one("MATCH (n) RETURN count(n) AS count")["count"]
       raise Error, "There are nodes in this database, exiting now." unless count.zero?
     end
 
+    progress = ProgressReporter.new(progress_io)
     node_batches = Hash.new { |hash, key| hash[key] = [] }
     relationship_batches = Hash.new { |hash, key| hash[key] = [] }
-    parse_dump(io, node_batches, relationship_batches)
+    total_nodes, total_relationships = parse_dump(io, node_batches, relationship_batches, progress)
 
-    temporary_id_property = "__neo4j_bolt_load_#{SecureRandom.hex(12)}"
+    temporary_token = SecureRandom.hex(12)
+    temporary_label = "__neo4j_bolt_load_#{temporary_token}"
+    temporary_id_property = "__neo4j_bolt_load_id_#{temporary_token}"
+    temporary_index = "neo4j_bolt_load_#{temporary_token}"
+    quoted_temporary_label = quote_identifier(temporary_label)
     quoted_temporary_id = quote_identifier(temporary_id_property)
+    quoted_temporary_index = quote_identifier(temporary_index)
+    needs_relationship_lookup = total_relationships.positive?
     loaded_nodes = 0
     loaded_relationships = 0
+    index_created = false
 
+    progress.update(
+      progress_message("Loading", loaded_nodes, loaded_relationships, total_nodes, total_relationships,
+                       batch_size: initial_batch_size),
+      force: true
+    )
+
+    original_error = nil
+    cleanup_error = nil
     begin
+      node_batch_size = initial_batch_size
       node_batches.each_value do |nodes|
-        nodes.each_slice(256) do |slice|
+        node_batch_size = adaptive_each_slice(nodes, node_batch_size, progress, "node") do |slice, current_batch_size|
           labels = slice.first.fetch("labels")
           label_clause = labels.map { |label| ":#{quote_identifier(label)}" }.join
+          label_clause += ":#{quoted_temporary_label}" if needs_relationship_lookup
+          temporary_assignment = if needs_relationship_lookup
+                                   "SET n.#{quoted_temporary_id} = item.id\n"
+                                 else
+                                   ""
+                                 end
           count = neo4j_query_expect_one(<<~CYPHER, nodes: slice)["count_n"]
             UNWIND $nodes AS item
             CREATE (n#{label_clause})
             SET n = item.properties
-            SET n.#{quoted_temporary_id} = item.id
-            RETURN count(n) AS count_n
+            #{temporary_assignment}RETURN count(n) AS count_n
           CYPHER
           raise Error, "Expected #{slice.size} nodes, got #{count}." unless count == slice.size
 
           loaded_nodes += slice.size
-          report_load_progress(loaded_nodes, loaded_relationships)
+          progress.update(progress_message(
+            "Loading", loaded_nodes, loaded_relationships, total_nodes, total_relationships,
+            batch_size: current_batch_size
+          ))
         end
       end
 
-      relationship_batches.each do |type, relationships|
-        relationships.each_slice(256) do |slice|
-          count = neo4j_query_expect_one(<<~CYPHER, relationships: slice)["count_r"]
-            UNWIND $relationships AS item
-            MATCH (from) WHERE from.#{quoted_temporary_id} = item.from
-            MATCH (to) WHERE to.#{quoted_temporary_id} = item.to
-            CREATE (from)-[r:#{quote_identifier(type)}]->(to)
-            SET r = item.properties
-            RETURN count(r) AS count_r
-          CYPHER
-          raise Error, "Expected #{slice.size} relationships, got #{count}." unless count == slice.size
+      if needs_relationship_lookup
+        progress.note("Building temporary relationship lookup index...")
+        neo4j_query(
+          "CREATE INDEX #{quoted_temporary_index} " \
+          "FOR (n:#{quoted_temporary_label}) ON (n.#{quoted_temporary_id})"
+        )
+        index_created = true
+        wait_for_load_index(temporary_index, progress)
+        progress.update(
+          progress_message("Loading", loaded_nodes, loaded_relationships, total_nodes, total_relationships,
+                           batch_size: initial_batch_size),
+          force: true
+        )
 
-          loaded_relationships += slice.size
-          report_load_progress(loaded_nodes, loaded_relationships)
+        relationship_batch_size = initial_batch_size
+        relationship_batches.each do |type, relationships|
+          relationship_batch_size = adaptive_each_slice(
+            relationships, relationship_batch_size, progress, "relationship"
+          ) do |slice, current_batch_size|
+            count = neo4j_query_expect_one(<<~CYPHER, relationships: slice)["count_r"]
+              UNWIND $relationships AS item
+              MATCH (from:#{quoted_temporary_label} {#{quoted_temporary_id}: item.from})
+              MATCH (to:#{quoted_temporary_label} {#{quoted_temporary_id}: item.to})
+              CREATE (from)-[r:#{quote_identifier(type)}]->(to)
+              SET r = item.properties
+              RETURN count(r) AS count_r
+            CYPHER
+            raise Error, "Expected #{slice.size} relationships, got #{count}." unless count == slice.size
+
+            loaded_relationships += slice.size
+            progress.update(progress_message(
+              "Loading", loaded_nodes, loaded_relationships, total_nodes, total_relationships,
+              batch_size: current_batch_size
+            ))
+          end
         end
       end
+    rescue Exception => error # rubocop:disable Lint/RescueException -- preserve original failure through cleanup
+      original_error = error
+      raise
     ensure
-      original_error = $!
-      begin
-        neo4j_query("MATCH (n) WHERE n.#{quoted_temporary_id} IS NOT NULL REMOVE n.#{quoted_temporary_id}") if loaded_nodes.positive?
-      rescue Error
-        raise if original_error.nil?
+      if index_created
+        begin
+          neo4j_query("DROP INDEX #{quoted_temporary_index} IF EXISTS")
+        rescue Error => error
+          cleanup_error ||= error
+        end
+      end
+
+      if needs_relationship_lookup
+        begin
+          progress&.note("Cleaning temporary load metadata...")
+          cleanup_load_metadata(
+            quoted_temporary_label, quoted_temporary_id, initial_batch_size, total_nodes, progress
+          )
+        rescue Error => error
+          cleanup_error ||= error
+        end
       end
+
+      raise cleanup_error if original_error.nil? && cleanup_error
     end
 
-    warn if loaded_nodes.positive? || loaded_relationships.positive?
+    progress.finish("Loaded: #{loaded_nodes} nodes, #{loaded_relationships} relationships")
     nil
+  ensure
+    progress&.close
   end
 
+
   private
 
   def run_and_convert(runner, query, data)
@@ -535,7 +836,11 @@ module Neo4jBolt
     version.to_s.split(".").first.to_i >= 5 ? "elementId" : "id"
   end
 
-  def parse_dump(io, node_batches, relationship_batches)
+  def parse_dump(io, node_batches, relationship_batches, progress = nil)
+    node_count = 0
+    relationship_count = 0
+    progress&.update("Reading dump: 0 nodes, 0 relationships", force: true)
+
     io.each_line.with_index(1) do |line, line_number|
       line = line.strip
       next if line.empty?
@@ -547,16 +852,108 @@ module Neo4jBolt
       if kind == "n"
         labels = value.fetch("labels")
         node_batches[labels.sort] << value
+        node_count += 1
       else
         relationship_batches[value.fetch("type")] << value
+        relationship_count += 1
+      end
+      if ((node_count + relationship_count) % LOAD_PROGRESS_STEP).zero?
+        progress&.update("Reading dump: #{node_count} nodes, #{relationship_count} relationships")
       end
     rescue JSON::ParserError, KeyError => error
       raise Error, "Invalid dump entry on line #{line_number}: #{error.message}"
     end
+
+    progress&.finish("Read dump: #{node_count} nodes, #{relationship_count} relationships")
+    [node_count, relationship_count]
+  end
+
+  def adaptive_each_slice(items, batch_size, progress, kind)
+    offset = 0
+    while offset < items.size
+      slice = items.slice(offset, [batch_size, items.size - offset].min)
+      begin
+        yield slice, batch_size
+        offset += slice.size
+      rescue Error => error
+        raise unless load_batch_memory_error?(error) && slice.size > 1
+
+        reduced_batch_size = [slice.size / 2, 1].max
+        batch_size = [batch_size, reduced_batch_size].min
+        progress&.note(
+          "Neo4j rejected a #{kind} batch of #{slice.size} for transaction memory; " \
+          "retrying with batches of #{batch_size}.",
+          level: :warning
+        )
+      end
+    end
+    batch_size
+  end
+
+  def load_batch_memory_error?(error)
+    upstream = error.cause
+    return false unless upstream
+
+    LOAD_MEMORY_ERROR_CODES.include?(upstream.respond_to?(:code) ? upstream.code : nil) ||
+      LOAD_MEMORY_GQL_STATUSES.include?(upstream.respond_to?(:gql_status) ? upstream.gql_status : nil)
+  end
+
+  def wait_for_load_index(index_name, progress)
+    loop do
+      row = neo4j_query_expect_one(<<~CYPHER, name: index_name)
+        SHOW INDEXES YIELD name, state, populationPercent
+        WHERE name = $name
+        RETURN state, populationPercent
+      CYPHER
+      state = row["state"]
+      population = row["populationPercent"] || 0
+      if state == "ONLINE"
+        progress&.finish("Temporary relationship lookup index ready.")
+        return
+      end
+      raise Error, "Temporary load index #{index_name} entered state #{state}." if state == "FAILED"
+
+      progress&.update("Building temporary relationship lookup index: #{population.to_i}%")
+      sleep 0.25
+    end
+  end
+
+  def cleanup_load_metadata(quoted_label, quoted_id, batch_size, total_nodes, progress)
+    cleaned_nodes = 0
+    loop do
+      begin
+        count = neo4j_query_expect_one(<<~CYPHER, limit: batch_size)["count_n"]
+          MATCH (n:#{quoted_label})
+          WITH n LIMIT $limit
+          REMOVE n.#{quoted_id}
+          REMOVE n:#{quoted_label}
+          RETURN count(n) AS count_n
+        CYPHER
+        break if count.zero?
+
+        cleaned_nodes += count
+        progress&.update("Cleaning temporary load metadata: #{cleaned_nodes}/#{total_nodes} nodes")
+      rescue Error => error
+        raise unless load_batch_memory_error?(error) && batch_size > 1
+
+        batch_size = [batch_size / 2, 1].max
+        progress&.note(
+          "Neo4j rejected a cleanup batch for transaction memory; retrying with batches of #{batch_size}.",
+          level: :warning
+        )
+      end
+    end
+    progress&.finish("Temporary load metadata removed.")
   end
 
-  def report_load_progress(nodes, relationships)
-    warn "Loaded #{nodes} nodes, #{relationships} relationships..." if Neo4jBolt.bolt_verbosity.to_i.positive?
+  def progress_message(action, nodes, relationships, total_nodes, total_relationships, batch_size: nil)
+    total = total_nodes + total_relationships
+    completed = nodes + relationships
+    percent = total.zero? ? 100 : (completed * 100 / total)
+    message = "#{action}: #{nodes}/#{total_nodes} nodes, " \
+              "#{relationships}/#{total_relationships} relationships (#{percent}%)"
+    message += " [batch #{batch_size}]" if batch_size
+    message
   end
 
   def log_query(query, data)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: neo4j_bolt
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.4.2
 platform: ruby
 authors:
 - Michael Specht