neo4j-core 8.1.4 → 9.0.0.alpha.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 62857b8d2f1a86d5c17adefa6b5866eb5efaf40e
-  data.tar.gz: a669a4e956ea744d392104ae98e80fb673f4bb24
+SHA256:
+  metadata.gz: 95c3a0a71e52de4f4ef573cc36ff3d206a1fcd3ba443a84890e3c42c6ca6cc39
+  data.tar.gz: '05885ecf4289f9fae5d67234f82324c24289e4ab0a1fff8ffc6569810e71a496'
 SHA512:
-  metadata.gz: d2b1b59513aaf3a619f0ea1b1fe526bbf57ac9105f88e8039be13c17c367972c3286aba0c9e0d0827ce80fb44eed5bbac89aaa8b1aba165438614958c42a81d5
-  data.tar.gz: b2fc295d422907d44afa707975ea6ba6bb5705cfa76984bd135532202c04f4e2524ea90e404f5b0582cd6979f0237bc47633b4dd8e042b31e820179acb2b7279
+  metadata.gz: ab2abc2fa87addc1ef0b2ebb0e3e67886d773d0542bf3a2850f17adc39c27e9faae92a0f528b8e8785de28bced39cbd19d5a264ee27f1584f1f34ccd21383049
+  data.tar.gz: 686135288df838098db8b40ede27157bc02d335eef6b041b18ee4c7f24a2f0f28bdc98f1d5ada1a15e864e9e24d7caf1acb8aae2003b44696041b9c9ad44d8c9

data/README.md

@@ -1,4 +1,10 @@
-# Neo4j-core [![Code Climate](https://codeclimate.com/github/neo4jrb/neo4j-core.svg)](https://codeclimate.com/github/neo4jrb/neo4j-core) [![Build Status](https://travis-ci.org/neo4jrb/neo4j-core.svg)](https://travis-ci.org/neo4jrb/neo4j-core) [![Coverage Status](https://coveralls.io/repos/neo4jrb/neo4j-core/badge.svg?branch=master)](https://coveralls.io/r/neo4jrb/neo4j-core?branch=master) [![PullReview stats](https://www.pullreview.com/github/neo4jrb/neo4j-core/badges/master.svg?)](https://www.pullreview.com/github/neo4jrb/neo4j-core/reviews/master)
+# Neo4j-core
+
+[![Actively Maintained](https://img.shields.io/badge/Maintenance%20Level-Actively%20Maintained-green.svg)](https://gist.github.com/cheerfulstoic/d107229326a01ff0f333a1d3476e068d)
+[![Code Climate](https://codeclimate.com/github/neo4jrb/neo4j-core.svg)](https://codeclimate.com/github/neo4jrb/neo4j-core)
+[![Build Status](https://travis-ci.org/neo4jrb/neo4j-core.svg)](https://travis-ci.org/neo4jrb/neo4j-core)
+[![Coverage Status](https://coveralls.io/repos/neo4jrb/neo4j-core/badge.svg?branch=master)](https://coveralls.io/r/neo4jrb/neo4j-core?branch=master)
+[![PullReview stats](https://www.pullreview.com/github/neo4jrb/neo4j-core/badges/master.svg?)](https://www.pullreview.com/github/neo4jrb/neo4j-core/reviews/master)
 
 A simple Ruby wrapper around the Neo4j graph database that works with the server and embedded Neo4j API. This gem can be used both from JRuby and normal MRI.
 It can be used standalone without the neo4j gem.
@@ -9,15 +15,20 @@ It can be used standalone without the neo4j gem.
 
 To make a basic connection to Neo4j to execute Cypher queries, first choose an adaptor.  Adaptors for HTTP, Bolt, and Embedded mode (jRuby only) are available.  You can create an adaptor like:
 
-    http_adaptor = Neo4j::Core::CypherSession::Adaptors::HTTP.new('http://neo4j:pass@localhost:7474')
+    require 'neo4j/core/cypher_session/adaptors/http'
+    http_adaptor = Neo4j::Core::CypherSession::Adaptors::HTTP.new('http://neo4j:pass@localhost:7474', options)
 
     # or
 
-    bolt_adaptor = Neo4j::Core::CypherSession::Adaptors::Bolt.new('bolt://neo4j:pass@localhost:7687', timeout: 10)
+    require 'neo4j/core/cypher_session/adaptors/bolt'
+    bolt_adaptor = Neo4j::Core::CypherSession::Adaptors::Bolt.new('bolt://neo4j:pass@localhost:7687', options)
 
     # or
 
-    neo4j_adaptor = Neo4j::Core::CypherSession::Adaptors::Embedded.new('/file/path/to/graph.db')
+    require 'neo4j/core/cypher_session/adaptors/embedded'
+    neo4j_adaptor = Neo4j::Core::CypherSession::Adaptors::Embedded.new('/file/path/to/graph.db', options)
+
+The options are specific to each adaptor.  See below for details.
 
 Once you have an adaptor you can create a session like so:
 
@@ -55,15 +66,67 @@ When doing batched queries, there is also a shortcut for getting a new `Neo4j::C
 
     results[0] # result
 
+### Adaptor Options
+
+As mentioned above, each of the adaptors take different sets of options.  They are enumerated below:
+
+#### Shared options
+
+All adaptors take `wrap_level` as an option.  This can be used to control how nodes, relationships, and path data is returned:
+
+ * `wrap_level: :none` will return Ruby hashes
+ * `wrap_level: :core_entity` will return objects from the `neo4j-core` gem (`Neo4j::Core::Node`, `Neo4j::Core::Relationship`, and `Neo4j::Core::Path`
+ * `wrap_level: :prop` allows you to define Ruby Procs to do your own wrapping.  This is how the `neo4j` gem provides `ActiveNode` and `ActiveRel` objects (see the [`node_wrapper.rb`](https://github.com/neo4jrb/neo4j/blob/master/lib/neo4j/active_node/node_wrapper.rb) and [`rel_wrapper.rb`](https://github.com/neo4jrb/neo4j/blob/master/lib/neo4j/active_rel/rel_wrapper.rb) files for examples on how this works
+
+All adaptors will also take either a `logger` option with a Ruby logger to define where it will log to.
+
+All adaptors will also take the `skip_instrumentation` option to skip logging of queries.
+
+All adaptors will also take the `verbose_query_logs` option which can be `true` or `false` (`false` being the default).  This will change the logging to output the source line of code which caused a query to be executed (note that the `skip_instrumentation` should not be set for logging to be produced).
+
+#### Bolt
+
+The Bolt adaptor takes `connect_timeout`, `read_timeout`, and `write_timeout` options which define appropriate timeouts.  The `connect_timeout` is 10 seconds and the `read_timeout` and `write_timeout` are -1 (no timeout).  This is to cause the underlying `net_tcp_client` gem to operate in blocking mode (as opposed to non-blocking mode).  When using non-blocking mode problems were found and since the official Neo4j drivers in other languages use blocking mode, this is what this gem uses by default.  This issue could potentially be a bug in the handling of the `EAGAIN` signal, but it was not investigated further. Set the read/write timeouts at your own risk.
+
+The Bolt adaptor also takes an `ssl` option which also corresponds to `net_tcp_client`'s `ssl` option (which, in turn, corresponds to Ruby's `OpenSSL::SSL::SSLContext`).  By default SSL is used.  For most cloud providers that use public certificate authorities this open generally won't be needed.  If you've setup Neo4j yourself you will need to provide the certificate like so:
+
+```ruby
+cert_store = OpenSSL::X509::Store.new
+cert_store.add_file('/the/path/to/your/neo4j.cert')
+ssl: {cert_store: cert_store}}
+bolt_adaptor = Neo4j::Core::CypherSession::Adaptors::Bolt.new('bolt://neo4j:pass@localhost:7687', ssl: {cert_store: cert_store})
+```
+
+You can also turn SSL off by simply specifying `ssl: false`
+
+#### HTTP
+
+Since the HTTP adaptor uses the `faraday` gem under the covers, it takes a `faraday_configurator` option.  This allows you to pass in a `Proc` which works just like a Faraday setup block:
+
+```ruby
+  faraday_configurator: proc do |faraday|
+    # The default configurator uses typhoeus so if you override the configurator you must specify this
+    faraday.adapter :typhoeus
+    # Optionally you can instead specify another adaptor
+    # faraday.use Faraday::Adapter::NetHttpPersistent
+
+    # If you need to set options which would normally be the second argument of `Faraday.new`, you can do the following:
+    faraday.options[:open_timeout] = 5
+    faraday.options[:timeout] = 65
+    # faraday.options[:ssl] = { verify: true }
+  end
+```
+
+#### Embedded
+
+The Embedded adaptor takes `properties_file` and `properties_map` options which are passed to `loadPropertiesFromFile` and `setConfig` on the `GraphDatabaseBuilder` class from the Neo4j Java API.
+
 ## Documentation
 
-### 3.0+ Documentation:
+Our documentation on ReadTheDocs covers both the `neo4j` and `neo4j-core` gems:
 
  * http://neo4jrb.readthedocs.org/en/stable/
 
-### 2.x Documentation
-
-https://github.com/neo4jrb/neo4j-core/tree/v2.x
 
 ## Support
 

data/lib/neo4j-core.rb

@@ -1,49 +1,3 @@
-require 'ext/kernel'
-
-require 'ostruct'
-require 'forwardable'
-require 'fileutils'
-
-require 'neo4j-core/version'
-require 'neo4j/property_validator'
-require 'neo4j/property_container'
-require 'neo4j/entity_marshal'
-require 'neo4j-core/active_entity'
-require 'neo4j-core/helpers'
-require 'neo4j-core/query_find_in_batches'
-require 'neo4j-core/query'
-
-require 'neo4j/entity_equality'
-require 'neo4j/node'
-require 'neo4j/label'
-require 'neo4j/session'
-require 'neo4j/ansi'
-
-require 'neo4j/relationship'
-require 'neo4j/transaction'
-
-require 'rake'
-load 'neo4j/core/rake_tasks_deprecation.rake'
-
-require 'logger'
-
-module Neo4j
-  module Core
-    ORIGINAL_FORMATTER = ::Logger::Formatter.new
-
-    def self.logger(stream = STDOUT)
-      @logger ||= Logger.new(stream).tap do |logger|
-        logger.formatter = method(:formatter)
-      end
-    end
-
-    def self.formatter(severity, datetime, progname, msg)
-      output = ''
-      if Thread.current != Thread.main
-        output += "#{ANSI::YELLOW}Thread: #{Thread.current.object_id}: #{ANSI::CLEAR}"
-      end
-      output += msg
-      ORIGINAL_FORMATTER.call(severity, datetime, progname, output)
-    end
-  end
-end
+# DO NOT ADD ANYTHING HERE
+# lib/neo4j/core.rb should be the base for requiring essential files
+require 'neo4j/core'

data/lib/neo4j/core.rb

@@ -0,0 +1,4 @@
+require 'neo4j/core/config'
+
+require 'neo4j/transaction'
+require 'neo4j/core/query'

data/lib/neo4j/core/config.rb

@@ -0,0 +1,13 @@
+module Neo4j
+  module Core
+    module Config
+      def self.wrapping_level(level = nil)
+        if level.nil?
+          @wrapping_level || :core_entity
+        else
+          @wrapping_level = level
+        end
+      end
+    end
+  end
+end

data/lib/neo4j/core/cypher_session/adaptors.rb

@@ -1,7 +1,9 @@
 require 'neo4j/core/cypher_session'
 require 'neo4j/core/instrumentable'
 require 'neo4j/core/label'
-require 'neo4j-core/version'
+require 'neo4j/core/version'
+require 'neo4j/core/logging'
+require 'neo4j/ansi'
 
 module Neo4j
   module Core
@@ -22,7 +24,6 @@ ERROR
           super(msg)
         end
 
-
         def self.new_from(code, message, stack_trace = nil)
           error_class_from(code).new(code, message, stack_trace)
         end
@@ -67,6 +68,7 @@ ERROR
           end
 
           attr_accessor :wrap_level
+          attr_reader :options
 
           Query = Struct.new(:cypher, :parameters, :pretty_cypher, :context)
 
@@ -162,31 +164,29 @@ ERROR
           end
 
           def setup_queries!(queries, transaction, options = {})
-            fail 'Query attempted without a connection' if !connected?
-            fail "Invalid transaction object: #{transaction.inspect}" if !transaction.is_a?(self.class.transaction_class)
+            validate_connection!(transaction)
 
-            # context option not yet implemented
-            self.class.instrument_queries(queries) unless options[:skip_instrumentation]
+            return if options[:skip_instrumentation]
+            queries.each do |query|
+              self.class.instrument_query(query, self) {}
+            end
           end
 
           EMPTY = ''
           NEWLINE_W_SPACES = "\n  "
 
-          instrument(:query, 'neo4j.core.cypher_query', %w[query]) do |_, _start, _finish, _id, payload|
+          instrument(:query, 'neo4j.core.cypher_query', %w[query adaptor]) do |_, _start, _finish, _id, payload|
             query = payload[:query]
             params_string = (query.parameters && !query.parameters.empty? ? "| #{query.parameters.inspect}" : EMPTY)
             cypher = query.pretty_cypher ? (NEWLINE_W_SPACES if query.pretty_cypher.include?("\n")).to_s + query.pretty_cypher.gsub(/\n/, NEWLINE_W_SPACES) : query.cypher
 
-            " #{ANSI::CYAN}#{query.context || 'CYPHER'}#{ANSI::CLEAR} #{cypher} #{params_string}"
+            source_line, line_number = Logging.first_external_path_and_line(caller_locations)
+
+            " #{ANSI::CYAN}#{query.context || 'CYPHER'}#{ANSI::CLEAR} #{cypher} #{params_string}" +
+              ("\n   ↳ #{source_line}:#{line_number}" if payload[:adaptor].options[:verbose_query_logs] && source_line).to_s
           end
 
           class << self
-            def instrument_queries(queries)
-              queries.each do |query|
-                instrument_query(query) {}
-              end
-            end
-
             def transaction_class
               fail '.transaction_class method not implemented on adaptor!'
             end
@@ -202,7 +202,7 @@ ERROR
             end
           end
 
-          def validate_query_set!(transaction, _queries, _options = {})
+          def validate_connection!(transaction)
             fail 'Query attempted without a connection' if !connected?
             fail "Invalid transaction object: #{transaction}" if !transaction.is_a?(self.class.transaction_class)
           end

data/lib/neo4j/core/cypher_session/adaptors/bolt.rb

@@ -3,10 +3,9 @@ require 'neo4j/core/cypher_session/adaptors/has_uri'
 require 'neo4j/core/cypher_session/adaptors/bolt/pack_stream'
 require 'neo4j/core/cypher_session/adaptors/bolt/chunk_writer_io'
 require 'neo4j/core/cypher_session/responses/bolt'
-require 'io/wait'
-require 'socket'
+require 'net/tcp_client'
 
-# TODO: Work with `Query` objects
+# TODO: Work with `Query` objects?
 module Neo4j
   module Core
     class CypherSession
@@ -24,6 +23,10 @@ module Neo4j
           def initialize(url, options = {})
             self.url = url
             @options = options
+            @net_tcp_client_options = {read_timeout: options.fetch(:read_timeout, -1),
+                                       write_timeout: options.fetch(:write_timeout, -1),
+                                       connect_timeout: options.fetch(:connect_timeout, 10),
+                                       ssl: options.fetch(:ssl, {})}
 
             open_socket
           end
@@ -43,13 +46,7 @@ module Neo4j
           def query_set(transaction, queries, options = {})
             setup_queries!(queries, transaction, skip_instrumentation: options[:skip_instrumentation])
 
-            if @socket.ready?
-              debug_remaining_buffer
-              fail "Making query, but expected there to be no buffer remaining!\n"\
-                   "Queries: #{queries.map(&:cypher)}"
-            end
-
-            self.class.instrument_request do
+            self.class.instrument_request(self) do
               send_query_jobs(queries)
 
               build_response(queries, options[:wrap_level] || @options[:wrap_level])
@@ -64,7 +61,7 @@ module Neo4j
           end
 
           def connected?
-            !!@socket
+            !!@tcp_client && !@tcp_client.closed?
           end
 
           def indexes(session)
@@ -90,10 +87,16 @@ module Neo4j
             Neo4j::Core::CypherSession::Transactions::Bolt
           end
 
-          instrument(:request, 'neo4j.core.bolt.request', %w[url body]) do |_, start, finish, _id, payload|
+          instrument(:request, 'neo4j.core.bolt.request', %w[adaptor body]) do |_, start, finish, _id, payload|
             ms = (finish - start) * 1000
+            adaptor = payload[:adaptor]
 
-            " #{ANSI::BLUE}BOLT REQUEST:#{ANSI::CLEAR} #{ANSI::YELLOW}#{ms.round}ms#{ANSI::CLEAR} #{payload[:url]}"
+            type = adaptor.ssl? ? '+TLS' : ' UNSECURE'
+            " #{ANSI::BLUE}BOLT#{type}:#{ANSI::CLEAR} #{ANSI::YELLOW}#{ms.round}ms#{ANSI::CLEAR} #{adaptor.url_without_password}"
+          end
+
+          def ssl?
+            @tcp_client.socket.is_a?(OpenSSL::SSL::SSLSocket)
           end
 
           private
@@ -117,17 +120,6 @@ module Neo4j
             fail CypherError.new_from(error_data['code'], error_data['message'])
           end
 
-          def debug_remaining_buffer
-            logger.debug 'Remaining buffer:'
-
-            i = 0
-            while @socket.ready?
-              i += 1
-              logger.debug "Message set #{i}:"
-              flush_messages
-            end
-          end
-
           def send_query_jobs(queries)
             send_job do |job|
               queries.each do |query|
@@ -141,8 +133,12 @@ module Neo4j
             Job.new(self)
           end
 
+          def secure_connection?
+            @is_secure_socket ||= @options.key?(:ssl)
+          end
+
           def open_socket
-            @socket = TCPSocket.open(host, port)
+            @tcp_client = Net::TCPClient.new(@net_tcp_client_options.merge(buffered: false, server: "#{host}:#{port}"))
           rescue Errno::ECONNREFUSED => e
             raise Neo4j::Core::CypherSession::ConnectionFailedError, e.message
           end
@@ -156,13 +152,12 @@ module Neo4j
             agreed_version = recvmsg(4).unpack('l>*')[0]
 
             if agreed_version.zero?
-              @socket.shutdown(Socket::SHUT_RDWR)
-              @socket.close
+              @tcp_client.close
 
               fail "Couldn't agree on a version (Sent versions #{SUPPORTED_VERSIONS.inspect})"
             end
 
-            logger.debug "Agreed to version: #{agreed_version}"
+            logger.debug { "Agreed to version: #{agreed_version}" }
           end
 
           def init
@@ -181,42 +176,42 @@ module Neo4j
             end
           end
 
+          # Don't need to calculate these every time.  Cache in memory
+          BYTE_STRINGS = (0..255).map { |byte| byte.to_s(16).rjust(2, '0') }
+
           STREAM_INSPECTOR = lambda do |stream|
-            stream.bytes.map { |byte| byte.to_s(16).rjust(2, '0') }.join(':')
+            stream.bytes.map { |byte| BYTE_STRINGS[byte] }.join(':')
           end
 
           def sendmsg(message)
             log_message :C, message
-
-            @socket.send(message, 0)
+            @tcp_client.write(message)
           end
 
-          def recvmsg(size, timeout = timeout_option)
-            Timeout.timeout(timeout) do
-              @socket.recv(size).tap do |result|
-                log_message :S, result
-              end
+          def recvmsg(size)
+            @tcp_client.read(size) do |result|
+              log_message :S, result
             end
-          rescue Timeout::Error
-            raise "Timed out waiting for #{size} bytes from Neo4j (after #{timeout} seconds)"
           end
 
           def flush_messages
             if structures = flush_response
               structures.map do |structure|
                 Message.new(structure.signature, *structure.list).tap do |message|
-                  log_message :S, message.type, message.args.join(' ')
+                  log_message :S, message.type, message.args.join(' ') if logger.debug?
                 end
               end
             end
           end
 
           def log_message(side, *args)
-            if args.size == 1
-              logger.debug "#{side}: #{STREAM_INSPECTOR.call(args[0])}"
-            else
-              type, message = args
-              logger.debug "#{side}: #{ANSI::CYAN}#{type.to_s.upcase}#{ANSI::CLEAR} #{message}"
+            logger.debug do
+              if args.size == 1
+                "#{side}: #{STREAM_INSPECTOR.call(args[0])}"
+              else
+                type, message = args
+                "#{side}: #{ANSI::CYAN}#{type.to_s.upcase}#{ANSI::CLEAR} #{message}"
+              end
             end
           end
 
@@ -234,10 +229,6 @@ module Neo4j
             [].tap { |r| while arg = unpacker.unpack_value!; r << arg; end }
           end
 
-          def timeout_option
-            @options.fetch(:timeout) { 10 }
-          end
-
           # Represents messages sent to or received from the server
           class Message
             TYPE_CODES = {