neo4j_bolt 0.4.4 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +63 -18
- data/bin/neo4j_bolt +22 -8
- data/lib/neo4j_bolt/version.rb +1 -1
- data/lib/neo4j_bolt.rb +223 -11
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: f9f38fc0e16014ae13f2f31d2ba77f119185249821da7d4f7858fa0ef3f9010f
|
|
4
|
+
data.tar.gz: 5ab718f1dd9829aa1f118a6614f7d29d184d63eb20b4ce7d3e36fc92fea5f204
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: df61ebc16c46698ccab04c94e9a8abd6120ef07d899b7e52e8e9a80d13bcc4b2f8d17bddff330f6d10ff794faabd705ebfba3d3531fdad311cb8ea5de02dbc8c
|
|
7
|
+
data.tar.gz: 6522b70e44f279f182c289ae3397dafd11d5dff4e72a3437247a5f25bd66d4185e6696b2aa2c96bce976f518d63a30c98c48ea86b2823d28383c3989df24b7eb
|
data/README.md
CHANGED
|
@@ -4,14 +4,14 @@ Neo4jBolt 0.4 is a small compatibility and convenience layer for Ruby applicatio
|
|
|
4
4
|
|
|
5
5
|
Neo4jBolt no longer implements the Bolt wire protocol itself.
|
|
6
6
|
|
|
7
|
-
`0.
|
|
7
|
+
`0.5.0` requires Ruby 3.4 or newer and pins `neo4j-ruby-driver` to `6.2.1.beta.4`. Applications on older Rubies can remain on the Neo4jBolt 0.3.x line; this prerelease is intentionally not an automatic upgrade for them.
|
|
8
8
|
|
|
9
9
|
## Installation
|
|
10
10
|
|
|
11
11
|
For this prerelease, specify the version explicitly:
|
|
12
12
|
|
|
13
13
|
```ruby
|
|
14
|
-
gem "neo4j_bolt", "0.
|
|
14
|
+
gem "neo4j_bolt", "0.5.0"
|
|
15
15
|
```
|
|
16
16
|
|
|
17
17
|
Then run `bundle install`. A running Neo4j database is required.
|
|
@@ -26,17 +26,20 @@ Neo4jBolt.bolt_port = 7687
|
|
|
26
26
|
Neo4jBolt.bolt_verbosity = 0
|
|
27
27
|
```
|
|
28
28
|
|
|
29
|
-
|
|
29
|
+
The connection endpoint, authentication, and database selection can use the standard Neo4j client environment variables:
|
|
30
30
|
|
|
31
31
|
```bash
|
|
32
|
+
export NEO4J_URI=neo4j://localhost:7687
|
|
32
33
|
export NEO4J_USERNAME=neo4j
|
|
33
34
|
export NEO4J_PASSWORD=secret
|
|
34
35
|
export NEO4J_DATABASE=neo4j
|
|
35
36
|
```
|
|
36
37
|
|
|
37
|
-
They are read when `neo4j_bolt` is loaded. Explicit Ruby configuration overrides those defaults:
|
|
38
|
+
They are read when `neo4j_bolt` is loaded. `NEO4J_URI` accepts `neo4j://HOST[:PORT]` and `bolt://HOST[:PORT]` endpoint notation; Neo4jBolt uses the host and port while retaining its existing direct Bolt connection behavior. The default port is 7687. Explicit Ruby configuration overrides those defaults:
|
|
38
39
|
|
|
39
40
|
```ruby
|
|
41
|
+
Neo4jBolt.bolt_host = "localhost"
|
|
42
|
+
Neo4jBolt.bolt_port = 7687
|
|
40
43
|
Neo4jBolt.bolt_username = "neo4j"
|
|
41
44
|
Neo4jBolt.bolt_password = "secret"
|
|
42
45
|
Neo4jBolt.bolt_database = "movies"
|
|
@@ -160,14 +163,31 @@ Managed entries retain the `neo4j_bolt_` prefix. Setup removes obsolete entries
|
|
|
160
163
|
|
|
161
164
|
## Dump and load
|
|
162
165
|
|
|
163
|
-
The textual format
|
|
166
|
+
The textual format remains line-oriented and backwards compatible. Dumps now include supported schema entries by default:
|
|
164
167
|
|
|
165
168
|
```text
|
|
166
|
-
|
|
167
|
-
|
|
169
|
+
u {"name":"person_email","entity_type":"NODE","label_or_type":"Person","properties":["email"]}
|
|
170
|
+
k {"name":"person_id","entity_type":"NODE","label_or_type":"Person","properties":["id"]}
|
|
171
|
+
i {"name":"person_name","entity_type":"NODE","label_or_type":"Person","properties":["name"]}
|
|
172
|
+
n {"id":0,"labels":["Person"],"properties":{"id":1,"name":"Ada","email":"ada@example.test"}}
|
|
173
|
+
n {"id":1,"labels":["Person"],"properties":{"id":2,"name":"Grace","email":"grace@example.test"}}
|
|
168
174
|
r {"from":0,"to":1,"type":"KNOWS","properties":{"since":2024}}
|
|
169
175
|
```
|
|
170
176
|
|
|
177
|
+
The record prefixes are:
|
|
178
|
+
|
|
179
|
+
- `u` — property uniqueness constraint
|
|
180
|
+
- `k` — node or relationship key constraint
|
|
181
|
+
- `i` — ordinary property index
|
|
182
|
+
- `n` — node
|
|
183
|
+
- `r` — relationship
|
|
184
|
+
|
|
185
|
+
Schema records preserve the schema entry name, whether it belongs to nodes or relationships, the label or relationship type, and the indexed/constrained properties. Constraint-backing indexes are not written as separate `i` records. Ordinary Neo4j 4.4 `BTREE` indexes and modern `RANGE` indexes are treated as the same portable property-index concept, so a logical dump can move between supported Neo4j generations. Neo4j's token lookup indexes are intentionally omitted because they are database infrastructure rather than application property indexes.
|
|
186
|
+
|
|
187
|
+
Other constraint or index types are not silently dropped. A schema dump fails with a clear error instead; use data-only dumping if the application intentionally needs only nodes and relationships.
|
|
188
|
+
|
|
189
|
+
Key constraints are dumped when they exist. Loading a dump containing a `k` record requires a Neo4j edition/version that can create that key; otherwise the load fails instead of silently weakening the schema.
|
|
190
|
+
|
|
171
191
|
```ruby
|
|
172
192
|
File.open("database.dump", "w") { |io| dump_database(io, progress_io: $stderr) }
|
|
173
193
|
File.open("database.dump", "r") do |io|
|
|
@@ -175,6 +195,13 @@ File.open("database.dump", "r") do |io|
|
|
|
175
195
|
end
|
|
176
196
|
```
|
|
177
197
|
|
|
198
|
+
Schema dumping and restoration can be disabled explicitly:
|
|
199
|
+
|
|
200
|
+
```ruby
|
|
201
|
+
File.open("data.dump", "w") { |io| dump_database(io, schema: false) }
|
|
202
|
+
File.open("database.dump", "r") { |io| load_database_dump(io, schema: false) }
|
|
203
|
+
```
|
|
204
|
+
|
|
178
205
|
Passing `progress_io:` is optional for the Ruby API. The CLI always reports dump/load progress on stderr, so dump data written to stdout remains safe to redirect or pipe. In `auto` mode a terminal gets a colored, in-place progress bar while redirected stderr receives periodic plain-text progress lines. Set `NEO4J_BOLT_PROGRESS=pretty` to force the terminal display through a container or other wrapper that hides the TTY, or `NEO4J_BOLT_PROGRESS=plain` to force log-friendly output. `NO_COLOR` disables colors without disabling the in-place display.
|
|
179
206
|
|
|
180
207
|
Both `dump` and `load` accept `--color COLOR` to choose the spinner and filled progress-bar accent. The default is `cyan`. The Ruby API exposes the same setting as `progress_color:`. Available colors are `red`, `green`, `yellow`, `blue`, `magenta`, `cyan`, `white`, `bright-red`, `bright-green`, `bright-yellow`, `bright-blue`, `bright-magenta`, `bright-cyan`, and `bright-white`.
|
|
@@ -184,13 +211,24 @@ neo4j_bolt dump --color magenta -o database.dump
|
|
|
184
211
|
neo4j_bolt load --color bright-blue database.dump
|
|
185
212
|
```
|
|
186
213
|
|
|
187
|
-
|
|
214
|
+
Both commands also accept `--data-only`. On `dump`, it omits `u`, `k`, and `i` records. On `load`, it accepts a full dump but ignores its schema records:
|
|
215
|
+
|
|
216
|
+
```bash
|
|
217
|
+
neo4j_bolt dump --data-only -o data.dump
|
|
218
|
+
neo4j_bolt load --data-only database.dump
|
|
219
|
+
```
|
|
188
220
|
|
|
189
|
-
|
|
221
|
+
The `0, 1, 2, ...` IDs are synthetic dump-local IDs. Database-internal IDs and element IDs are never written to the persistent format. Dumping keeps the count, node, and relationship reads in one transaction, so the driver-provided entity identity used to connect the two streamed result sets stays within Neo4j's transaction-scoped identity guarantee. To keep dump numbering deterministic, ordering uses `elementId()` on modern Neo4j and an internal `id()` fallback only on Neo4j 4.4, where `elementId()` does not exist. Old Neo4jBolt dumps containing only `n` and `r` records remain loadable.
|
|
222
|
+
|
|
223
|
+
For relational loads, the adapter assigns a random temporary label and the fixed reserved property `__neo4j_bolt_load_id` to imported nodes. After the nodes are committed it builds a temporary `neo4j_bolt_` index on that property, uses indexed lookups while creating relationships, drops the index, and removes the temporary metadata in batches. The fixed property key is reused by every load instead of registering a new random property-key token each time. Neo4j can keep the property key registered after all values have been removed, so it may remain visible in Browser's property-key list; no imported node retains the property after successful cleanup. Relational loads refuse to run if the dump itself, or existing data during `force_append`, currently uses the reserved property. No database-internal identity is carried from one load transaction to another. Loading a relational dump therefore requires permission to create and drop an index.
|
|
190
224
|
|
|
191
225
|
Loads start with batches of 5,000 records by default. This is an initial ceiling rather than a claimed optimum: if Neo4j returns a transaction-memory/resource error whose server semantics guarantee rollback, the loader halves the failed batch and retries it, then keeps the smaller size for the rest of that phase. Node and relationship phases adapt independently. Other errors are not retried, because a generic connection failure cannot safely prove that a `CREATE` transaction did not commit. Callers can change the initial ceiling with `initial_batch_size:`; the CLI exposes the same setting as `--batch-size`.
|
|
192
226
|
|
|
193
|
-
Loading requires an empty database unless `force_append: true` is passed. `
|
|
227
|
+
Loading data requires an empty database unless `force_append: true` is passed. For a dump that contains `u`, `k`, or `i` records and is being loaded with schema enabled, the target must also have no application constraints or property indexes. Neo4j's built-in token lookup indexes do not count as application schema. Old `n`/`r`-only dumps, and loads using `schema: false` / `--data-only`, keep the old data-only emptiness rule and may load into a data-empty database that already has application schema.
|
|
228
|
+
|
|
229
|
+
With a normal full restore, the schema preflight happens before any data is written and the saved schema is created strictly after the data load. With `force_append: true`, existing schema is allowed and restored schema entries use `IF NOT EXISTS`. Existing constraints and indexes are never removed by dump loading.
|
|
230
|
+
|
|
231
|
+
Data is loaded before schema records are restored, so bulk loading does not pay constraint/index maintenance costs for every inserted record. `load_database_dump` owns its batch transactions and therefore rejects being called from inside `transaction`.
|
|
194
232
|
|
|
195
233
|
## CLI
|
|
196
234
|
|
|
@@ -200,23 +238,28 @@ The `neo4j_bolt` executable retains these commands:
|
|
|
200
238
|
| --- | --- |
|
|
201
239
|
| `neo4j_bolt console` | Open an IRB console with Neo4jBolt loaded |
|
|
202
240
|
| `neo4j_bolt clear --srsly` | Delete all nodes and relationships |
|
|
203
|
-
| `neo4j_bolt dump [--color COLOR]` | Write the textual database dump |
|
|
204
|
-
| `neo4j_bolt load [--force] [--batch-size N] [--color COLOR] PATH` | Load a textual dump |
|
|
241
|
+
| `neo4j_bolt dump [--data-only] [--color COLOR]` | Write the textual database dump |
|
|
242
|
+
| `neo4j_bolt load [--force] [--data-only] [--batch-size N] [--color COLOR] PATH` | Load a textual dump |
|
|
205
243
|
| `neo4j_bolt index ls` | List constraints and indexes |
|
|
206
244
|
| `neo4j_bolt index rm --force` | Remove all constraints and indexes |
|
|
207
245
|
| `neo4j_bolt visualize` | Generate a GraphViz document |
|
|
208
246
|
|
|
209
|
-
Use `--host HOST:PORT`, `--username USER`, `--password PASSWORD`, and `--database DATABASE` to override connection settings.
|
|
247
|
+
Use `--host HOST:PORT`, `--username USER`, `--password PASSWORD`, and `--database DATABASE` to override connection settings. Host and port otherwise inherit `NEO4J_URI` when present, falling back to `localhost:7687`; username, password, and database inherit `NEO4J_USERNAME`, `NEO4J_PASSWORD`, and `NEO4J_DATABASE`. Prefer the password environment variable for routine use because command-line arguments may be visible in shell history or process listings. `gli` remains a runtime dependency for the CLI, and `pry` remains for the separate `bin/console` executable.
|
|
210
248
|
|
|
211
249
|
## Tested Neo4j versions
|
|
212
250
|
|
|
213
|
-
The
|
|
251
|
+
The complete integration suite is run against both Community and Enterprise for each pinned Neo4j generation:
|
|
214
252
|
|
|
215
253
|
- `neo4j:4.4.48-community`
|
|
254
|
+
- `neo4j:4.4.48-enterprise`
|
|
216
255
|
- `neo4j:5.26.28-community`
|
|
256
|
+
- `neo4j:5.26.28-enterprise`
|
|
217
257
|
- `neo4j:2026.06.0-community`
|
|
258
|
+
- `neo4j:2026.06.0-enterprise`
|
|
259
|
+
|
|
260
|
+
No compatibility beyond this matrix is claimed for `0.5.0`.
|
|
218
261
|
|
|
219
|
-
|
|
262
|
+
Enterprise test containers are started with `NEO4J_ACCEPT_LICENSE_AGREEMENT=yes`. The Community targets verify that Community-compatible features remain portable; the Enterprise targets additionally exercise successful node-key dump/load instead of merely observing that Community rejects node keys.
|
|
220
263
|
|
|
221
264
|
Run one modern LTS target:
|
|
222
265
|
|
|
@@ -254,13 +297,15 @@ To point RSpec itself at an already disposable database, set `NEO4J_BOLT_TEST_HO
|
|
|
254
297
|
| CLI commands | Preserved |
|
|
255
298
|
| `BoltSocket`, `BoltBuffer`, protocol markers/state/parser/packer | Intentionally removed private implementation details |
|
|
256
299
|
|
|
257
|
-
## 0.
|
|
300
|
+
## 0.5.0 migration notes
|
|
258
301
|
|
|
259
302
|
- Ruby 3.4 or newer is required; Ruby 2.x/3.0–3.3 applications should stay on 0.3.x until upgraded.
|
|
260
303
|
- The exact prerelease upstream dependency is pinned while no stable `neo4j-ruby-driver` 6.2.x exists.
|
|
261
304
|
- Connections are pooled and safe for concurrent use instead of one mutable socket per including object.
|
|
262
|
-
-
|
|
263
|
-
-
|
|
305
|
+
- `NEO4J_URI` now supplies the default host and port for `bolt://` and `neo4j://` endpoint notation; an explicit CLI `--host` or Ruby `bolt_host`/`bolt_port` setting overrides it.
|
|
306
|
+
- Basic authentication and database selection are exposed through `bolt_username`, `bolt_password`, and `bolt_database`, with `NEO4J_USERNAME`, `NEO4J_PASSWORD`, and `NEO4J_DATABASE` as environment defaults.
|
|
307
|
+
- TLS, routing behavior, URI query parameters, and other upstream-driver configuration are still not exposed through the compatibility API.
|
|
308
|
+
- Relational loads reuse one reserved `__neo4j_bolt_load_id` property key while retaining the existing indexed lookup algorithm, preventing a new property-key token from being registered on every load.
|
|
264
309
|
- `BoltSocket`, `BoltBuffer`, `ServerState`, `BoltMarker`, `UnexpectedServerResponse`, `State`, and `CypherError` were undocumented wire internals and are removed.
|
|
265
310
|
|
|
266
311
|
## Development
|
data/bin/neo4j_bolt
CHANGED
|
@@ -21,15 +21,18 @@ class App
|
|
|
21
21
|
version Neo4jBolt::VERSION
|
|
22
22
|
|
|
23
23
|
flag [:v, :verbosity], :default_value => 0
|
|
24
|
-
flag [:h, :host], :default_value =>
|
|
24
|
+
flag [:h, :host], :default_value => nil,
|
|
25
|
+
:desc => 'Neo4j host:port (default: NEO4J_URI or localhost:7687)'
|
|
25
26
|
flag [:u, :username], :default_value => nil, :desc => 'Neo4j username (default: NEO4J_USERNAME)'
|
|
26
27
|
flag [:p, :password], :default_value => nil, :desc => 'Neo4j password (default: NEO4J_PASSWORD)'
|
|
27
28
|
flag [:d, :database], :default_value => nil, :desc => 'Neo4j database (default: NEO4J_DATABASE)'
|
|
28
29
|
|
|
29
30
|
pre do |global_options, command, options, args|
|
|
30
31
|
host = global_options[:host]
|
|
31
|
-
|
|
32
|
-
|
|
32
|
+
unless host.nil?
|
|
33
|
+
Neo4jBolt.bolt_host = host.split(':').first
|
|
34
|
+
Neo4jBolt.bolt_port = host.split(':').last.to_i
|
|
35
|
+
end
|
|
33
36
|
Neo4jBolt.bolt_verbosity = global_options[:verbosity].to_i
|
|
34
37
|
Neo4jBolt.bolt_username = global_options[:username] unless global_options[:username].nil?
|
|
35
38
|
Neo4jBolt.bolt_password = global_options[:password] unless global_options[:password].nil?
|
|
@@ -50,13 +53,20 @@ class App
|
|
|
50
53
|
# --------------------------------------------
|
|
51
54
|
|
|
52
55
|
desc 'Dump database'
|
|
53
|
-
long_desc 'Dump
|
|
56
|
+
long_desc 'Dump nodes, relationships, constraints, and property indexes.'
|
|
54
57
|
command :dump do |c|
|
|
55
58
|
c.flag [:o, 'out-file'.to_sym], :default_value => '/dev/stdout'
|
|
56
59
|
c.flag [:color], :default_value => 'cyan', :desc => 'progress accent color'
|
|
60
|
+
c.switch ['data-only'.to_sym], :default_value => false, :negatable => false,
|
|
61
|
+
:desc => 'dump nodes and relationships only'
|
|
57
62
|
c.action do |global_options, options|
|
|
58
63
|
File.open(options['out-file'.to_sym], 'w') do |f|
|
|
59
|
-
dump_database(
|
|
64
|
+
dump_database(
|
|
65
|
+
f,
|
|
66
|
+
progress_io: $stderr,
|
|
67
|
+
progress_color: options[:color],
|
|
68
|
+
schema: !options['data-only'.to_sym]
|
|
69
|
+
)
|
|
60
70
|
end
|
|
61
71
|
end
|
|
62
72
|
end
|
|
@@ -64,13 +74,16 @@ class App
|
|
|
64
74
|
# --------------------------------------------
|
|
65
75
|
|
|
66
76
|
desc 'Load database dump'
|
|
67
|
-
long_desc 'Load nodes and
|
|
77
|
+
long_desc 'Load nodes, relationships, constraints, and property indexes from a database dump.'
|
|
68
78
|
command :load do |c|
|
|
69
79
|
# c.flag [:i, :in_file], :desc => 'input path', :required => true
|
|
70
|
-
c.switch [:f, :force], :default_value => false,
|
|
80
|
+
c.switch [:f, :force], :default_value => false,
|
|
81
|
+
:desc => 'force loading into a database that already contains data or application schema'
|
|
71
82
|
c.flag [:b, 'batch-size'.to_sym], :default_value => Neo4jBolt::LOAD_INITIAL_BATCH_SIZE,
|
|
72
83
|
:desc => 'initial batch size; automatically reduced on transaction-memory errors'
|
|
73
84
|
c.flag [:color], :default_value => 'cyan', :desc => 'progress accent color'
|
|
85
|
+
c.switch ['data-only'.to_sym], :default_value => false, :negatable => false,
|
|
86
|
+
:desc => 'ignore constraints and indexes stored in the dump'
|
|
74
87
|
c.action do |global_options, options, args|
|
|
75
88
|
help_now!('input path is required') if args.empty?
|
|
76
89
|
path = args.shift
|
|
@@ -80,7 +93,8 @@ class App
|
|
|
80
93
|
force_append: options[:force],
|
|
81
94
|
progress_io: $stderr,
|
|
82
95
|
progress_color: options[:color],
|
|
83
|
-
initial_batch_size: options['batch-size'.to_sym]
|
|
96
|
+
initial_batch_size: options['batch-size'.to_sym],
|
|
97
|
+
schema: !options['data-only'.to_sym]
|
|
84
98
|
)
|
|
85
99
|
end
|
|
86
100
|
end
|
data/lib/neo4j_bolt/version.rb
CHANGED
data/lib/neo4j_bolt.rb
CHANGED
|
@@ -4,6 +4,7 @@ require "json"
|
|
|
4
4
|
require "securerandom"
|
|
5
5
|
require "set"
|
|
6
6
|
require "thread"
|
|
7
|
+
require "uri"
|
|
7
8
|
require "neo4j/driver"
|
|
8
9
|
require "neo4j_bolt/version"
|
|
9
10
|
|
|
@@ -18,8 +19,9 @@ module Neo4jBolt
|
|
|
18
19
|
LOAD_MEMORY_GQL_STATUSES = %w[51N72 51N73].freeze
|
|
19
20
|
LOAD_PROGRESS_STEP = 5_000
|
|
20
21
|
DUMP_PROGRESS_STEP = 1_000
|
|
22
|
+
LOAD_ID_PROPERTY = "__neo4j_bolt_load_id"
|
|
21
23
|
private_constant :LOAD_MEMORY_ERROR_CODES, :LOAD_MEMORY_GQL_STATUSES,
|
|
22
|
-
:LOAD_PROGRESS_STEP, :DUMP_PROGRESS_STEP
|
|
24
|
+
:LOAD_PROGRESS_STEP, :DUMP_PROGRESS_STEP, :LOAD_ID_PROPERTY
|
|
23
25
|
|
|
24
26
|
class ProgressReporter
|
|
25
27
|
UPDATE_INTERVAL = 0.5
|
|
@@ -396,8 +398,29 @@ module Neo4jBolt
|
|
|
396
398
|
end
|
|
397
399
|
end
|
|
398
400
|
|
|
399
|
-
|
|
400
|
-
|
|
401
|
+
neo4j_uri = ENV["NEO4J_URI"]
|
|
402
|
+
if neo4j_uri.nil? || neo4j_uri.empty?
|
|
403
|
+
self.bolt_host = "localhost"
|
|
404
|
+
self.bolt_port = 7687
|
|
405
|
+
else
|
|
406
|
+
begin
|
|
407
|
+
parsed_neo4j_uri = URI.parse(neo4j_uri)
|
|
408
|
+
rescue URI::InvalidURIError => error
|
|
409
|
+
raise Error, "Invalid NEO4J_URI: #{error.message}"
|
|
410
|
+
end
|
|
411
|
+
|
|
412
|
+
unless %w[bolt neo4j].include?(parsed_neo4j_uri.scheme) && parsed_neo4j_uri.host
|
|
413
|
+
raise Error, "NEO4J_URI must be a bolt:// or neo4j:// URI with a host"
|
|
414
|
+
end
|
|
415
|
+
unless parsed_neo4j_uri.userinfo.nil? &&
|
|
416
|
+
["", "/"].include?(parsed_neo4j_uri.path.to_s) &&
|
|
417
|
+
parsed_neo4j_uri.query.nil? && parsed_neo4j_uri.fragment.nil?
|
|
418
|
+
raise Error, "NEO4J_URI may only contain scheme, host, and port"
|
|
419
|
+
end
|
|
420
|
+
|
|
421
|
+
self.bolt_host = parsed_neo4j_uri.host
|
|
422
|
+
self.bolt_port = parsed_neo4j_uri.port || 7687
|
|
423
|
+
end
|
|
401
424
|
self.bolt_verbosity = 0
|
|
402
425
|
self.bolt_username = ENV["NEO4J_USERNAME"]
|
|
403
426
|
self.bolt_password = ENV["NEO4J_PASSWORD"]
|
|
@@ -571,11 +594,13 @@ module Neo4jBolt
|
|
|
571
594
|
nil
|
|
572
595
|
end
|
|
573
596
|
|
|
574
|
-
def dump_database(io, progress_io: nil, progress_color: "cyan")
|
|
597
|
+
def dump_database(io, progress_io: nil, progress_color: "cyan", schema: true)
|
|
575
598
|
progress = ProgressReporter.new(progress_io, color: progress_color)
|
|
576
599
|
dumped_nodes = 0
|
|
577
600
|
dumped_relationships = 0
|
|
578
601
|
|
|
602
|
+
dump_schema(io) if schema
|
|
603
|
+
|
|
579
604
|
transaction do
|
|
580
605
|
identity_function = dump_identity_function
|
|
581
606
|
total_nodes = neo4j_query_expect_one("MATCH (n) RETURN count(n) AS count")["count"]
|
|
@@ -632,7 +657,7 @@ module Neo4jBolt
|
|
|
632
657
|
end
|
|
633
658
|
|
|
634
659
|
def load_database_dump(io, force_append: false, progress_io: nil, progress_color: "cyan",
|
|
635
|
-
initial_batch_size: LOAD_INITIAL_BATCH_SIZE)
|
|
660
|
+
initial_batch_size: LOAD_INITIAL_BATCH_SIZE, schema: true)
|
|
636
661
|
raise Error, "load_database_dump cannot run inside a transaction" if transaction_context
|
|
637
662
|
|
|
638
663
|
initial_batch_size = Integer(initial_batch_size)
|
|
@@ -646,16 +671,41 @@ module Neo4jBolt
|
|
|
646
671
|
progress = ProgressReporter.new(progress_io, color: progress_color)
|
|
647
672
|
node_batches = Hash.new { |hash, key| hash[key] = [] }
|
|
648
673
|
relationship_batches = Hash.new { |hash, key| hash[key] = [] }
|
|
649
|
-
total_nodes, total_relationships
|
|
674
|
+
total_nodes, total_relationships, schema_entries =
|
|
675
|
+
parse_dump(io, node_batches, relationship_batches, progress)
|
|
676
|
+
|
|
677
|
+
if !force_append && schema && !schema_entries.empty?
|
|
678
|
+
ensure_database_has_no_application_schema!
|
|
679
|
+
end
|
|
650
680
|
|
|
651
681
|
temporary_token = SecureRandom.hex(12)
|
|
652
682
|
temporary_label = "__neo4j_bolt_load_#{temporary_token}"
|
|
653
|
-
temporary_id_property =
|
|
683
|
+
temporary_id_property = LOAD_ID_PROPERTY
|
|
654
684
|
temporary_index = "neo4j_bolt_load_#{temporary_token}"
|
|
655
685
|
quoted_temporary_label = quote_identifier(temporary_label)
|
|
656
686
|
quoted_temporary_id = quote_identifier(temporary_id_property)
|
|
657
687
|
quoted_temporary_index = quote_identifier(temporary_index)
|
|
658
688
|
needs_relationship_lookup = total_relationships.positive?
|
|
689
|
+
|
|
690
|
+
if needs_relationship_lookup
|
|
691
|
+
dump_uses_reserved_property = node_batches.each_value.any? do |nodes|
|
|
692
|
+
nodes.any? { |node| node.fetch("properties", {}).key?(LOAD_ID_PROPERTY) }
|
|
693
|
+
end
|
|
694
|
+
if dump_uses_reserved_property
|
|
695
|
+
raise Error, "Dump contains reserved Neo4jBolt load property #{LOAD_ID_PROPERTY.inspect}."
|
|
696
|
+
end
|
|
697
|
+
|
|
698
|
+
if force_append
|
|
699
|
+
reserved_property_in_use = neo4j_query(
|
|
700
|
+
"MATCH (n) WHERE n.#{quoted_temporary_id} IS NOT NULL RETURN 1 AS present LIMIT 1"
|
|
701
|
+
).any?
|
|
702
|
+
if reserved_property_in_use
|
|
703
|
+
raise Error,
|
|
704
|
+
"Database contains reserved Neo4jBolt load property #{LOAD_ID_PROPERTY.inspect}."
|
|
705
|
+
end
|
|
706
|
+
end
|
|
707
|
+
end
|
|
708
|
+
|
|
659
709
|
loaded_nodes = 0
|
|
660
710
|
loaded_relationships = 0
|
|
661
711
|
index_created = false
|
|
@@ -759,6 +809,10 @@ module Neo4jBolt
|
|
|
759
809
|
raise cleanup_error if original_error.nil? && cleanup_error
|
|
760
810
|
end
|
|
761
811
|
|
|
812
|
+
if schema && !schema_entries.empty?
|
|
813
|
+
restore_dump_schema(schema_entries, progress, if_not_exists: force_append)
|
|
814
|
+
end
|
|
815
|
+
|
|
762
816
|
progress.finish("Loaded: #{loaded_nodes} nodes, #{loaded_relationships} relationships")
|
|
763
817
|
nil
|
|
764
818
|
ensure
|
|
@@ -884,6 +938,157 @@ module Neo4jBolt
|
|
|
884
938
|
"`#{identifier.to_s.gsub("`", "``")}`"
|
|
885
939
|
end
|
|
886
940
|
|
|
941
|
+
def dump_schema(io)
|
|
942
|
+
entries = []
|
|
943
|
+
constraint_names = Set.new
|
|
944
|
+
|
|
945
|
+
neo4j_query("SHOW CONSTRAINTS").each do |row|
|
|
946
|
+
kind = dump_constraint_kind(row["type"])
|
|
947
|
+
value = schema_dump_value(row, kind)
|
|
948
|
+
constraint_names << value.fetch(:name)
|
|
949
|
+
entries << [kind, value]
|
|
950
|
+
end
|
|
951
|
+
|
|
952
|
+
neo4j_query("SHOW INDEXES").each do |row|
|
|
953
|
+
next unless dumpable_property_index?(row, constraint_names)
|
|
954
|
+
|
|
955
|
+
entries << ["i", schema_dump_value(row, "i")]
|
|
956
|
+
end
|
|
957
|
+
|
|
958
|
+
kind_order = { "u" => 0, "k" => 1, "i" => 2 }
|
|
959
|
+
entries.sort_by { |kind, value| [kind_order.fetch(kind), value.fetch(:name)] }.each do |kind, value|
|
|
960
|
+
io.puts "#{kind} #{JSON.generate(value)}"
|
|
961
|
+
end
|
|
962
|
+
end
|
|
963
|
+
|
|
964
|
+
def dump_constraint_kind(type)
|
|
965
|
+
case type
|
|
966
|
+
when "UNIQUENESS", "NODE_PROPERTY_UNIQUENESS", "RELATIONSHIP_PROPERTY_UNIQUENESS"
|
|
967
|
+
"u"
|
|
968
|
+
when "NODE_KEY", "RELATIONSHIP_KEY"
|
|
969
|
+
"k"
|
|
970
|
+
else
|
|
971
|
+
raise Error,
|
|
972
|
+
"Cannot dump #{type.inspect} constraint; use schema: false (or --data-only in the CLI)"
|
|
973
|
+
end
|
|
974
|
+
end
|
|
975
|
+
|
|
976
|
+
def dumpable_property_index?(row, constraint_names)
|
|
977
|
+
return false if row["owningConstraint"]
|
|
978
|
+
return false if row["uniqueness"] == "UNIQUE"
|
|
979
|
+
return false if constraint_names.include?(row["name"])
|
|
980
|
+
return false if row["type"] == "LOOKUP"
|
|
981
|
+
return true if %w[BTREE RANGE].include?(row["type"])
|
|
982
|
+
|
|
983
|
+
raise Error,
|
|
984
|
+
"Cannot dump #{row['type'].inspect} index #{row['name'].inspect}; " \
|
|
985
|
+
"use schema: false (or --data-only in the CLI)"
|
|
986
|
+
end
|
|
987
|
+
|
|
988
|
+
def schema_dump_value(row, kind)
|
|
989
|
+
name = row["name"]
|
|
990
|
+
entity_type = row["entityType"]
|
|
991
|
+
labels_or_types = row["labelsOrTypes"]
|
|
992
|
+
properties = row["properties"]
|
|
993
|
+
|
|
994
|
+
unless name.is_a?(String) && !name.empty? &&
|
|
995
|
+
%w[NODE RELATIONSHIP].include?(entity_type) &&
|
|
996
|
+
labels_or_types.is_a?(Array) && labels_or_types.size == 1 &&
|
|
997
|
+
properties.is_a?(Array) && !properties.empty? &&
|
|
998
|
+
properties.all? { |property| property.is_a?(String) && !property.empty? }
|
|
999
|
+
raise Error, "Cannot dump #{kind} schema entry #{name.inspect}: unsupported schema shape"
|
|
1000
|
+
end
|
|
1001
|
+
|
|
1002
|
+
{
|
|
1003
|
+
name: name,
|
|
1004
|
+
entity_type: entity_type,
|
|
1005
|
+
label_or_type: labels_or_types.first,
|
|
1006
|
+
properties: properties
|
|
1007
|
+
}
|
|
1008
|
+
end
|
|
1009
|
+
|
|
1010
|
+
def restore_dump_schema(entries, progress, if_not_exists: false)
|
|
1011
|
+
progress&.note("Restoring schema: #{entries.size} entries...")
|
|
1012
|
+
entries.each do |kind, value|
|
|
1013
|
+
neo4j_query(schema_create_statement(kind, value, if_not_exists: if_not_exists))
|
|
1014
|
+
end
|
|
1015
|
+
end
|
|
1016
|
+
|
|
1017
|
+
def schema_create_statement(kind, value, if_not_exists: false)
|
|
1018
|
+
validate_schema_dump_value!(kind, value)
|
|
1019
|
+
|
|
1020
|
+
name = quote_identifier(value.fetch("name"))
|
|
1021
|
+
if_not_exists_clause = if_not_exists ? " IF NOT EXISTS" : ""
|
|
1022
|
+
entity_type = value.fetch("entity_type")
|
|
1023
|
+
label_or_type = quote_identifier(value.fetch("label_or_type"))
|
|
1024
|
+
properties = value.fetch("properties")
|
|
1025
|
+
variable = entity_type == "NODE" ? "n" : "r"
|
|
1026
|
+
pattern = if entity_type == "NODE"
|
|
1027
|
+
"(#{variable}:#{label_or_type})"
|
|
1028
|
+
else
|
|
1029
|
+
"()-[#{variable}:#{label_or_type}]-()"
|
|
1030
|
+
end
|
|
1031
|
+
property_expressions = properties.map { |property| "#{variable}.#{quote_identifier(property)}" }
|
|
1032
|
+
|
|
1033
|
+
case kind
|
|
1034
|
+
when "u", "k"
|
|
1035
|
+
property_expression = if property_expressions.one?
|
|
1036
|
+
property_expressions.first
|
|
1037
|
+
else
|
|
1038
|
+
"(#{property_expressions.join(', ')})"
|
|
1039
|
+
end
|
|
1040
|
+
constraint_type = if kind == "u"
|
|
1041
|
+
"UNIQUE"
|
|
1042
|
+
elsif entity_type == "NODE"
|
|
1043
|
+
"NODE KEY"
|
|
1044
|
+
else
|
|
1045
|
+
"RELATIONSHIP KEY"
|
|
1046
|
+
end
|
|
1047
|
+
"CREATE CONSTRAINT #{name}#{if_not_exists_clause} FOR #{pattern} " \
|
|
1048
|
+
"REQUIRE #{property_expression} IS #{constraint_type}"
|
|
1049
|
+
when "i"
|
|
1050
|
+
"CREATE INDEX #{name}#{if_not_exists_clause} FOR #{pattern} ON (#{property_expressions.join(', ')})"
|
|
1051
|
+
else
|
|
1052
|
+
raise Error, "Unexpected schema dump kind: #{kind}"
|
|
1053
|
+
end
|
|
1054
|
+
end
|
|
1055
|
+
|
|
1056
|
+
def validate_schema_dump_value!(kind, value)
|
|
1057
|
+
unless %w[u k i].include?(kind) && value.is_a?(Hash)
|
|
1058
|
+
raise Error, "Invalid #{kind} schema dump entry"
|
|
1059
|
+
end
|
|
1060
|
+
|
|
1061
|
+
name = value.fetch("name")
|
|
1062
|
+
entity_type = value.fetch("entity_type")
|
|
1063
|
+
label_or_type = value.fetch("label_or_type")
|
|
1064
|
+
properties = value.fetch("properties")
|
|
1065
|
+
|
|
1066
|
+
return if name.is_a?(String) && !name.empty? &&
|
|
1067
|
+
%w[NODE RELATIONSHIP].include?(entity_type) &&
|
|
1068
|
+
label_or_type.is_a?(String) && !label_or_type.empty? &&
|
|
1069
|
+
properties.is_a?(Array) && !properties.empty? &&
|
|
1070
|
+
properties.all? { |property| property.is_a?(String) && !property.empty? }
|
|
1071
|
+
|
|
1072
|
+
raise Error, "Invalid #{kind} schema dump entry"
|
|
1073
|
+
end
|
|
1074
|
+
|
|
1075
|
+
def ensure_database_has_no_application_schema!
|
|
1076
|
+
constraints = neo4j_query("SHOW CONSTRAINTS").filter_map { |row| row["name"] }
|
|
1077
|
+
indexes = neo4j_query("SHOW INDEXES").filter_map do |row|
|
|
1078
|
+
next if row["type"] == "LOOKUP"
|
|
1079
|
+
next if row["owningConstraint"]
|
|
1080
|
+
next if row["uniqueness"] == "UNIQUE"
|
|
1081
|
+
|
|
1082
|
+
row["name"]
|
|
1083
|
+
end
|
|
1084
|
+
schema_names = constraints + indexes
|
|
1085
|
+
return if schema_names.empty?
|
|
1086
|
+
|
|
1087
|
+
raise Error,
|
|
1088
|
+
"There are constraints or indexes in this database, exiting now: " \
|
|
1089
|
+
"#{schema_names.sort.join(', ')}"
|
|
1090
|
+
end
|
|
1091
|
+
|
|
887
1092
|
# Neo4j 4.4 has no elementId() function. Its id() fallback is used only for
|
|
888
1093
|
# deterministic in-process ordering; those database IDs never enter a dump.
|
|
889
1094
|
def dump_identity_function
|
|
@@ -896,6 +1101,7 @@ module Neo4jBolt
|
|
|
896
1101
|
def parse_dump(io, node_batches, relationship_batches, progress = nil)
|
|
897
1102
|
node_count = 0
|
|
898
1103
|
relationship_count = 0
|
|
1104
|
+
schema_entries = []
|
|
899
1105
|
progress&.update("Reading dump: 0 nodes, 0 relationships", force: true)
|
|
900
1106
|
|
|
901
1107
|
io.each_line.with_index(1) do |line, line_number|
|
|
@@ -903,16 +1109,22 @@ module Neo4jBolt
|
|
|
903
1109
|
next if line.empty?
|
|
904
1110
|
|
|
905
1111
|
kind, json = line.split(" ", 2)
|
|
906
|
-
|
|
1112
|
+
unless json && %w[n r u k i].include?(kind)
|
|
1113
|
+
raise Error, "Invalid dump entry on line #{line_number}"
|
|
1114
|
+
end
|
|
907
1115
|
|
|
908
1116
|
value = JSON.parse(json)
|
|
909
|
-
|
|
1117
|
+
case kind
|
|
1118
|
+
when "n"
|
|
910
1119
|
labels = value.fetch("labels")
|
|
911
1120
|
node_batches[labels.sort] << value
|
|
912
1121
|
node_count += 1
|
|
913
|
-
|
|
1122
|
+
when "r"
|
|
914
1123
|
relationship_batches[value.fetch("type")] << value
|
|
915
1124
|
relationship_count += 1
|
|
1125
|
+
else
|
|
1126
|
+
validate_schema_dump_value!(kind, value)
|
|
1127
|
+
schema_entries << [kind, value]
|
|
916
1128
|
end
|
|
917
1129
|
if ((node_count + relationship_count) % LOAD_PROGRESS_STEP).zero?
|
|
918
1130
|
progress&.update("Reading dump: #{node_count} nodes, #{relationship_count} relationships")
|
|
@@ -922,7 +1134,7 @@ module Neo4jBolt
|
|
|
922
1134
|
end
|
|
923
1135
|
|
|
924
1136
|
progress&.finish("Read dump: #{node_count} nodes, #{relationship_count} relationships")
|
|
925
|
-
[node_count, relationship_count]
|
|
1137
|
+
[node_count, relationship_count, schema_entries]
|
|
926
1138
|
end
|
|
927
1139
|
|
|
928
1140
|
def adaptive_each_slice(items, batch_size, progress, kind)
|