perobs 4.2.0 → 4.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 7790ee42586bb2b8fca115f93ed277c4a8057f7a7027b356baea7b066da953e5
4
- data.tar.gz: 110e0710a84ef544a4874cf868ec1662dfc900077d1894b24f53bcdaeaeeed34
3
+ metadata.gz: 90c77003da7bd628fa753bfe35918d8c702b433ac5113ba9fc206f7afa114f6c
4
+ data.tar.gz: c38f786cb10a040048b7d08b75c16baf034c116a3f9648ece3af7b50e7898446
5
5
  SHA512:
6
- metadata.gz: d95d845c7e8bd183f53b60369415bde86cd766c224bcbc2c52c870c60542a786f359a63eecc4e8829055cee6a1674bc952277749da183432b4b3abae7536efbb
7
- data.tar.gz: 5fa1712fb01118d955d86396aec87b319c75856f6602ccf1a19f475a3dc64dc65a540e35271e6f2c6c7ee6a10c5cb03da30d43945a0dc264c7b0e48f575269d1
6
+ metadata.gz: 1fea934c359190c3b171c99e4dbffb3af58ec2f9dd755eedc0acb7db556f0e2471bd3c527a2910e5d8f17cd59b3f1ca4079436bc80bd11ea5d6efa60f55a54e9
7
+ data.tar.gz: db71320a9672848a3e35088e7a8b3389278c987a1bea905336e3f8bfff6daeb27564724b6f9d681d364684a57ce8593021041626615dace155e1849b0944003b
data/README.md CHANGED
@@ -6,11 +6,12 @@ them from PEROBS::Object. They will be in memory when needed and
6
6
  transparently stored into a persistent storage.
7
7
 
8
8
  This library is ideal for Ruby applications that work on huge, mostly
9
- constant data sets and usually handle a small subset of the data at a
9
+ static data sets and usually process a small subset of the data at a
10
10
  time. To ensure data consistency of a larger data set, you can use
11
11
  transactions to make modifications of multiple objects atomicaly.
12
12
  Transactions can be nested and are aborted when an exception is
13
- raised.
13
+ raised. PEROBS is thread-safe, so you can use it in a multi-threaded
14
+ application.
14
15
 
15
16
  ## Usage
16
17
 
@@ -108,7 +109,7 @@ class Person < PEROBS::Object
108
109
  attr_init(:father) do { @store.new(Person, 'Dad') }
109
110
  end
110
111
 
111
- def merry(spouse)
112
+ def marry(spouse)
112
113
  self.spouse = spouse
113
114
  self.status = :married
114
115
  end
@@ -120,15 +121,18 @@ class Person < PEROBS::Object
120
121
 
121
122
  end
122
123
 
123
- store = PEROBS::Store.new('family')
124
- store['grandpa'] = joe = store.new(Person, 'Joe')
125
- store['grandma'] = jane = store.new(Person, 'Jane')
126
- jim = store.new(Person, 'Jim')
127
- jim.father = joe
128
- joe.kids << jim
129
- jim.mother = jane
130
- jane.kids << jim
131
- store.exit
124
+ begin
125
+ store = PEROBS::Store.new('family')
126
+ store['grandpa'] = joe = store.new(Person, 'Joe')
127
+ store['grandma'] = jane = store.new(Person, 'Jane')
128
+ jim = store.new(Person, 'Jim')
129
+ jim.father = joe
130
+ joe.kids << jim
131
+ jim.mother = jane
132
+ jane.kids << jim
133
+ ensure
134
+ store.exit
135
+ end
132
136
  ```
133
137
 
134
138
  When you run this script, a folder named 'family' will be created. It
@@ -166,9 +170,15 @@ object to another object.
166
170
 
167
171
  ### Caveats and known issues
168
172
 
169
- PEROBS is currently not thread-safe. You cannot simultaneously access
170
- the database from multiple application. The library uses locks to
171
- ensure that only one Store object is accessing the database at a time.
173
+ You cannot simultaneously access the database from multiple
174
+ applications concurrently. The library uses locks to ensure that only
175
+ one Store object is accessing the database at a time.
176
+
177
+ In case the application terminates without calling Store::exit(), the
178
+ database or the database index could get corrupted. To check the
179
+ consistency of your database you can use Store::check(). To check and
180
+ repair the database you can call Store::repair(). Depending on the
181
+ size of your database, these operations can last minutes to hours.
172
182
 
173
183
  ## Installation
174
184
 
@@ -188,7 +198,8 @@ Or install it yourself as:
188
198
 
189
199
  ## Copyright and License
190
200
 
191
- Copyright (c) 2015, 2016, 2017 by Chris Schlaeger <chris@taskjuggler.org>
201
+ Copyright (c) 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022 by
202
+ Chris Schlaeger <chris@taskjuggler.org>
192
203
 
193
204
  PEROBS and all accompanying files are licensed under this MIT License
194
205
 
data/lib/perobs/BTree.rb CHANGED
@@ -70,7 +70,7 @@ module PEROBS
70
70
  @nodes.register_custom_data('first_leaf')
71
71
  @nodes.register_custom_data('last_leaf')
72
72
  @nodes.register_custom_data('btree_size')
73
- @node_cache = PersistentObjectCache.new(2**16, -1, BTreeNode, self)
73
+ @node_cache = PersistentObjectCache.new(2**13, 2**13, BTreeNode, self)
74
74
  @root = @first_leaf = @last_leaf = nil
75
75
  @size = 0
76
76
 
@@ -190,7 +190,7 @@ module PEROBS
190
190
  "Number of leave nodes: #{stats.leave_nodes}; " +
191
191
  "Number of leaves: #{stats.leaves}"
192
192
 
193
- !stats.nil?
193
+ true
194
194
  end
195
195
 
196
196
  # Register a new node as root node of the tree.
@@ -59,7 +59,7 @@ module PEROBS
59
59
  # if not
60
60
  def initialize(tree, node_address = nil, parent = nil, is_leaf = true,
61
61
  prev_sibling = nil, next_sibling = nil,
62
- keys = [], values = [], children = [])
62
+ keys = nil, values = nil, children = nil)
63
63
  @tree = tree
64
64
  if node_address == 0
65
65
  PEROBS.log.fatal "Node address may not be 0"
@@ -68,17 +68,17 @@ module PEROBS
68
68
  @parent = link(parent)
69
69
  @prev_sibling = link(prev_sibling)
70
70
  @next_sibling = link(next_sibling)
71
- @keys = keys
71
+ @keys = keys || []
72
72
  if (@is_leaf = is_leaf)
73
- @values = values
74
- @children = []
73
+ @values = values || []
74
+ @children = nil
75
75
  else
76
- @children = children
77
- @values = []
76
+ @children = children || []
77
+ @values = nil
78
78
  end
79
79
  end
80
80
 
81
- # Create a new SpaceTreeNode. This method should be used for the creation
81
+ # Create a new BTreeNode. This method should be used for the creation
82
82
  # of new nodes instead of calling the constructor directly.
83
83
  # @param tree [BTree] The tree the new node should belong to
84
84
  # @param parent [BTreeNode] The parent node
@@ -130,7 +130,8 @@ module PEROBS
130
130
  ary = bytes.unpack(BTreeNode::node_bytes_format(tree))
131
131
  # Read is_leaf
132
132
  if ary[0] != 0 && ary[0] != 1
133
- PEROBS.log.fatal "First byte of a BTreeNode entry must be 0 or 1"
133
+ PEROBS.log.fatal "First byte of a BTreeNode entry at address " +
134
+ "#{address} must be 0 or 1 but is #{ary[0]}"
134
135
  end
135
136
  is_leaf = ary[0] == 0 ? false : true
136
137
  # This is the number of keys this node has.
@@ -350,6 +351,7 @@ module PEROBS
350
351
  unless @parent
351
352
  # The node is the root node. We need to create a parent node first.
352
353
  self.parent = link(BTreeNode::create(@tree, nil, false))
354
+ @tree.node_cache.insert(self)
353
355
  @parent.set_child(0, self)
354
356
  @tree.set_root(@parent)
355
357
  end
@@ -382,6 +384,7 @@ module PEROBS
382
384
  end
383
385
 
384
386
  i = search_key_index(key)
387
+ @tree.node_cache.insert(self)
385
388
  if @keys[i] == key
386
389
  # Overwrite existing entries
387
390
  @keys[i] = key
@@ -390,7 +393,6 @@ module PEROBS
390
393
  else
391
394
  @children[i + 1] = link(value_or_child)
392
395
  end
393
- @tree.node_cache.insert(self)
394
396
 
395
397
  return false
396
398
  else
@@ -401,7 +403,6 @@ module PEROBS
401
403
  else
402
404
  @children.insert(i + 1, link(value_or_child))
403
405
  end
404
- @tree.node_cache.insert(self)
405
406
 
406
407
  return true
407
408
  end
@@ -417,8 +418,8 @@ module PEROBS
417
418
  update_branch_key(key) if index == 0
418
419
 
419
420
  # Delete the corresponding value.
420
- removed_value = @values.delete_at(index)
421
421
  @tree.node_cache.insert(self)
422
+ removed_value = @values.delete_at(index)
422
423
 
423
424
  if @keys.length < min_keys
424
425
  if @prev_sibling && @prev_sibling.parent == @parent
@@ -481,9 +482,9 @@ module PEROBS
481
482
  PEROBS.log.fatal "Leaf nodes are too big to merge"
482
483
  end
483
484
 
485
+ @tree.node_cache.insert(self)
484
486
  @keys += node.keys
485
487
  @values += node.values
486
- @tree.node_cache.insert(self)
487
488
 
488
489
  node.parent.remove_child(node)
489
490
  end
@@ -494,11 +495,11 @@ module PEROBS
494
495
  end
495
496
 
496
497
  index = @parent.search_node_index(node) - 1
498
+ @tree.node_cache.insert(self)
497
499
  @keys << @parent.keys[index]
498
500
  @keys += node.keys
499
501
  node.children.each { |c| c.parent = link(self) }
500
502
  @children += node.children
501
- @tree.node_cache.insert(self)
502
503
 
503
504
  node.parent.remove_child(node)
504
505
  end
@@ -525,6 +526,7 @@ module PEROBS
525
526
  "#{dest_node.is_leaf} node must be of same kind"
526
527
  end
527
528
 
529
+ @tree.node_cache.insert(dest_node)
528
530
  dest_node.keys[dst_idx, count] = @keys[src_idx, count]
529
531
  if @is_leaf
530
532
  # For leaves we copy the keys and corresponding values.
@@ -537,17 +539,17 @@ module PEROBS
537
539
  dest_node.set_child(dst_idx + i, @children[src_idx + i])
538
540
  end
539
541
  end
540
- @tree.node_cache.insert(dest_node)
541
542
  end
542
543
 
543
544
  def parent=(p)
544
- @parent = p
545
545
  @tree.node_cache.insert(self)
546
+ @parent = p
546
547
 
547
548
  p
548
549
  end
549
550
 
550
551
  def prev_sibling=(node)
552
+ @tree.node_cache.insert(self)
551
553
  @prev_sibling = node
552
554
  if node.nil? && @is_leaf
553
555
  # If this node is a leaf node without a previous sibling we need to
@@ -555,14 +557,12 @@ module PEROBS
555
557
  @tree.set_first_leaf(BTreeNodeLink.new(@tree, self))
556
558
  end
557
559
 
558
- @tree.node_cache.insert(self)
559
-
560
560
  node
561
561
  end
562
562
 
563
563
  def next_sibling=(node)
564
- @next_sibling = node
565
564
  @tree.node_cache.insert(self)
565
+ @next_sibling = node
566
566
  if node.nil? && @is_leaf
567
567
  # If this node is a leaf node without a next sibling we need to
568
568
  # register it as the last leaf node.
@@ -573,25 +573,25 @@ module PEROBS
573
573
  end
574
574
 
575
575
  def set_child(index, child)
576
+ @tree.node_cache.insert(self)
576
577
  if child
577
578
  @children[index] = link(child)
578
579
  @children[index].parent = link(self)
579
580
  else
580
581
  @children[index] = nil
581
582
  end
582
- @tree.node_cache.insert(self)
583
583
 
584
584
  child
585
585
  end
586
586
 
587
587
  def trim(idx)
588
- @keys = @keys[0..idx - 1]
588
+ @tree.node_cache.insert(self)
589
+ @keys.slice!(idx, @keys.length - idx)
589
590
  if @is_leaf
590
- @values = @values[0..idx - 1]
591
+ @values.slice!(idx, @values.length - idx)
591
592
  else
592
- @children = @children[0..idx]
593
+ @children.slice!(idx + 1, @children.length - idx - 1)
593
594
  end
594
- @tree.node_cache.insert(self)
595
595
  end
596
596
 
597
597
  # Search the keys of the node that fits the given key. The result is
@@ -654,13 +654,18 @@ module PEROBS
654
654
  # @yield [key, value]
655
655
  # @return [nil or Hash] nil in case of errors or a hash with some
656
656
  # statistical information about the tree
657
- def check
657
+ def check(&block)
658
658
  stats = Stats.new(nil, 0, 0, 0)
659
659
 
660
660
  traverse do |node, position, stack|
661
661
  if position == 0
662
662
  stats.nodes_count += 1
663
663
  if node.parent
664
+ unless node.parent.is_a?(BTreeNodeLink)
665
+ node.error "parent is a #{node.parent.class} instead of a " +
666
+ "BTreeNodeLink"
667
+ return nil
668
+ end
664
669
  # After a split the nodes will only have half the maximum keys.
665
670
  # For branch nodes one of the split nodes will have even 1 key
666
671
  # less as this will become the branch key in a parent node.
@@ -695,6 +700,16 @@ module PEROBS
695
700
  else
696
701
  stats.branch_depth = node.tree_level
697
702
  end
703
+ if node.prev_sibling && !node.prev_sibling.is_a?(BTreeNodeLink)
704
+ node.error "prev_sibling is a #{node.prev_sibling.class} " +
705
+ "instead of a BTreeNodeLink"
706
+ return nil
707
+ end
708
+ if node.next_sibling && !node.next_sibling.is_a?(BTreeNodeLink)
709
+ node.error "next_sibling is a #{node.next_sibling.class} " +
710
+ "instead of a BTreeNodeLink"
711
+ return nil
712
+ end
698
713
  if node.prev_sibling.nil? && @tree.first_leaf != node
699
714
  node.error "Leaf node #{node.node_address} has no previous " +
700
715
  "sibling but is not the first leaf of the tree"
@@ -708,9 +723,9 @@ module PEROBS
708
723
  unless node.keys.size == node.values.size
709
724
  node.error "Key count (#{node.keys.size}) and value " +
710
725
  "count (#{node.values.size}) don't match"
711
- return nil
726
+ return nil
712
727
  end
713
- unless node.children.empty?
728
+ unless node.children.nil?
714
729
  node.error "@children must be nil for a leaf node"
715
730
  return nil
716
731
  end
@@ -718,14 +733,14 @@ module PEROBS
718
733
  stats.leave_nodes += 1
719
734
  stats.leaves += node.keys.length
720
735
  else
721
- unless node.values.empty?
736
+ unless node.values.nil?
722
737
  node.error "@values must be nil for a branch node"
723
738
  return nil
724
739
  end
725
740
  unless node.children.size == node.keys.size + 1
726
741
  node.error "Key count (#{node.keys.size}) must be one " +
727
742
  "less than children count (#{node.children.size})"
728
- return nil
743
+ return nil
729
744
  end
730
745
  node.children.each_with_index do |child, i|
731
746
  unless child.is_a?(BTreeNodeLink)
@@ -789,7 +804,9 @@ module PEROBS
789
804
  else
790
805
  if block_given?
791
806
  # If a block was given, call this block with the key and value.
792
- return nil unless yield(node.keys[index], node.values[index])
807
+ unless yield(node.keys[index], node.values[index])
808
+ return nil
809
+ end
793
810
  end
794
811
  end
795
812
  end
@@ -135,27 +135,29 @@ module PEROBS
135
135
  node = self
136
136
 
137
137
  # Traverse the tree to find the right node to add or replace the value.
138
+ idx = index
138
139
  while node do
139
140
  # Once we have reached a leaf node we can insert or replace the value.
140
141
  if node.is_leaf?
141
- if index >= node.values.size
142
+ if idx >= node.values.size
142
143
  node.fatal "Set index (#{index}) larger than values array " +
143
- "(#{node.values.size})."
144
+ "(#{idx} >= #{node.values.size})."
144
145
  end
145
- node.values[index] = value
146
+ node.values[idx] = value
146
147
  return
147
148
  else
148
149
  # Descend into the right child node to add the value to.
149
- cidx = node.search_child_index(index)
150
- if (index -= node.offsets[cidx]) < 0
151
- node.fatal "Index (#{index}) became negative"
150
+ cidx = node.search_child_index(idx)
151
+ if (idx -= node.offsets[cidx]) < 0
152
+ node.fatal "Idx (#{idx}) became negative while looking for " +
153
+ "index #{index}."
152
154
  end
153
155
  node = node.children[cidx]
154
156
  end
155
157
  end
156
158
 
157
159
  node.fatal "Could not find proper node to set the value while " +
158
- "looking for index #{index}"
160
+ "looking for index #{index}."
159
161
  end
160
162
 
161
163
  # Insert the given value at the given index. All following values will be
@@ -610,7 +612,7 @@ module PEROBS
610
612
  end
611
613
  end
612
614
 
613
- # @param index [offset] offset to search the child index for
615
+ # @param offset [Integer] offset to search the child index for
614
616
  # @return [Integer] Index of the matching offset or the insert position.
615
617
  def search_child_index(offset)
616
618
  # Handle special case for empty offsets list.
@@ -811,7 +813,7 @@ module PEROBS
811
813
 
812
814
  # Print and log an error message for the node.
813
815
  def fatal(msg)
814
- msg = "Fatal error in BigArray node @#{@_id}: #{msg}\n" + @tree.to_s
816
+ msg = "Fatal error in BigArray node @#{@_id}: #{msg}\n"
815
817
  $stderr.puts msg
816
818
  PEROBS.log.fatal msg
817
819
  end
data/lib/perobs/Cache.rb CHANGED
@@ -122,12 +122,25 @@ module PEROBS
122
122
  # @param id [Integer] ID of the cached PEROBS::ObjectBase
123
123
  def object_by_id(id)
124
124
  idx = id & @mask
125
- # The index is just a hash. We still need to check if the object IDs are
126
- # actually the same before we can return the object.
127
- if (obj = @writes[idx]) && obj._id == id
128
- # The object was in the write cache.
129
- return obj
130
- elsif (obj = @reads[idx]) && obj._id == id
125
+
126
+ if @transaction_stack.empty?
127
+ # The index is just a hash. We still need to check if the object IDs are
128
+ # actually the same before we can return the object.
129
+ if (obj = @writes[idx]) && obj._id == id
130
+ # The object was in the write cache.
131
+ return obj
132
+ end
133
+ else
134
+ # During transactions, the read cache is used to provide fast access
135
+ # to modified objects. But it does not store all modified objects
136
+ # since there can be hash collisions. So we also have to check all
137
+ # transaction objects first.
138
+ if (obj = @transaction_objects[id])
139
+ return obj
140
+ end
141
+ end
142
+
143
+ if (obj = @reads[idx]) && obj._id == id
131
144
  # The object was in the read cache.
132
145
  return obj
133
146
  end
@@ -152,10 +165,20 @@ module PEROBS
152
165
  # active, the write cache is flushed before the transaction is started.
153
166
  def begin_transaction
154
167
  if @transaction_stack.empty?
168
+ if @transaction_thread
169
+ PEROBS.log.fatal 'transaction_thread must be nil'
170
+ end
171
+ @transaction_thread = Thread.current
155
172
  # The new transaction is the top-level transaction. Flush the write
156
173
  # buffer to save the current state of all objects.
157
174
  flush
158
175
  else
176
+ # Nested transactions are currently only supported within the same
177
+ # thread. If we are in another thread, raise TransactionInOtherThread
178
+ # to pause the calling thread for a bit.
179
+ if @transaction_thread != Thread.current
180
+ raise TransactionInOtherThread
181
+ end
159
182
  # Save a copy of all objects that were modified during the enclosing
160
183
  # transaction.
161
184
  @transaction_stack.last.each do |id|
@@ -179,6 +202,7 @@ module PEROBS
179
202
  # into the backend storage.
180
203
  @transaction_stack.pop.each { |id| @transaction_objects[id]._sync }
181
204
  @transaction_objects = ::Hash.new
205
+ @transaction_thread = nil
182
206
  else
183
207
  # A nested transaction completed successfully. We add the list of
184
208
  # modified objects to the list of the enclosing transaction.
@@ -200,6 +224,7 @@ module PEROBS
200
224
  @transaction_stack.pop.each do |id|
201
225
  @transaction_objects[id]._restore(@transaction_stack.length)
202
226
  end
227
+ @transaction_thread = nil
203
228
  end
204
229
 
205
230
  # Clear all cached entries. You must call flush before calling this
@@ -211,6 +236,7 @@ module PEROBS
211
236
  @reads = ::Array.new(2 ** @bits)
212
237
  @writes = ::Array.new(2 ** @bits)
213
238
  @transaction_stack = ::Array.new
239
+ @transaction_thread = nil
214
240
  @transaction_objects = ::Hash.new
215
241
  end
216
242
 
@@ -425,6 +425,8 @@ module PEROBS
425
425
  pack("Q#{@custom_data_values.length}"))
426
426
  end
427
427
  @f.flush
428
+ rescue IOError => e
429
+ PEROBS.log.fatal "Cannot write EquiBlobsFile header: " + e.message
428
430
  end
429
431
  end
430
432