perobs 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 9dd54b9f62dc6b5cc7129d25b9ba87e2d7aa3775
4
+ data.tar.gz: 1431e7ec23c7bf2c18fa65b7c3e14b33bc696b2b
5
+ SHA512:
6
+ metadata.gz: dbf7166adf28acabef48594bb80721512f0b30156f66965e7077f6b4089e429c5d951b621c0c3322bc6c2a0b269e47042994dd9449d75cb00858e0d2a23bbbbc
7
+ data.tar.gz: 73ae5cbfd48a5bc3a53194961398ec6a0ff57a4ac4ed606ba0e3ab1922fcba89cbc72cb90327e2256e2c9709a2a2707bdea8a8bd9124ff973531b800ffc2f304
data/.gitignore ADDED
@@ -0,0 +1,14 @@
1
+ /.bundle/
2
+ /.yardoc
3
+ /Gemfile.lock
4
+ /_yardoc/
5
+ /coverage/
6
+ /doc/
7
+ /pkg/
8
+ /spec/reports/
9
+ /tmp/
10
+ *.bundle
11
+ *.so
12
+ *.o
13
+ *.a
14
+ mkmf.log
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in perobs.gemspec
4
+ gemspec
data/LICENSE.txt ADDED
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2015 Chris Schlaeger
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,113 @@
1
+ # PEROBS - PErsistent Ruby OBject Store
2
+
3
+ PEROBS is a library that provides a persistent object store for Ruby
4
+ objects. Objects of your classes can be made persistent by deriving
5
+ them from PEROBS::Object. They will be in memory when needed and
6
+ transparently stored into a persistent storage. Currently only
7
+ filesystem based storage is supported, but back-ends for key/value
8
+ databases can be easily added.
9
+
10
+ This library is ideal for Ruby applications that work on huge, mostly
11
+ constant data sets and usually handle a small subset of the data at a
12
+ time. To ensure data consistency of a larger data set, you can use
13
+ transactions to make modifications of multiple objects atomic.
14
+ Transactions can be nested and are aborted when an exception is
15
+ raised.
16
+
17
+ ## Usage
18
+
19
+ It features a garbage collector that removes all objects that are no
20
+ longer in use. A build-in cache keeps access latencies to recently
21
+ used objects low and lazily flushes modified objects into the
22
+ persistend back-end.
23
+
24
+ Persistent objects must be created by deriving your class from
25
+ PEROBS::Object. Only instance variables that are declared via
26
+ po_attr will be persistent. All objects that are stored in persitant
27
+ instance variables must provide a to_json method that generates JSON
28
+ syntax that can be parsed into their original object again. It is
29
+ recommended that references to other objects are all going to persistent
30
+ objects again.
31
+
32
+ There are currently 3 kinds of persistent objects available:
33
+
34
+ * PEROBS::Object is the base class for all your classes that should be
35
+ persistent.
36
+
37
+ * PEROBS::Array provides an interface similar to the built-in Array class
38
+ but its objects are automatically stored.
39
+
40
+ * PEROBS::Hash provides an interface similar to the built-in Hash
41
+ class but its objects are automatically stored.
42
+
43
+ In addition to these classes, you also need to create a PEROBS::Store
44
+ object that owns your persistent objects. The store provides the
45
+ persistent database. If you are using the default serializer (JSON),
46
+ you can only use the subset of Ruby types that JSON supports.
47
+ Alternatively, you can use Marshal or YAML which support almost every
48
+ Ruby data type.
49
+
50
+ Here is an example how to use PEROBS. Let's define a class that models
51
+ a person with their family relations.
52
+
53
+ ```
54
+ require 'perobs'
55
+
56
+ class Person < PEROBS::Object
57
+
58
+ po_attr :name, :mother, :father, :kids
59
+
60
+ def initialize(store, name)
61
+ super
62
+ attr_init(:name, name)
63
+ attr_init(:kids, PEROBS::Array.new)
64
+ end
65
+
66
+ def to_s
67
+ "#{@name} is the child of #{self.mother ? self.mother.name : 'unknown'} " +
68
+ "and #{self.father ? self.father.name : 'unknown'}.
69
+ end
70
+
71
+ end
72
+
73
+ store = PEROBS::Store.new('family')
74
+ store['grandpa'] = joe = Person.new('Joe')
75
+ store['grandma'] = jane = Person.new('Jane')
76
+ jim = Person.new('Jim')
77
+ jim.father = joe
78
+ joe.kids << jim
79
+ jim.mother = jane
80
+ jane.kids << jim
81
+ store.sync
82
+ ```
83
+
84
+ When you run this script, a folder named 'family' will be created. It
85
+ contains the 3 Person objects.
86
+
87
+ ## Installation
88
+
89
+ Add this line to your application's Gemfile:
90
+
91
+ ```ruby
92
+ gem 'perobs'
93
+ ```
94
+
95
+ And then execute:
96
+
97
+ $ bundle
98
+
99
+ Or install it yourself as:
100
+
101
+ $ gem install perobs
102
+
103
+ ## Usage
104
+
105
+ TODO: Write usage instructions here
106
+
107
+ ## Contributing
108
+
109
+ 1. Fork it ( https://github.com/scrapper/perobs/fork )
110
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
111
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
112
+ 4. Push to the branch (`git push origin my-new-feature`)
113
+ 5. Create a new Pull Request
data/Rakefile ADDED
@@ -0,0 +1,22 @@
1
+ # Add the lib directory to the search path if it isn't included already
2
+ # lib = File.expand_path('../lib', __FILE__)
3
+ # $:.unshift lib unless $:.include?(lib)
4
+
5
+ require "bundler/gem_tasks"
6
+ require "rspec/core/rake_task"
7
+ require 'rake/clean'
8
+ require 'yard'
9
+ YARD::Rake::YardocTask.new
10
+
11
+ Dir.glob( 'tasks/*.rake').each do |fn|
12
+ begin
13
+ load fn;
14
+ rescue LoadError
15
+ puts "#{fn.split('/')[1]} tasks unavailable: #{$!}"
16
+ end
17
+ end
18
+
19
+ task :default => :spec
20
+ task :test => :spec
21
+
22
+ desc 'Run all unit and spec tests'
@@ -0,0 +1,173 @@
1
+ # encoding: UTF-8
2
+ #
3
+ # = Array.rb -- Persistent Ruby Object Store
4
+ #
5
+ # Copyright (c) 2015 by Chris Schlaeger <chris@taskjuggler.org>
6
+ #
7
+ # MIT License
8
+ #
9
+ # Permission is hereby granted, free of charge, to any person obtaining
10
+ # a copy of this software and associated documentation files (the
11
+ # "Software"), to deal in the Software without restriction, including
12
+ # without limitation the rights to use, copy, modify, merge, publish,
13
+ # distribute, sublicense, and/or sell copies of the Software, and to
14
+ # permit persons to whom the Software is furnished to do so, subject to
15
+ # the following conditions:
16
+ #
17
+ # The above copyright notice and this permission notice shall be
18
+ # included in all copies or substantial portions of the Software.
19
+ #
20
+ # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
21
+ # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
22
+ # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
23
+ # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
24
+ # LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
25
+ # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
26
+ # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
27
+
28
+ require 'perobs/ObjectBase'
29
+
30
+ module PEROBS
31
+
32
+ # An Array that is transparently persisted onto the back-end storage. It is
33
+ # very similar to the Ruby built-in Array class but has some additional
34
+ # limitations. The hash key must always be a String.
35
+ class Array < ObjectBase
36
+
37
+ # Create a new PersistentArray object.
38
+ # @param store [Store] The Store this hash is stored in
39
+ # @param size [Fixnum] The requested size of the Array
40
+ # @param default [Any] The default value that is returned when no value is
41
+ # stored for a specific key.
42
+ def initialize(store, size = 0, default = nil)
43
+ super(store)
44
+ @data = ::Array.new(size, default)
45
+ end
46
+
47
+ # Equivalent to Array::[]
48
+ def [](index)
49
+ _dereferenced(@data[index])
50
+ end
51
+
52
+ # Equivalent to Array::[]=
53
+ def []=(index, obj)
54
+ @data[index] = _referenced(obj)
55
+ @store.cache.cache_write(self)
56
+
57
+ obj
58
+ end
59
+
60
+ # Equivalent to Array::<<
61
+ def <<(obj)
62
+ @store.cache.cache_write(self)
63
+ @data << _referenced(obj)
64
+ end
65
+
66
+ # Equivalent to Array::+
67
+ def +(ary)
68
+ @store.cache.cache_write(self)
69
+ @data + ary
70
+ end
71
+
72
+ # Equivalent to Array::push
73
+ def push(obj)
74
+ @store.cache.cache_write(self)
75
+ @data.push(_referenced(obj))
76
+ end
77
+
78
+ # Equivalent to Array::pop
79
+ def pop
80
+ @store.cache.cache_write(self)
81
+ _dereferenced(@data.pop)
82
+ end
83
+
84
+ # Equivalent to Array::clear
85
+ def clear
86
+ @store.cache.cache_write(self)
87
+ @data.clear
88
+ end
89
+
90
+ # Equivalent to Array::delete
91
+ def delete(obj)
92
+ @store.cache.cache_write(self)
93
+ @data.delete { |v| _dereferenced(v) == obj }
94
+ end
95
+
96
+ # Equivalent to Array::delete_at
97
+ def delete_at(index)
98
+ @store.cache.cache_write(self)
99
+ @data.delete_at(index)
100
+ end
101
+
102
+ # Equivalent to Array::delete_if
103
+ def delete_if
104
+ @data.delete_if do |item|
105
+ yield(_dereferenced(item))
106
+ end
107
+ end
108
+
109
+ # Equivalent to Array::each
110
+ def each
111
+ @data.each do |item|
112
+ yield(_dereferenced(item))
113
+ end
114
+ end
115
+
116
+ # Equivalent to Array::empty?
117
+ def empty?
118
+ @data.empty?
119
+ end
120
+
121
+ # Equivalent to Array::include?
122
+ def include?(obj)
123
+ @data.each { |v| return true if _dereferenced(v) == obj }
124
+
125
+ false
126
+ end
127
+
128
+ # Equivalent to Array::length
129
+ def length
130
+ @data.length
131
+ end
132
+ alias size length
133
+
134
+ # Equivalent to Array::map
135
+ def map
136
+ @data.map do |item|
137
+ yield(_dereferenced(item))
138
+ end
139
+ end
140
+ alias collect map
141
+
142
+ # Return a list of all object IDs of all persistend objects that this Array
143
+ # is referencing.
144
+ # @return [Array of Fixnum or Bignum] IDs of referenced objects
145
+ def _referenced_object_ids
146
+ @data.each.select { |v| v && v.is_a?(POReference) }.map { |o| o.id }
147
+ end
148
+
149
+ # This method should only be used during store repair operations. It will
150
+ # delete all referenced to the given object ID.
151
+ # @param id [Fixnum/Bignum] targeted object ID
152
+ def _delete_reference_to_id(id)
153
+ @data.delete_if { |v| v && v.is_a?(POReference) && v.id == id }
154
+ end
155
+
156
+ # Restore the persistent data from a single data structure.
157
+ # This is a library internal method. Do not use outside of this library.
158
+ # @param data [Array] the actual Array object
159
+ # @private
160
+ def _deserialize(data)
161
+ @data = data
162
+ end
163
+
164
+ private
165
+
166
+ def _serialize
167
+ @data
168
+ end
169
+
170
+ end
171
+
172
+ end
173
+
@@ -0,0 +1,242 @@
1
+ # encoding: UTF-8
2
+ #
3
+ # = BlockDB.rb -- Persistent Ruby Object Store
4
+ #
5
+ # Copyright (c) 2015 by Chris Schlaeger <chris@taskjuggler.org>
6
+ #
7
+ # MIT License
8
+ #
9
+ # Permission is hereby granted, free of charge, to any person obtaining
10
+ # a copy of this software and associated documentation files (the
11
+ # "Software"), to deal in the Software without restriction, including
12
+ # without limitation the rights to use, copy, modify, merge, publish,
13
+ # distribute, sublicense, and/or sell copies of the Software, and to
14
+ # permit persons to whom the Software is furnished to do so, subject to
15
+ # the following conditions:
16
+ #
17
+ # The above copyright notice and this permission notice shall be
18
+ # included in all copies or substantial portions of the Software.
19
+ #
20
+ # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
21
+ # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
22
+ # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
23
+ # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
24
+ # LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
25
+ # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
26
+ # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
27
+
28
+ require 'json'
29
+ require 'json/add/core'
30
+ require 'json/add/struct'
31
+
32
+ module PEROBS
33
+
34
+ # This class manages the usage of the data blocks in the corresponding
35
+ # HashedBlocks object.
36
+ class BlockDB
37
+
38
+ # Create a new BlockDB object.
39
+ def initialize(dir, block_size)
40
+ @dir = dir
41
+ @block_size = block_size
42
+
43
+ @index_file_name = File.join(dir, 'index.json')
44
+ @block_file_name = File.join(dir, 'data')
45
+ read_index
46
+ end
47
+
48
+ # Write the given bytes with the given ID into the DB.
49
+ # @param id [Fixnum or Bignum] ID
50
+ # @param raw [String] sequence of bytes
51
+ def write_object(id, raw)
52
+ bytes = raw.bytesize
53
+ start_address = reserve_blocks(id, bytes)
54
+ if write_to_block_file(raw, start_address) != bytes
55
+ raise RuntimeError, 'Object length does not match written bytes'
56
+ end
57
+ write_index
58
+ end
59
+
60
+ # Read the entry for the given ID and return it as bytes.
61
+ # @param id [Fixnum or Bignum] ID
62
+ # @return [String] sequence of bytes
63
+ def read_object(id)
64
+ read_from_block_file(*find(id))
65
+ end
66
+
67
+
68
+ # Find the data for the object with given id.
69
+ # @param id [Fixnum or Bignum] Object ID
70
+ # @return [Array] Returns an Array with two Fixnum entries. The first is
71
+ # the number of bytes and the second is the starting offset in the
72
+ # block storage file.
73
+ def find(id)
74
+ @entries.each do |entry|
75
+ if entry['id'] == id
76
+ return [ entry['bytes'], entry['first_block'] * @block_size ]
77
+ end
78
+ end
79
+
80
+ nil
81
+ end
82
+
83
+ # Write a string of bytes into the file at the given address.
84
+ # @param raw [String] bytes to write
85
+ # @param address [Fixnum] offset in the file
86
+ # @return [Fixnum] number of bytes written
87
+ def write_to_block_file(raw, address)
88
+ begin
89
+ File.write(@block_file_name, raw, address)
90
+ rescue => e
91
+ raise IOError,
92
+ "Cannot write block file #{@block_file_name}: #{e.message}"
93
+ end
94
+ end
95
+
96
+ # Read _bytes_ bytes from the file starting at offset _address_.
97
+ # @param bytes [Fixnum] number of bytes to read
98
+ # @param address [Fixnum] offset in the file
99
+ def read_from_block_file(bytes, address)
100
+ begin
101
+ File.read(@block_file_name, bytes, address)
102
+ rescue => e
103
+ raise IOError,
104
+ "Cannot read block file #{@block_file_name}: #{e.message}"
105
+ end
106
+ end
107
+
108
+ # Clear the mark on all entries in the index.
109
+ def clear_marks
110
+ @entries.each { |e| e['marked'] = false}
111
+ write_index
112
+ end
113
+
114
+ # Set a mark on the entry with the given ID.
115
+ # @param id [Fixnum or Bignum] ID of the entry
116
+ def mark(id)
117
+ found = false
118
+ @entries.each do |entry|
119
+ if entry['id'] == id
120
+ entry['marked'] = true
121
+ found = true
122
+ break
123
+ end
124
+ end
125
+
126
+ unless found
127
+ raise ArgumentError, "Cannot find an entry for ID #{id} to mark"
128
+ end
129
+
130
+ write_index
131
+ end
132
+
133
+ # Check if the entry for a given ID is marked.
134
+ # @param id [Fixnum or Bignum] ID of the entry
135
+ # @return [TrueClass or FalseClass] true if marked, false otherwise
136
+ def is_marked?(id)
137
+ @entries.each do |entry|
138
+ return entry['marked'] if entry['id'] == id
139
+ end
140
+
141
+ raise ArgumentError, "Cannot find an entry for ID #{id} to check"
142
+ end
143
+
144
+ # Remove all entries from the index that have not been marked.
145
+ def delete_unmarked_entries
146
+ @entries.delete_if { |e| e['marked'] == false }
147
+ write_index
148
+ end
149
+
150
+ private
151
+
152
+ # Reserve the blocks needed for the specified number of bytes with the
153
+ # given ID.
154
+ # @param id [Fixnum or Bignum] ID of the entry
155
+ # @param bytes [Fixnum] number of bytes for this entry
156
+ # @return [Fixnum] the start address of the reserved block
157
+ def reserve_blocks(id, bytes)
158
+ # size of the entry in blocks
159
+ blocks = size_in_blocks(bytes)
160
+ # index of first block after the last seen entry
161
+ end_of_last_entry = 0
162
+ # block index of best fit segment
163
+ best_fit_start = nil
164
+ # best fir segment size in blocks
165
+ best_fit_blocks = nil
166
+ # If there is already an entry for an object with the _id_, we mark it
167
+ # for deletion.
168
+ entry_to_delete = nil
169
+
170
+ @entries.each do |entry|
171
+ if entry['id'] == id
172
+ # We've found an old entry for this ID.
173
+ if entry['blocks'] >= blocks
174
+ # The old entry still fits. Let's just reuse it.
175
+ entry['bytes'] = bytes
176
+ entry['blocks'] = blocks
177
+ return entry['first_block'] * @block_size
178
+ end
179
+ # It does not fit. Ignore the entry and mark it for deletion.
180
+ entry_to_delete = entry
181
+ next
182
+ end
183
+
184
+ gap = entry['first_block'] - end_of_last_entry
185
+ if gap >= blocks &&
186
+ (best_fit_blocks.nil? || gap < best_fit_blocks)
187
+ # We've found a segment that fits the requested bytes and fits
188
+ # better than any previous find.
189
+ best_fit_start = end_of_last_entry
190
+ best_fit_blocks = gap
191
+ end
192
+ end_of_last_entry = entry['first_block'] + entry['blocks']
193
+ end
194
+
195
+ # Delete the old entry if requested.
196
+ @entries.delete(entry_to_delete) if entry_to_delete
197
+
198
+ # Create a new entry and insert it.
199
+ entry = {
200
+ 'id' => id,
201
+ 'bytes' => bytes,
202
+ 'first_block' => best_fit_start || end_of_last_entry,
203
+ 'blocks' => blocks,
204
+ 'marked' => false
205
+ }
206
+ @entries << entry
207
+ @entries.sort! { |e1, e2| e1['first_block'] <=> e2['first_block'] }
208
+
209
+ entry['first_block'] * @block_size
210
+ end
211
+
212
+ def read_index
213
+ if File.exists?(@index_file_name)
214
+ begin
215
+ @entries = JSON.parse(File.read(@index_file_name))
216
+ rescue => e
217
+ raise RuntimeError,
218
+ "BlockDB file #{@index_file_name} corrupted: #{e.message}"
219
+ end
220
+ else
221
+ @entries = []
222
+ end
223
+ end
224
+
225
+ def write_index
226
+ begin
227
+ File.write(@index_file_name, @entries.to_json)
228
+ rescue => e
229
+ raise RuntimeError,
230
+ "Cannot write BlockDB index file #{@index_file_name}: " +
231
+ e.message
232
+ end
233
+ end
234
+
235
+ def size_in_blocks(bytes)
236
+ bytes / @block_size + (bytes % @block_size != 0 ? 1 : 0)
237
+ end
238
+
239
+ end
240
+
241
+ end
242
+