msgpack 0.4.7 → 0.5.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (60) hide show
  1. data/.gitignore +17 -0
  2. data/ChangeLog +47 -0
  3. data/README.rdoc +102 -0
  4. data/Rakefile +88 -0
  5. data/doclib/msgpack.rb +55 -0
  6. data/doclib/msgpack/buffer.rb +193 -0
  7. data/doclib/msgpack/core_ext.rb +101 -0
  8. data/doclib/msgpack/error.rb +14 -0
  9. data/doclib/msgpack/packer.rb +131 -0
  10. data/doclib/msgpack/unpacker.rb +130 -0
  11. data/ext/msgpack/buffer.c +679 -0
  12. data/ext/msgpack/buffer.h +442 -0
  13. data/ext/msgpack/buffer_class.c +507 -0
  14. data/ext/msgpack/buffer_class.h +32 -0
  15. data/ext/msgpack/compat.h +112 -0
  16. data/ext/msgpack/core_ext.c +129 -0
  17. data/ext/{pack.h → msgpack/core_ext.h} +7 -7
  18. data/ext/msgpack/extconf.rb +17 -0
  19. data/ext/msgpack/packer.c +137 -0
  20. data/ext/msgpack/packer.h +319 -0
  21. data/ext/msgpack/packer_class.c +285 -0
  22. data/ext/{unpack.h → msgpack/packer_class.h} +11 -7
  23. data/ext/msgpack/rbinit.c +33 -0
  24. data/ext/msgpack/rmem.c +110 -0
  25. data/ext/msgpack/rmem.h +100 -0
  26. data/ext/msgpack/sysdep.h +115 -0
  27. data/ext/msgpack/sysdep_endian.h +50 -0
  28. data/ext/msgpack/sysdep_types.h +46 -0
  29. data/ext/msgpack/unpacker.c +669 -0
  30. data/ext/msgpack/unpacker.h +112 -0
  31. data/ext/msgpack/unpacker_class.c +376 -0
  32. data/{msgpack/pack_define.h → ext/msgpack/unpacker_class.h} +12 -8
  33. data/lib/msgpack.rb +10 -0
  34. data/{ext → lib/msgpack}/version.rb +1 -1
  35. data/msgpack.gemspec +25 -0
  36. data/spec/buffer_io_spec.rb +237 -0
  37. data/spec/buffer_spec.rb +572 -0
  38. data/{test → spec}/cases.json +0 -0
  39. data/{test/cases.mpac → spec/cases.msg} +0 -0
  40. data/{test/cases_compact.mpac → spec/cases_compact.msg} +0 -0
  41. data/spec/cases_spec.rb +39 -0
  42. data/spec/format_spec.rb +225 -0
  43. data/spec/packer_spec.rb +127 -0
  44. data/spec/random_compat.rb +24 -0
  45. data/spec/spec_helper.rb +21 -0
  46. data/spec/unpacker_spec.rb +128 -0
  47. metadata +171 -34
  48. data/ext/compat.h +0 -99
  49. data/ext/extconf.rb +0 -7
  50. data/ext/pack.c +0 -314
  51. data/ext/rbinit.c +0 -66
  52. data/ext/unpack.c +0 -1001
  53. data/msgpack/pack_template.h +0 -771
  54. data/msgpack/sysdep.h +0 -195
  55. data/msgpack/unpack_define.h +0 -93
  56. data/msgpack/unpack_template.h +0 -413
  57. data/test/test_cases.rb +0 -46
  58. data/test/test_encoding.rb +0 -68
  59. data/test/test_helper.rb +0 -10
  60. data/test/test_pack_unpack.rb +0 -308
@@ -0,0 +1,17 @@
1
+ *.o
2
+ *.so
3
+ *.gem
4
+ *.class
5
+ .bundle
6
+ Gemfile*
7
+ pkg
8
+ test/debug.log
9
+ *~
10
+ /rdoc
11
+ tmp
12
+ .classpath
13
+ .project
14
+ .settings
15
+ /nbproject/private/
16
+ ext/java/build
17
+ ext/java/msgpack.jar
@@ -0,0 +1,47 @@
1
+
2
+ 2012-12-20 version 0.5.0:
3
+
4
+ * Rewrote all code and improved performance significantly
5
+ * Added MessagePack::Buffer class
6
+ * Added MessagePack::Packer class
7
+ * Added Packer#buffer and Unpacker#buffer accessors which return MessagePack::Buffer
8
+ * Added Packer#write_{array,map}_header and Unpacker#read_{array,map}_header methods
9
+ * Added Packer#write_nil and Unpacker#skip_nil methods
10
+ * Added Packer#write -> #pack alias and Unpacker#read method
11
+ * Added exception classes - UnpackError, MalformedFormatError, StackError and TypeError
12
+ * Added MessagePack.dup -> .pack and MessagePack.load -> .unpack aliases
13
+ * Added Packer#empty?, #size and #clear methods
14
+ * Added Packer#write_to(io) method to flush serialized data to IO efficiently
15
+ * Added Unpacker#skip method to skip an object efficiently
16
+ * Removed obsoleted Unpacker#fill, #execute, #execute_limit, #finished? and #data methods
17
+ * Removed obsoleted Unapcker#stream and #stream= methods. Use unpacker.buffer.io instead
18
+
19
+
20
+ 2012-05-05 version 0.4.7:
21
+
22
+ * Fixed serialization of double values on ARM OABI architectures
23
+ * Fixed byteorder problem on big-endian platforms
24
+ * Don't use MRI internals in the Ruby extension for Rubinius
25
+ * Detect whether st.h is present and don't use RUBY_VM as the condition for
26
+ Rubinius
27
+
28
+ 2011-08-08 version 0.4.6:
29
+
30
+ * Fixed compile error problem on Mac OS X Lion
31
+
32
+ 2011-05-09 version 0.4.5:
33
+
34
+ * Improves compatibility with JRuby
35
+
36
+ 2010-11-28 version 0.4.4:
37
+
38
+ * Adds Unpacker#feed_each method
39
+ * Improves compatibility with Rubinius
40
+ * Improves compatibility with ruby-1.8.5
41
+ * Encodings of String instances to UTF-8 on Ruby 1.9
42
+
43
+ 2010-06-29 version 0.4.3:
44
+
45
+ * Adds MessagePack::VERSION constant
46
+ * Fixes SEGV problem caused by GC bug at MessagePack_Unpacker_mark
47
+
@@ -0,0 +1,102 @@
1
+
2
+ = MessagePack
3
+
4
+ MessagePack[http://msgpack.org] is an efficient binary serialization format.
5
+ It lets you exchange data among multiple languages like JSON but it's faster and smaller.
6
+ For example, small integers (like flags or error code) are encoded into a single byte,
7
+ and typical short strings only require an extra byte in addition to the strings themselves.
8
+
9
+ If you ever wished to use JSON for convenience (storing an image with metadata) but could
10
+ not for technical reasons (binary data, size, speed...), MessagePack is a perfect replacement.
11
+
12
+ require 'msgpack'
13
+ msg = [1,2,3].to_msgpack #=> "\x93\x01\x02\x03"
14
+ MessagePack.unpack(msg) #=> [1,2,3]
15
+
16
+ Use RubyGems to install:
17
+
18
+ gem install msgpack
19
+
20
+
21
+ = Use cases
22
+
23
+ * Store objects efficiently serialized by msgpack on memcached or Redis
24
+ * In fact Redis supports msgpack in EVAL-scripts[http://redis.io/commands/eval]
25
+ * Upload data in efficient format from mobile devices such as smartphones
26
+ * MessagePack works on iPhone/iPad and Android. See also Objective-C[https://github.com/msgpack/msgpack-objectivec] and Java[https://github.com/msgpack/msgpack-java] implementations
27
+ * Design a portable protocol to communicate with embedded devices
28
+ * Check also Fluentd[http://fluentd.org/] which is a log collector which uses msgpack for the log format (they say it uses JSON but actually it's msgpack, which is compatible with JSON)
29
+ * Exchange objects between software components written in different languages
30
+ * You'll need a flexible but efficient format so that components exchange objects while keeping compatibility
31
+
32
+
33
+ = Portability
34
+
35
+ MessagePack for Ruby should run on x86, ARM, PowerPC, SPARC and other CPU architectures.
36
+
37
+ And it works with MRI (CRuby) and Rubinius.
38
+ Patches to improve portability is highly welcomed.
39
+
40
+
41
+ = Serializing objects
42
+
43
+ Use *MessagePack.pack* or *to_msgpack*:
44
+
45
+ require 'msgpack'
46
+ msg = MessagePack.pack(obj) # or
47
+ msg = obj.to_msgpack
48
+
49
+ == Streaming serialization
50
+
51
+ Packer provides advanced API to serialize objects in streaming style:
52
+
53
+ # serialize a 2-element array [e1, e2]
54
+ pk = MessagePack::Packer.new(io)
55
+ pk.write_array_header(2).write(e1).write(e2).flush
56
+
57
+ See API reference for details.
58
+
59
+ = Deserializing objects
60
+
61
+ Use *MessagePack.unpack*:
62
+
63
+ require 'msgpack'
64
+ obj = MessagePack.unpack(msg)
65
+
66
+ == Streaming deserialization
67
+
68
+ Unpacker provides advanced API to deserialize objects in streaming style:
69
+
70
+ # deserialize objects from an IO
71
+ u = MessagePack::Unpacker.new(io)
72
+ u.each do |obj|
73
+ # ...
74
+ end
75
+
76
+ or event-driven style which works well with EventMachine:
77
+
78
+ # event-driven deserialization
79
+ def on_read(data)
80
+ @u ||= MessagePack::Unpacker.new
81
+ @u.feed_each(data) {|obj|
82
+ # ...
83
+ }
84
+ end
85
+
86
+ See API reference for details.
87
+
88
+
89
+ = Buffer API
90
+
91
+ MessagePack for Ruby provides a buffer API so that you can read or write data by hand, not via Packer or Unpacker API.
92
+
93
+ This MessagePack::Buffer API is backed with a fixed-length shared memory pool which is very fast for small data (<= 4KB),
94
+ and has zero-copy capability which significantly affects performance to handle large binary data.
95
+
96
+
97
+ = Copyright
98
+
99
+ Author:: FURUHASHI Sadayuki <frsyuki@gmail.com>
100
+ Copyright:: Copyright (c) 2008-2012 FURUHASHI Sadayuki
101
+ License:: Apache License, Version 2.0
102
+
@@ -0,0 +1,88 @@
1
+
2
+ require 'bundler'
3
+ Bundler::GemHelper.install_tasks
4
+
5
+ require 'fileutils'
6
+
7
+ require 'rspec/core'
8
+ require 'rspec/core/rake_task'
9
+ require 'yard'
10
+
11
+ RSpec::Core::RakeTask.new(:spec) do |t|
12
+ t.rspec_opts = ["-c", "-f progress"]
13
+ t.rspec_opts << "-Ilib"
14
+ t.pattern = 'spec/**/*_spec.rb'
15
+ t.verbose = true
16
+ end
17
+
18
+ task :spec => :compile
19
+
20
+ desc 'Run RSpec code examples and measure coverage'
21
+ task :coverage do |t|
22
+ ENV['SIMPLE_COV'] = '1'
23
+ Rake::Task["spec"].invoke
24
+ end
25
+
26
+ desc 'Generate YARD document'
27
+ YARD::Rake::YardocTask.new(:doc) do |t|
28
+ t.files = ['lib/msgpack/version.rb','doclib/**/*.rb']
29
+ t.options = []
30
+ t.options << '--debug' << '--verbose' if $trace
31
+ end
32
+
33
+ spec = eval File.read("msgpack.gemspec")
34
+
35
+ if RUBY_PLATFORM =~ /java/
36
+ require 'rake/javaextensiontask'
37
+
38
+ Rake::JavaExtensionTask.new('msgpack', spec) do |ext|
39
+ ext.ext_dir = 'ext/java'
40
+ #jruby_home = RbConfig::CONFIG['prefix']
41
+ #jars = ["#{jruby_home}/lib/jruby.jar"] + FileList['lib/*.jar']
42
+ #ext.classpath = jars.map { |x| File.expand_path x }.join ':'
43
+ end
44
+
45
+ else
46
+ require 'rake/extensiontask'
47
+
48
+ Rake::ExtensionTask.new('msgpack', spec) do |ext|
49
+ ext.cross_compile = true
50
+ ext.lib_dir = File.join(*['lib', 'msgpack', ENV['FAT_DIR']].compact)
51
+ #ext.cross_platform = 'i386-mswin32'
52
+ end
53
+ end
54
+
55
+ task :default => :build
56
+
57
+
58
+ ###
59
+ ## Cross compile memo
60
+ ##
61
+ ## Ubuntu Ubuntu 10.04.1 LTS
62
+ ##
63
+ #
64
+ ### install mingw32 cross compiler
65
+ # sudo apt-get install gcc-mingw32
66
+ #
67
+ ### install rbenv
68
+ # git clone git://github.com/sstephenson/rbenv.git ~/.rbenv
69
+ # echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
70
+ # echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
71
+ # exec $SHELL -l
72
+ #
73
+ ### install cross-compiled ruby 1.9.3
74
+ # rbenv install 1.9.3-p327
75
+ # gem install rake-compiler
76
+ # rake-compiler cross-ruby VERSION=1.9.3-p327
77
+ #
78
+ ### install cross-compiled ruby 1.8.7
79
+ # rbenv install 1.8.7-p371
80
+ # gem install rake-compiler
81
+ # rake-compiler cross-ruby VERSION=1.8.7-p371
82
+ #
83
+ ### build gem
84
+ # rbenv shell 1.8.7-p371
85
+ # gem install bundler && bundle
86
+ # rake cross native gem RUBY_CC_VERSION=1.8.7:1.9.3
87
+ #
88
+
@@ -0,0 +1,55 @@
1
+
2
+ module MessagePack
3
+ #
4
+ # Serializes an object into an IO or String.
5
+ #
6
+ # @overload dump(obj)
7
+ # @return [String] serialized data
8
+ #
9
+ # @overload dump(obj, io)
10
+ # @return [IO]
11
+ #
12
+ def self.dump(arg)
13
+ end
14
+
15
+ #
16
+ # Serializes an object into an IO or String. Alias of dump.
17
+ #
18
+ # @overload dump(obj)
19
+ # @return [String] serialized data
20
+ #
21
+ # @overload dump(obj, io)
22
+ # @return [IO]
23
+ #
24
+ def self.pack(arg)
25
+ end
26
+
27
+ #
28
+ # Deserializes an object from an IO or String.
29
+ #
30
+ # @overload load(string)
31
+ # @param string [String] data to deserialize
32
+ #
33
+ # @overload load(io)
34
+ # @param io [IO]
35
+ #
36
+ # @return [Object] deserialized object
37
+ #
38
+ def self.load(arg)
39
+ end
40
+
41
+ #
42
+ # Deserializes an object from an IO or String. Alias of load.
43
+ #
44
+ # @overload unpack(string)
45
+ # @param string [String] data to deserialize
46
+ #
47
+ # @overload unpack(io)
48
+ # @param io [IO]
49
+ #
50
+ # @return [Object] deserialized object
51
+ #
52
+ def self.unpack(arg)
53
+ end
54
+ end
55
+
@@ -0,0 +1,193 @@
1
+ module MessagePack
2
+
3
+ class Buffer
4
+ #
5
+ # Creates a MessagePack::Buffer instance.
6
+ #
7
+ # @overload initialize(options={})
8
+ # @param options [Hash]
9
+ #
10
+ # @overload initialize(io, options={})
11
+ # @param io [IO]
12
+ # @param options [Hash]
13
+ # This buffer writes written data into the IO when it is filled.
14
+ # This buffer reads data from the IO when it is empty.
15
+ #
16
+ # _io_ must respond to readpartial(length, [,string]) or read(string) method and
17
+ # write(string) or append(string) method.
18
+ #
19
+ # Supported options:
20
+ #
21
+ # * *:io_buffer_size* buffer size to read data from the internal IO. (default: 32768)
22
+ # * *:read_reference_threshold* the threshold size to enable zero-copy deserialize optimization. Read strings longer than this threshold will refer the original string instead of copying it. (default: 256) (supported in MRI only)
23
+ # * *:write_reference_threshold* the threshold size to enable zero-copy serialize optimization. The buffer refers written strings longer than this threshold instead of copying it. (default: 524288) (supported in MRI only)
24
+ #
25
+ def initialize(*args)
26
+ end
27
+
28
+ #
29
+ # Makes the buffer empty
30
+ #
31
+ # @return nil
32
+ #
33
+ def clear
34
+ end
35
+
36
+ #
37
+ # Returns byte size of the buffer.
38
+ #
39
+ # @return nil
40
+ #
41
+ def size
42
+ end
43
+
44
+ #
45
+ # Returns _true_ if the buffer is empty.
46
+ # This method is slightly faster than _size_.
47
+ #
48
+ # @return [Boolean]
49
+ #
50
+ def empty?
51
+ end
52
+
53
+ #
54
+ # Appends the given data to the buffer.
55
+ #
56
+ # @param data [String]
57
+ # @return [Integer] byte size written
58
+ #
59
+ def write(data)
60
+ end
61
+
62
+ #
63
+ # Appends the given data to the buffer.
64
+ #
65
+ # @param data [String]
66
+ # @return [Buffer] self
67
+ #
68
+ def <<(data)
69
+ end
70
+
71
+ #
72
+ # Consumes _n_ bytes from the head of the buffer and returns consumed data.
73
+ # If the size of the buffer is less than _n_, it reads all of data in the buffer.
74
+ #
75
+ # If _n_ is 0, it does nothing and returns an empty string.
76
+ # If the optional _buffer_ argument is given, the content of the string will be replaced with the consumed data.
77
+ #
78
+ # @overload read
79
+ #
80
+ # @overload read(n)
81
+ # @param n [Integer] bytes to read
82
+ #
83
+ # @overload read(n, buffer)
84
+ # @param n [Integer] bytes to read
85
+ # @param buffer [String] buffer to read into
86
+ #
87
+ # @return [String]
88
+ #
89
+ def read(n)
90
+ end
91
+
92
+ #
93
+ # Consumes _n_ bytes from the head of the buffer and returns consumed data.
94
+ # If the size of the buffer is less than _n_, it does nothing and raises EOFError.
95
+ #
96
+ # If _n_ is 0, it does nothing and returns an empty string.
97
+ # If the optional _buffer_ argument is given, the content of the string will be replaced with the consumed data.
98
+ #
99
+ # @overload read_all
100
+ #
101
+ # @overload read_all(n)
102
+ # @param n [Integer] bytes to read
103
+ #
104
+ # @overload read_all(n, buffer)
105
+ # @param n [Integer] bytes to read
106
+ # @param buffer [String] buffer to read into
107
+ #
108
+ # @return [String]
109
+ #
110
+ def read_all(n, buffer=nil)
111
+ end
112
+
113
+ #
114
+ # Consumes _n_ bytes from the head of the buffer.
115
+ # If the size of the buffer is less than _n_, it skips all of data in the buffer and returns integer less than _n_.
116
+ #
117
+ # If _n_ is 0, it does nothing and returns _0_.
118
+ #
119
+ # @param n [Integer] byte size to skip
120
+ # @return [Integer] byte size actually skipped
121
+ #
122
+ def skip(n)
123
+ end
124
+
125
+ #
126
+ # Consumes _n_ bytes from the head of the buffer.
127
+ # If the size of the buffer is less than _n_, it does nothing and raises EOFError.
128
+ # If _n_ is 0, it does nothing.
129
+ #
130
+ # @param n [Integer] byte size to skip
131
+ # @return [Buffer] self
132
+ #
133
+ def skip_all(n)
134
+ end
135
+
136
+ #
137
+ # Returns all data in the buffer as a string.
138
+ # Destructive update to the returned string does NOT effect the buffer.
139
+ #
140
+ # @return [String]
141
+ #
142
+ def to_str
143
+ end
144
+
145
+ #
146
+ # Returns content of the buffer as an array of strings.
147
+ #
148
+ # This method is sometimes faster than to_s because the internal
149
+ # structure of the buffer is a queue of buffer chunks.
150
+ #
151
+ # @return [Array] array of strings
152
+ #
153
+ def to_a
154
+ end
155
+
156
+ #
157
+ # Internal io
158
+ #
159
+ # @return IO
160
+ #
161
+ attr_reader :io
162
+
163
+ #
164
+ # Flushes data in the internal buffer to the internal IO.
165
+ # If internal IO is not set, it does nothing.
166
+ #
167
+ # @return [Buffer] self
168
+ #
169
+ def flush
170
+ end
171
+
172
+ #
173
+ # Closes internal IO if its set.
174
+ # If internal IO is not set, it does nothing
175
+ #
176
+ # @return nil
177
+ #
178
+ def close
179
+ end
180
+
181
+ #
182
+ # Writes all of data in the internal buffer into the given IO.
183
+ # This method consumes and removes data from the internal buffer.
184
+ # _io_ must respond to write(data) method.
185
+ #
186
+ # @param io [IO]
187
+ # @return [Integer] byte size of written data
188
+ #
189
+ def write_to(io)
190
+ end
191
+ end
192
+
193
+ end