maxmind-db 1.1.0 → 1.1.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/CHANGELOG.md +5 -0
- data/Gemfile +1 -4
- data/README.dev.md +6 -2
- data/README.md +1 -9
- data/lib/maxmind/db.rb +38 -34
- data/lib/maxmind/db/decoder.rb +4 -2
- data/lib/maxmind/db/errors.rb +1 -1
- data/lib/maxmind/db/file_reader.rb +3 -2
- data/lib/maxmind/db/memory_reader.rb +4 -4
- data/lib/maxmind/db/metadata.rb +28 -2
- data/maxmind-db.gemspec +7 -2
- data/test/data/LICENSE +4 -0
- data/test/data/MaxMind-DB-spec.md +570 -0
- data/test/data/MaxMind-DB-test-metadata-pointers.mmdb +0 -0
- data/test/data/README.md +4 -0
- data/test/data/bad-data/README.md +7 -0
- data/test/data/bad-data/libmaxminddb/libmaxminddb-offset-integer-overflow.mmdb +0 -0
- data/test/data/bad-data/maxminddb-golang/cyclic-data-structure.mmdb +0 -0
- data/test/data/bad-data/maxminddb-golang/invalid-bytes-length.mmdb +1 -0
- data/test/data/bad-data/maxminddb-golang/invalid-data-record-offset.mmdb +0 -0
- data/test/data/bad-data/maxminddb-golang/invalid-map-key-length.mmdb +0 -0
- data/test/data/bad-data/maxminddb-golang/invalid-string-length.mmdb +1 -0
- data/test/data/bad-data/maxminddb-golang/metadata-is-an-uint128.mmdb +1 -0
- data/test/data/bad-data/maxminddb-golang/unexpected-bytes.mmdb +0 -0
- data/test/data/perltidyrc +12 -0
- data/test/data/source-data/GeoIP2-Anonymous-IP-Test.json +48 -0
- data/test/data/source-data/GeoIP2-City-Test.json +12852 -0
- data/test/data/source-data/GeoIP2-Connection-Type-Test.json +102 -0
- data/test/data/source-data/GeoIP2-Country-Test.json +15916 -0
- data/test/data/source-data/GeoIP2-DensityIncome-Test.json +14 -0
- data/test/data/source-data/GeoIP2-Domain-Test.json +452 -0
- data/test/data/source-data/GeoIP2-Enterprise-Test.json +673 -0
- data/test/data/source-data/GeoIP2-ISP-Test.json +12593 -0
- data/test/data/source-data/GeoIP2-Precision-Enterprise-Test.json +2018 -0
- data/test/data/source-data/GeoIP2-Static-IP-Score-Test.json +2132 -0
- data/test/data/source-data/GeoIP2-User-Count-Test.json +2837 -0
- data/test/data/source-data/GeoLite2-ASN-Test.json +37 -0
- data/test/data/source-data/README +15 -0
- data/test/data/test-data/GeoIP2-Anonymous-IP-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-City-Test-Broken-Double-Format.mmdb +0 -0
- data/test/data/test-data/GeoIP2-City-Test-Invalid-Node-Count.mmdb +0 -0
- data/test/data/test-data/GeoIP2-City-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-Connection-Type-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-Country-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-DensityIncome-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-Domain-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-Enterprise-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-ISP-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-Precision-Enterprise-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-Static-IP-Score-Test.mmdb +0 -0
- data/test/data/test-data/GeoIP2-User-Count-Test.mmdb +0 -0
- data/test/data/test-data/GeoLite2-ASN-Test.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-no-ipv4-search-tree.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-string-value-entries.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-broken-pointers-24.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-broken-search-tree-24.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-decoder.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-ipv4-24.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-ipv4-28.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-ipv4-32.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-ipv6-24.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-ipv6-28.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-ipv6-32.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-metadata-pointers.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-mixed-24.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-mixed-28.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-mixed-32.mmdb +0 -0
- data/test/data/test-data/MaxMind-DB-test-nested.mmdb +0 -0
- data/test/data/test-data/README.md +26 -0
- data/test/data/test-data/maps-with-pointers.raw +0 -0
- data/test/data/test-data/write-test-data.pl +640 -0
- data/test/data/tidyall.ini +5 -0
- data/test/mmdb_util.rb +1 -1
- data/test/test_decoder.rb +1 -1
- data/test/test_reader.rb +14 -1
- metadata +122 -6
- data/Gemfile.lock +0 -34
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 485d10049eaba35df9d49f7aa4637857a3fe84f38dec3e2a163c675fc8f2e474
|
4
|
+
data.tar.gz: 964008df8d8bc5bc8c558fb004cbb7eda6843295cb3abd5fea3fc193da58981d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 92181b893b438f63a874d915817f4e325aa4a20fb12e8e373e6d6f4885235472353391db4508f313f7d4b5b9ab33e6f09cd05a5b88e718b59dc352dc18f415be
|
7
|
+
data.tar.gz: a03201ffac72cac0228545ce2f1a3a5e3d84ad807d33fac18bfffd3da4a5be69d0bceee96c5c87429b394410ed3e3efc17f20893700b7811558c7a383d4773e4
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,10 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## 1.1.1 (2020-06-23)
|
4
|
+
|
5
|
+
* Fixed the memory reader's inspect method to no longer attempt to modify a
|
6
|
+
frozen string. Pull request by Tietew. GitHub #35.
|
7
|
+
|
3
8
|
## 1.1.0 (2020-01-08)
|
4
9
|
|
5
10
|
* The method `get_with_prefix_length` was added. This method returns both
|
data/Gemfile
CHANGED
data/README.dev.md
CHANGED
@@ -1,11 +1,15 @@
|
|
1
1
|
# How to release
|
2
|
-
*
|
3
|
-
*
|
2
|
+
* Ensure tests pass: `rake`
|
3
|
+
* Update changelog: Set version and release date
|
4
|
+
* Set version in `maxmind-db.gemspec`
|
5
|
+
* Add them: `git add -p`
|
4
6
|
* Commit: `git commit -m v1.0.0`
|
5
7
|
* Tag: `git tag -a v1.0.0 -m v1.0.0`
|
6
8
|
* Clean up to be sure nothing stray gets into gem: `git clean -dxff`
|
7
9
|
* Create `.gem` file: `gem build maxmind-db.gemspec`
|
8
10
|
* Complete prerequisites (see below)
|
11
|
+
* You only need to do this if `~/.gem/credentials` is missing
|
12
|
+
`:rubygems_api_key`.
|
9
13
|
* Upload to rubygems.org: `gem push maxmind-db-1.0.0.gem`
|
10
14
|
* Push: `git push`
|
11
15
|
* Push tag: `git push --tags`
|
data/README.md
CHANGED
@@ -13,13 +13,6 @@ file format that stores data indexed by IP address subnets (IPv4 or IPv6).
|
|
13
13
|
gem install maxmind-db
|
14
14
|
```
|
15
15
|
|
16
|
-
Or from source:
|
17
|
-
|
18
|
-
```
|
19
|
-
gem build maxmind-db.spec
|
20
|
-
gem install ./maxmind-db-xxx.gem
|
21
|
-
```
|
22
|
-
|
23
16
|
## Usage
|
24
17
|
|
25
18
|
```ruby
|
@@ -43,8 +36,7 @@ For more information see the
|
|
43
36
|
|
44
37
|
## Requirements
|
45
38
|
|
46
|
-
This code requires Ruby version 2.4 or higher.
|
47
|
-
are not supported.
|
39
|
+
This code requires Ruby version 2.4 or higher.
|
48
40
|
|
49
41
|
## Contributing
|
50
42
|
|
data/lib/maxmind/db.rb
CHANGED
@@ -7,7 +7,7 @@ require 'maxmind/db/file_reader.rb'
|
|
7
7
|
require 'maxmind/db/memory_reader.rb'
|
8
8
|
require 'maxmind/db/metadata.rb'
|
9
9
|
|
10
|
-
module MaxMind
|
10
|
+
module MaxMind
|
11
11
|
# DB provides a way to read {MaxMind DB
|
12
12
|
# files}[https://maxmind.github.io/MaxMind-DB/].
|
13
13
|
#
|
@@ -31,11 +31,6 @@ module MaxMind # :nodoc:
|
|
31
31
|
# end
|
32
32
|
#
|
33
33
|
# reader.close
|
34
|
-
#
|
35
|
-
# == Exceptions
|
36
|
-
#
|
37
|
-
# DB raises an InvalidDatabaseError if the database is corrupt or invalid. It
|
38
|
-
# can raise other exceptions, such as ArgumentError, if other errors occur.
|
39
34
|
class DB
|
40
35
|
# Choose the default method to open the database. Currently the default is
|
41
36
|
# MODE_FILE.
|
@@ -47,7 +42,9 @@ module MaxMind # :nodoc:
|
|
47
42
|
MODE_MEMORY = :MODE_MEMORY
|
48
43
|
# Treat the database parameter as containing a database already read into
|
49
44
|
# memory. It must be a binary string. This primarily exists for testing.
|
50
|
-
|
45
|
+
#
|
46
|
+
# @!visibility private
|
47
|
+
MODE_PARAM_IS_BUFFER = :MODE_PARAM_IS_BUFFER
|
51
48
|
|
52
49
|
DATA_SECTION_SEPARATOR_SIZE = 16
|
53
50
|
private_constant :DATA_SECTION_SEPARATOR_SIZE
|
@@ -59,7 +56,9 @@ module MaxMind # :nodoc:
|
|
59
56
|
private_constant :METADATA_MAX_SIZE
|
60
57
|
|
61
58
|
# Return the metadata associated with the {MaxMind
|
62
|
-
# DB}[https://maxmind.github.io/MaxMind-DB/]
|
59
|
+
# DB}[https://maxmind.github.io/MaxMind-DB/]
|
60
|
+
#
|
61
|
+
# @return [MaxMind::DB::Metadata]
|
63
62
|
attr_reader :metadata
|
64
63
|
|
65
64
|
# Create a DB. A DB provides a way to read {MaxMind DB
|
@@ -70,20 +69,20 @@ module MaxMind # :nodoc:
|
|
70
69
|
# is safe to use after forking only if you use MODE_MEMORY or if your
|
71
70
|
# version of Ruby supports IO#pread.
|
72
71
|
#
|
73
|
-
#
|
72
|
+
# @param database [String] a path to a {MaxMind
|
73
|
+
# DB}[https://maxmind.github.io/MaxMind-DB/].
|
74
74
|
#
|
75
|
-
#
|
76
|
-
# DB
|
75
|
+
# @param options [Hash<Symbol, Symbol>] options controlling the behavior of
|
76
|
+
# the DB.
|
77
77
|
#
|
78
|
-
#
|
79
|
-
#
|
78
|
+
# @option options [Symbol] :mode Defines how to open the database. It may
|
79
|
+
# be one of MODE_AUTO, MODE_FILE, or MODE_MEMORY. If you don't provide
|
80
|
+
# one, DB uses MODE_AUTO. Refer to the definition of those constants for
|
81
|
+
# an explanation of their meaning.
|
80
82
|
#
|
81
|
-
#
|
83
|
+
# @raise [InvalidDatabaseError] if the database is corrupt or invalid.
|
82
84
|
#
|
83
|
-
# [
|
84
|
-
# MODE_FILE, or MODE_MEMORY. If you don't provide one, DB uses
|
85
|
-
# MODE_AUTO. Refer to the definition of those constants for an
|
86
|
-
# explanation of their meaning.
|
85
|
+
# @raise [ArgumentError] if the mode is invalid.
|
87
86
|
def initialize(database, options = {})
|
88
87
|
options[:mode] = MODE_AUTO unless options.key?(:mode)
|
89
88
|
|
@@ -125,35 +124,44 @@ module MaxMind # :nodoc:
|
|
125
124
|
end
|
126
125
|
end
|
127
126
|
|
128
|
-
# Return the record for the
|
127
|
+
# Return the record for the IP address in the {MaxMind
|
129
128
|
# DB}[https://maxmind.github.io/MaxMind-DB/]. The record can be one of
|
130
129
|
# several types and depends on the contents of the database.
|
131
130
|
#
|
132
|
-
# If no record is found for
|
131
|
+
# If no record is found for the IP address, +get+ returns +nil+.
|
132
|
+
#
|
133
|
+
# @param ip_address [String] a string in the standard notation. It may be
|
134
|
+
# IPv4 or IPv6.
|
133
135
|
#
|
134
|
-
#
|
136
|
+
# @raise [ArgumentError] if you attempt to look up an IPv6 address in an
|
137
|
+
# IPv4-only database.
|
135
138
|
#
|
136
|
-
#
|
137
|
-
#
|
139
|
+
# @raise [InvalidDatabaseError] if the database is corrupt or invalid.
|
140
|
+
#
|
141
|
+
# @return [Object, nil]
|
138
142
|
def get(ip_address)
|
139
143
|
record, = get_with_prefix_length(ip_address)
|
140
144
|
|
141
145
|
record
|
142
146
|
end
|
143
147
|
|
144
|
-
# Return an array containing the record for the
|
148
|
+
# Return an array containing the record for the IP address in the
|
145
149
|
# {MaxMind DB}[https://maxmind.github.io/MaxMind-DB/] and its associated
|
146
150
|
# network prefix length. The record can be one of several types and
|
147
151
|
# depends on the contents of the database.
|
148
152
|
#
|
149
|
-
# If no record is found for
|
153
|
+
# If no record is found for the IP address, the record will be +nil+ and
|
150
154
|
# the prefix length will be the value for the missing network.
|
151
155
|
#
|
152
|
-
#
|
153
|
-
#
|
156
|
+
# @param ip_address [String] a string in the standard notation. It may be
|
157
|
+
# IPv4 or IPv6.
|
158
|
+
#
|
159
|
+
# @raise [ArgumentError] if you attempt to look up an IPv6 address in an
|
160
|
+
# IPv4-only database.
|
154
161
|
#
|
155
|
-
#
|
156
|
-
#
|
162
|
+
# @raise [InvalidDatabaseError] if the database is corrupt or invalid.
|
163
|
+
#
|
164
|
+
# @return [Array<(Object, Integer)>]
|
157
165
|
def get_with_prefix_length(ip_address)
|
158
166
|
ip = IPAddr.new(ip_address)
|
159
167
|
# We could check the IP has the correct prefix (32 or 128) but I do not
|
@@ -221,8 +229,6 @@ module MaxMind # :nodoc:
|
|
221
229
|
|
222
230
|
# Read a record from the indicated node. Index indicates whether it's the
|
223
231
|
# left (0) or right (1) record.
|
224
|
-
#
|
225
|
-
# rubocop:disable Metrics/CyclomaticComplexity
|
226
232
|
def read_node(node_number, index)
|
227
233
|
base_offset = node_number * @node_byte_size
|
228
234
|
|
@@ -253,7 +259,6 @@ module MaxMind # :nodoc:
|
|
253
259
|
|
254
260
|
raise InvalidDatabaseError, "Unsupported record size: #{@record_size}"
|
255
261
|
end
|
256
|
-
# rubocop:enable Metrics/CyclomaticComplexity
|
257
262
|
|
258
263
|
def resolve_data_pointer(pointer)
|
259
264
|
offset_in_file = pointer - @node_count + @search_tree_size
|
@@ -290,8 +295,7 @@ module MaxMind # :nodoc:
|
|
290
295
|
|
291
296
|
# Close the DB and return resources to the system.
|
292
297
|
#
|
293
|
-
#
|
294
|
-
# an error.
|
298
|
+
# @return [void]
|
295
299
|
def close
|
296
300
|
@io.close
|
297
301
|
end
|
data/lib/maxmind/db/decoder.rb
CHANGED
@@ -2,14 +2,16 @@
|
|
2
2
|
|
3
3
|
require 'maxmind/db/errors'
|
4
4
|
|
5
|
-
module MaxMind
|
5
|
+
module MaxMind
|
6
6
|
class DB
|
7
7
|
# +Decoder+ decodes a {MaxMind DB}[https://maxmind.github.io/MaxMind-DB/]
|
8
8
|
# data section.
|
9
9
|
#
|
10
10
|
# Typically you will interact with this class through a Reader rather than
|
11
11
|
# directly.
|
12
|
-
|
12
|
+
#
|
13
|
+
# @!visibility private
|
14
|
+
class Decoder
|
13
15
|
# Create a +Decoder+.
|
14
16
|
#
|
15
17
|
# +io+ is the DB. It must provide a +read+ method. It must be opened in
|
data/lib/maxmind/db/errors.rb
CHANGED
@@ -1,8 +1,9 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
module MaxMind
|
3
|
+
module MaxMind
|
4
4
|
class DB
|
5
|
-
|
5
|
+
# @!visibility private
|
6
|
+
class MemoryReader
|
6
7
|
def initialize(filename, options = {})
|
7
8
|
if options[:is_buffer]
|
8
9
|
@buf = filename
|
@@ -18,8 +19,7 @@ module MaxMind # :nodoc:
|
|
18
19
|
|
19
20
|
# Override to not show @buf in inspect to avoid showing it in irb.
|
20
21
|
def inspect
|
21
|
-
|
22
|
-
s << '@size=' << @size.inspect << '>'
|
22
|
+
"#<#{self.class.name}:0x#{self.class.object_id.to_s(16)}, @size=#{@size.inspect}>"
|
23
23
|
end
|
24
24
|
|
25
25
|
def close; end
|
data/lib/maxmind/db/metadata.rb
CHANGED
@@ -1,42 +1,64 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
module MaxMind
|
3
|
+
module MaxMind
|
4
4
|
class DB
|
5
5
|
# Metadata holds metadata about a {MaxMind
|
6
|
-
# DB}[https://maxmind.github.io/MaxMind-DB/] file.
|
6
|
+
# DB}[https://maxmind.github.io/MaxMind-DB/] file. See
|
7
|
+
# https://maxmind.github.io/MaxMind-DB/#database-metadata for the
|
8
|
+
# specification.
|
7
9
|
class Metadata
|
8
10
|
# The number of nodes in the database.
|
11
|
+
#
|
12
|
+
# @return [Integer]
|
9
13
|
attr_reader :node_count
|
10
14
|
|
11
15
|
# The bit size of a record in the search tree.
|
16
|
+
#
|
17
|
+
# @return [Integer]
|
12
18
|
attr_reader :record_size
|
13
19
|
|
14
20
|
# The IP version of the data in the database. A value of 4 means the
|
15
21
|
# database only supports IPv4. A database with a value of 6 may support
|
16
22
|
# both IPv4 and IPv6 lookups.
|
23
|
+
#
|
24
|
+
# @return [Integer]
|
17
25
|
attr_reader :ip_version
|
18
26
|
|
19
27
|
# A string identifying the database type. e.g., "GeoIP2-City".
|
28
|
+
#
|
29
|
+
# @return [String]
|
20
30
|
attr_reader :database_type
|
21
31
|
|
22
32
|
# An array of locale codes supported by the database.
|
33
|
+
#
|
34
|
+
# @return [Array<String>]
|
23
35
|
attr_reader :languages
|
24
36
|
|
25
37
|
# The major version number of the binary format used when creating the
|
26
38
|
# database.
|
39
|
+
#
|
40
|
+
# @return [Integer]
|
27
41
|
attr_reader :binary_format_major_version
|
28
42
|
|
29
43
|
# The minor version number of the binary format used when creating the
|
30
44
|
# database.
|
45
|
+
#
|
46
|
+
# @return [Integer]
|
31
47
|
attr_reader :binary_format_minor_version
|
32
48
|
|
33
49
|
# The Unix epoch for the build time of the database.
|
50
|
+
#
|
51
|
+
# @return [Integer]
|
34
52
|
attr_reader :build_epoch
|
35
53
|
|
36
54
|
# A hash from locales to text descriptions of the database.
|
55
|
+
#
|
56
|
+
# @return [Hash<String, String>]
|
37
57
|
attr_reader :description
|
38
58
|
|
39
59
|
# +m+ is a hash representing the metadata map.
|
60
|
+
#
|
61
|
+
# @!visibility private
|
40
62
|
def initialize(map)
|
41
63
|
@node_count = map['node_count']
|
42
64
|
@record_size = map['record_size']
|
@@ -50,11 +72,15 @@ module MaxMind # :nodoc:
|
|
50
72
|
end
|
51
73
|
|
52
74
|
# The size of a node in bytes.
|
75
|
+
#
|
76
|
+
# @return [Integer]
|
53
77
|
def node_byte_size
|
54
78
|
@record_size / 4
|
55
79
|
end
|
56
80
|
|
57
81
|
# The size of the search tree in bytes.
|
82
|
+
#
|
83
|
+
# @return [Integer]
|
58
84
|
def search_tree_size
|
59
85
|
@node_count * node_byte_size
|
60
86
|
end
|
data/maxmind-db.gemspec
CHANGED
@@ -5,7 +5,7 @@ Gem::Specification.new do |s|
|
|
5
5
|
s.files = Dir['**/*']
|
6
6
|
s.name = 'maxmind-db'
|
7
7
|
s.summary = 'A gem for reading MaxMind DB files.'
|
8
|
-
s.version = '1.1.
|
8
|
+
s.version = '1.1.1'
|
9
9
|
|
10
10
|
s.description = 'A gem for reading MaxMind DB files. MaxMind DB is a binary file format that stores data indexed by IP address subnets (IPv4 or IPv6).'
|
11
11
|
s.email = 'support@maxmind.com'
|
@@ -14,9 +14,14 @@ Gem::Specification.new do |s|
|
|
14
14
|
s.metadata = {
|
15
15
|
'bug_tracker_uri' => 'https://github.com/maxmind/MaxMind-DB-Reader-ruby/issues',
|
16
16
|
'changelog_uri' => 'https://github.com/maxmind/MaxMind-DB-Reader-ruby/blob/master/CHANGELOG.md',
|
17
|
-
'documentation_uri' => 'https://
|
17
|
+
'documentation_uri' => 'https://www.rubydoc.info/gems/maxmind-db',
|
18
18
|
'homepage_uri' => 'https://github.com/maxmind/MaxMind-DB-Reader-ruby',
|
19
19
|
'source_code_uri' => 'https://github.com/maxmind/MaxMind-DB-Reader-ruby',
|
20
20
|
}
|
21
21
|
s.required_ruby_version = '>= 2.4.0'
|
22
|
+
|
23
|
+
s.add_development_dependency 'minitest'
|
24
|
+
s.add_development_dependency 'rake'
|
25
|
+
s.add_development_dependency 'rubocop'
|
26
|
+
s.add_development_dependency 'rubocop-performance'
|
22
27
|
end
|
data/test/data/LICENSE
ADDED
@@ -0,0 +1,4 @@
|
|
1
|
+
This work is licensed under the Creative Commons Attribution-ShareAlike 3.0
|
2
|
+
Unported License. To view a copy of this license, visit
|
3
|
+
http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative
|
4
|
+
Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
|
@@ -0,0 +1,570 @@
|
|
1
|
+
---
|
2
|
+
layout: default
|
3
|
+
title: MaxMind DB File Format Specification
|
4
|
+
version: v2.0
|
5
|
+
---
|
6
|
+
# MaxMind DB File Format Specification
|
7
|
+
|
8
|
+
## Description
|
9
|
+
|
10
|
+
The MaxMind DB file format is a database format that maps IPv4 and IPv6
|
11
|
+
addresses to data records using an efficient binary search tree.
|
12
|
+
|
13
|
+
## Version
|
14
|
+
|
15
|
+
This spec documents **version 2.0** of the MaxMind DB binary format.
|
16
|
+
|
17
|
+
The version number consists of separate major and minor version numbers. It
|
18
|
+
should not be considered a decimal number. In other words, version 2.10 comes
|
19
|
+
after version 2.9.
|
20
|
+
|
21
|
+
Code which is capable of reading a given major version of the format should
|
22
|
+
not be broken by minor version changes to the format.
|
23
|
+
|
24
|
+
## Overview
|
25
|
+
|
26
|
+
The binary database is split into three parts:
|
27
|
+
|
28
|
+
1. The binary search tree. Each level of the tree corresponds to a single bit
|
29
|
+
in the 128 bit representation of an IPv6 address.
|
30
|
+
2. The data section. These are the values returned to the client for a
|
31
|
+
specific IP address, e.g. "US", "New York", or a more complex map type made up
|
32
|
+
of multiple fields.
|
33
|
+
3. Database metadata. Information about the database itself.
|
34
|
+
|
35
|
+
## Database Metadata
|
36
|
+
|
37
|
+
This portion of the database is stored at the end of the file. It is
|
38
|
+
documented first because understanding some of the metadata is key to
|
39
|
+
understanding how the other sections work.
|
40
|
+
|
41
|
+
This section can be found by looking for a binary sequence matching
|
42
|
+
"\xab\xcd\xefMaxMind.com". The *last* occurrence of this string in the file
|
43
|
+
marks the end of the data section and the beginning of the metadata. Since we
|
44
|
+
allow for arbitrary binary data in the data section, some other piece of data
|
45
|
+
could contain these values. This is why you need to find the last occurrence
|
46
|
+
of this sequence.
|
47
|
+
|
48
|
+
The maximum allowable size for the metadata section, including the marker that
|
49
|
+
starts the metadata, is 128KiB.
|
50
|
+
|
51
|
+
The metadata is stored as a map data structure. This structure is described
|
52
|
+
later in the spec. Changing a key's data type or removing a key would
|
53
|
+
constitute a major version change for this spec.
|
54
|
+
|
55
|
+
Except where otherwise specified, each key listed is required for the database
|
56
|
+
to be considered valid.
|
57
|
+
|
58
|
+
Adding a key constitutes a minor version change. Removing a key or changing
|
59
|
+
its type constitutes a major version change.
|
60
|
+
|
61
|
+
The list of known keys for the current version of the format is as follows:
|
62
|
+
|
63
|
+
### node\_count
|
64
|
+
|
65
|
+
This is an unsigned 32-bit integer indicating the number of nodes in the
|
66
|
+
search tree.
|
67
|
+
|
68
|
+
### record\_size
|
69
|
+
|
70
|
+
This is an unsigned 16-bit integer. It indicates the number of bits in a
|
71
|
+
record in the search tree. Note that each node consists of *two* records.
|
72
|
+
|
73
|
+
### ip\_version
|
74
|
+
|
75
|
+
This is an unsigned 16-bit integer which is always 4 or 6. It indicates
|
76
|
+
whether the database contains IPv4 or IPv6 address data.
|
77
|
+
|
78
|
+
### database\_type
|
79
|
+
|
80
|
+
This is a string that indicates the structure of each data record associated
|
81
|
+
with an IP address. The actual definition of these structures is left up to
|
82
|
+
the database creator.
|
83
|
+
|
84
|
+
Names starting with "GeoIP" are reserved for use by MaxMind (and "GeoIP" is a
|
85
|
+
trademark anyway).
|
86
|
+
|
87
|
+
### languages
|
88
|
+
|
89
|
+
An array of strings, each of which is a locale code. A given record may
|
90
|
+
contain data items that have been localized to some or all of these
|
91
|
+
locales. Records should not contain localized data for locales not included in
|
92
|
+
this array.
|
93
|
+
|
94
|
+
This is an optional key, as this may not be relevant for all types of data.
|
95
|
+
|
96
|
+
### binary\_format\_major\_version
|
97
|
+
|
98
|
+
This is an unsigned 16-bit integer indicating the major version number for the
|
99
|
+
database's binary format.
|
100
|
+
|
101
|
+
### binary\_format\_minor\_version
|
102
|
+
|
103
|
+
This is an unsigned 16-bit integer indicating the minor version number for the
|
104
|
+
database's binary format.
|
105
|
+
|
106
|
+
### build\_epoch
|
107
|
+
|
108
|
+
This is an unsigned 64-bit integer that contains the database build timestamp
|
109
|
+
as a Unix epoch value.
|
110
|
+
|
111
|
+
### description
|
112
|
+
|
113
|
+
This key will always point to a map. The keys of that map will be language
|
114
|
+
codes, and the values will be a description in that language as a UTF-8
|
115
|
+
string.
|
116
|
+
|
117
|
+
The codes may include additional information such as script or country
|
118
|
+
identifiers, like "zh-TW" or "mn-Cyrl-MN". The additional identifiers will be
|
119
|
+
separated by a dash character ("-").
|
120
|
+
|
121
|
+
This key is optional. However, creators of databases are strongly
|
122
|
+
encouraged to include a description in at least one language.
|
123
|
+
|
124
|
+
### Calculating the Search Tree Section Size
|
125
|
+
|
126
|
+
The formula for calculating the search tree section size *in bytes* is as
|
127
|
+
follows:
|
128
|
+
|
129
|
+
( ( $record_size * 2 ) / 8 ) * $number_of_nodes
|
130
|
+
|
131
|
+
The end of the search tree marks the beginning of the data section.
|
132
|
+
|
133
|
+
## Binary Search Tree Section
|
134
|
+
|
135
|
+
The database file starts with a binary search tree. The number of nodes in the
|
136
|
+
tree is dependent on how many unique netblocks are needed for the particular
|
137
|
+
database. For example, the city database needs many more small netblocks than
|
138
|
+
the country database.
|
139
|
+
|
140
|
+
The top most node is always located at the beginning of the search tree
|
141
|
+
section's address space. The top node is node 0.
|
142
|
+
|
143
|
+
Each node consists of two records, each of which is a pointer to an address in
|
144
|
+
the file.
|
145
|
+
|
146
|
+
The pointers can point to one of three things. First, it may point to another
|
147
|
+
node in the search tree address space. These pointers are followed as part of
|
148
|
+
the IP address search algorithm, described below.
|
149
|
+
|
150
|
+
The pointer can point to a value equal to `$number_of_nodes`. If this is the
|
151
|
+
case, it means that the IP address we are searching for is not in the
|
152
|
+
database.
|
153
|
+
|
154
|
+
Finally, it may point to an address in the data section. This is the data
|
155
|
+
relevant to the given netblock.
|
156
|
+
|
157
|
+
### Node Layout
|
158
|
+
|
159
|
+
Each node in the search tree consists of two records, each of which is a
|
160
|
+
pointer. The record size varies by database, but inside a single database node
|
161
|
+
records are always the same size. A record may be anywhere from 24 to 128 bits
|
162
|
+
long, depending on the number of nodes in the tree. These pointers are
|
163
|
+
stored in big-endian format (most significant byte first).
|
164
|
+
|
165
|
+
Here are some examples of how the records are laid out in a node for 24, 28,
|
166
|
+
and 32 bit records. Larger record sizes follow this same pattern.
|
167
|
+
|
168
|
+
#### 24 bits (small database), one node is 6 bytes
|
169
|
+
|
170
|
+
| <------------- node --------------->|
|
171
|
+
| 23 .. 0 | 23 .. 0 |
|
172
|
+
|
173
|
+
#### 28 bits (medium database), one node is 7 bytes
|
174
|
+
|
175
|
+
| <------------- node --------------->|
|
176
|
+
| 23 .. 0 | 27..24 | 27..24 | 23 .. 0 |
|
177
|
+
|
178
|
+
Note 4 bits of each pointer are combined into the middle byte. For both
|
179
|
+
records, they are prepended and end up in the most significant position.
|
180
|
+
|
181
|
+
#### 32 bits (large database), one node is 8 bytes
|
182
|
+
|
183
|
+
| <------------- node --------------->|
|
184
|
+
| 31 .. 0 | 31 .. 0 |
|
185
|
+
|
186
|
+
### Search Lookup Algorithm
|
187
|
+
|
188
|
+
The first step is to convert the IP address to its big-endian binary
|
189
|
+
representation. For an IPv4 address, this becomes 32 bits. For IPv6 you get
|
190
|
+
128 bits.
|
191
|
+
|
192
|
+
The leftmost bit corresponds to the first node in the search tree. For each
|
193
|
+
bit, a value of 0 means we choose the left record in a node, and a value of 1
|
194
|
+
means we choose the right record.
|
195
|
+
|
196
|
+
The record value is always interpreted as an unsigned integer. The maximum
|
197
|
+
size of the integer is dependent on the number of bits in a record (24, 28, or
|
198
|
+
32).
|
199
|
+
|
200
|
+
If the record value is a number that is less than the *number of nodes* (not
|
201
|
+
in bytes, but the actual node count) in the search tree (this is stored in the
|
202
|
+
database metadata), then the value is a node number. In this case, we find
|
203
|
+
that node in the search tree and repeat the lookup algorithm from there.
|
204
|
+
|
205
|
+
If the record value is equal to the number of nodes, that means that we do not
|
206
|
+
have any data for the IP address, and the search ends here.
|
207
|
+
|
208
|
+
If the record value is *greater* than the number of nodes in the search tree,
|
209
|
+
then it is an actual pointer value pointing into the data section. The value
|
210
|
+
of the pointer is relative to the start of the data section, *not* the
|
211
|
+
start of the file.
|
212
|
+
|
213
|
+
In order to determine where in the data section we should start looking, we use
|
214
|
+
the following formula:
|
215
|
+
|
216
|
+
$data_section_offset = ( $record_value - $node_count ) - 16
|
217
|
+
|
218
|
+
The 16 is the size of the data section separator. We subtract it because we
|
219
|
+
want to permit pointing to the first byte of the data section. Recall that
|
220
|
+
the record value cannot equal the node count as that means there is no
|
221
|
+
data. Instead, we choose to start values that go to the data section at
|
222
|
+
`$node_count + 16`. (This has the side effect that record values
|
223
|
+
`$node_count + 1` through `$node_count + 15` inclusive are not valid).
|
224
|
+
|
225
|
+
This is best demonstrated by an example:
|
226
|
+
|
227
|
+
Let's assume we have a 24-bit tree with 1,000 nodes. Each node contains 48
|
228
|
+
bits, or 6 bytes. The size of the tree is 6,000 bytes.
|
229
|
+
|
230
|
+
When a record in the tree contains a number that is less than 1,000, this
|
231
|
+
is a *node number*, and we look up that node. If a record contains a value
|
232
|
+
greater than or equal to 1,016, we know that it is a data section value. We
|
233
|
+
subtract the node count (1,000) and then subtract 16 for the data section
|
234
|
+
separator, giving us the number 0, the first byte of the data section.
|
235
|
+
|
236
|
+
If a record contained the value 6,000, this formula would give us an offset of
|
237
|
+
4,984 into the data section.
|
238
|
+
|
239
|
+
In order to determine where in the file this offset really points to, we also
|
240
|
+
need to know where the data section starts. This can be calculated by
|
241
|
+
determining the size of the search tree in bytes and then adding an additional
|
242
|
+
16 bytes for the data section separator:
|
243
|
+
|
244
|
+
$offset_in_file = $data_section_offset
|
245
|
+
+ $search_tree_size_in_bytes
|
246
|
+
+ 16
|
247
|
+
|
248
|
+
Since we subtract and then add 16, the final formula to determine the
|
249
|
+
offset in the file can be simplified to:
|
250
|
+
|
251
|
+
$offset_in_file = ( $record_value - $node_count )
|
252
|
+
+ $search_tree_size_in_bytes
|
253
|
+
|
254
|
+
### IPv4 addresses in an IPv6 tree
|
255
|
+
|
256
|
+
When storing IPv4 addresses in an IPv6 tree, they are stored as-is, so they
|
257
|
+
occupy the first 32-bits of the address space (from 0 to 2**32 - 1).
|
258
|
+
|
259
|
+
Creators of databases should decide on a strategy for handling the various
|
260
|
+
mappings between IPv4 and IPv6.
|
261
|
+
|
262
|
+
The strategy that MaxMind uses for its GeoIP databases is to include a pointer
|
263
|
+
from the `::ffff:0:0/96` subnet to the root node of the IPv4 address space in
|
264
|
+
the tree. This accounts for the
|
265
|
+
[IPv4-mapped IPv6 address](http://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses).
|
266
|
+
|
267
|
+
MaxMind also includes a pointer from the `2002::/16` subnet to the root node
|
268
|
+
of the IPv4 address space in the tree. This accounts for the
|
269
|
+
[6to4 mapping](http://en.wikipedia.org/wiki/6to4) subnet.
|
270
|
+
|
271
|
+
Database creators are encouraged to document whether they are doing something
|
272
|
+
similar for their databases.
|
273
|
+
|
274
|
+
The Teredo subnet cannot be accounted for in the tree. Instead, code that
|
275
|
+
searches the tree can offer to decode the IPv4 portion of a Teredo address and
|
276
|
+
look that up.
|
277
|
+
|
278
|
+
## Data Section Separator
|
279
|
+
|
280
|
+
There are 16 bytes of NULLs in between the search tree and the data
|
281
|
+
section. This separator exists in order to make it possible for a verification
|
282
|
+
tool to distinguish between the two sections.
|
283
|
+
|
284
|
+
This separator is not considered part of the data section itself. In other
|
285
|
+
words, the data section starts at `$size_of_search_tree + 16` bytes in the
|
286
|
+
file.
|
287
|
+
|
288
|
+
## Output Data Section
|
289
|
+
|
290
|
+
Each output data field has an associated type, and that type is encoded as a
|
291
|
+
number that begins the data field. Some types are variable length. In those
|
292
|
+
cases, the type indicator is also followed by a length. The data payload
|
293
|
+
always comes at the end of the field.
|
294
|
+
|
295
|
+
All binary data is stored in big-endian format.
|
296
|
+
|
297
|
+
Note that the *interpretation* of a given data type's meaning is decided by
|
298
|
+
higher-level APIs, not by the binary format itself.
|
299
|
+
|
300
|
+
### pointer - 1
|
301
|
+
|
302
|
+
A pointer to another part of the data section's address space. The pointer
|
303
|
+
will point to the beginning of a field. It is illegal for a pointer to point
|
304
|
+
to another pointer.
|
305
|
+
|
306
|
+
Pointer values start from the beginning of the data section, *not* the
|
307
|
+
beginning of the file.
|
308
|
+
|
309
|
+
### UTF-8 string - 2
|
310
|
+
|
311
|
+
A variable length byte sequence that contains valid utf8. If the length is
|
312
|
+
zero then this is an empty string.
|
313
|
+
|
314
|
+
### double - 3
|
315
|
+
|
316
|
+
This is stored as an IEEE-754 double (binary64) in big-endian format. The
|
317
|
+
length of a double is always 8 bytes.
|
318
|
+
|
319
|
+
### bytes - 4
|
320
|
+
|
321
|
+
A variable length byte sequence containing any sort of binary data. If the
|
322
|
+
length is zero then this a zero-length byte sequence.
|
323
|
+
|
324
|
+
This is not currently used but may be used in the future to embed non-text
|
325
|
+
data (images, etc.).
|
326
|
+
|
327
|
+
### integer formats
|
328
|
+
|
329
|
+
Integers are stored in variable length binary fields.
|
330
|
+
|
331
|
+
We support 16-bit, 32-bit, 64-bit, and 128-bit unsigned integers. We also
|
332
|
+
support 32-bit signed integers.
|
333
|
+
|
334
|
+
A 128-bit integer can use up to 16 bytes, but may use fewer. Similarly, a
|
335
|
+
32-bit integer may use from 0-4 bytes. The number of bytes used is determined
|
336
|
+
by the length specifier in the control byte. See below for details.
|
337
|
+
|
338
|
+
A length of zero always indicates the number 0.
|
339
|
+
|
340
|
+
When storing a signed integer, the left-most bit is the sign. A 1 is negative
|
341
|
+
and a 0 is positive.
|
342
|
+
|
343
|
+
The type numbers for our integer types are:
|
344
|
+
|
345
|
+
* unsigned 16-bit int - 5
|
346
|
+
* unsigned 32-bit int - 6
|
347
|
+
* signed 32-bit int - 8
|
348
|
+
* unsigned 64-bit int - 9
|
349
|
+
* unsigned 128-bit int - 10
|
350
|
+
|
351
|
+
The unsigned 32-bit and 128-bit types may be used to store IPv4 and IPv6
|
352
|
+
addresses, respectively.
|
353
|
+
|
354
|
+
The signed 32-bit integers are stored using the 2's complement representation.
|
355
|
+
|
356
|
+
### map - 7
|
357
|
+
|
358
|
+
A map data type contains a set of key/value pairs. Unlike other data types,
|
359
|
+
the length information for maps indicates how many key/value pairs it
|
360
|
+
contains, not its length in bytes. This size can be zero.
|
361
|
+
|
362
|
+
See below for the algorithm used to determine the number of pairs in the
|
363
|
+
hash. This algorithm is also used to determine the length of a field's
|
364
|
+
payload.
|
365
|
+
|
366
|
+
### array - 11
|
367
|
+
|
368
|
+
An array type contains a set of ordered values. The length information for
|
369
|
+
arrays indicates how many values it contains, not its length in bytes. This
|
370
|
+
size can be zero.
|
371
|
+
|
372
|
+
This type uses the same algorithm as maps for determining the length of a
|
373
|
+
field's payload.
|
374
|
+
|
375
|
+
### data cache container - 12
|
376
|
+
|
377
|
+
This is a special data type that marks a container used to cache repeated
|
378
|
+
data. For example, instead of repeating the string "United States" over and
|
379
|
+
over in the database, we store it in the cache container and use pointers
|
380
|
+
*into* this container instead.
|
381
|
+
|
382
|
+
Nothing in the database will ever contain a pointer to this field
|
383
|
+
itself. Instead, various fields will point into the container.
|
384
|
+
|
385
|
+
The primary reason for making this a separate data type versus simply inlining
|
386
|
+
the cached data is so that a database dumper tool can skip this cache when
|
387
|
+
dumping the data section. The cache contents will end up being dumped as
|
388
|
+
pointers into it are followed.
|
389
|
+
|
390
|
+
### end marker - 13
|
391
|
+
|
392
|
+
The end marker marks the end of the data section. It is not strictly
|
393
|
+
necessary, but including this marker allows a data section deserializer to
|
394
|
+
process a stream of input, rather than having to find the end of the section
|
395
|
+
before beginning the deserialization.
|
396
|
+
|
397
|
+
This data type is not followed by a payload, and its size is always zero.
|
398
|
+
|
399
|
+
### boolean - 14
|
400
|
+
|
401
|
+
A true or false value. The length information for a boolean type will always
|
402
|
+
be 0 or 1, indicating the value. There is no payload for this field.
|
403
|
+
|
404
|
+
### float - 15
|
405
|
+
|
406
|
+
This is stored as an IEEE-754 float (binary32) in big-endian format. The
|
407
|
+
length of a float is always 4 bytes.
|
408
|
+
|
409
|
+
This type is provided primarily for completeness. Because of the way floating
|
410
|
+
point numbers are stored, this type can easily lose precision when serialized
|
411
|
+
and then deserialized. If this is an issue for you, consider using a double
|
412
|
+
instead.
|
413
|
+
|
414
|
+
### Data Field Format
|
415
|
+
|
416
|
+
Each field starts with a control byte. This control byte provides information
|
417
|
+
about the field's data type and payload size.
|
418
|
+
|
419
|
+
The first three bits of the control byte tell you what type the field is. If
|
420
|
+
these bits are all 0, then this is an "extended" type, which means that the
|
421
|
+
*next* byte contains the actual type. Otherwise, the first three bits will
|
422
|
+
contain a number from 1 to 7, the actual type for the field.
|
423
|
+
|
424
|
+
We've tried to assign the most commonly used types as numbers 1-7 as an
|
425
|
+
optimization.
|
426
|
+
|
427
|
+
With an extended type, the type number in the second byte is the number
|
428
|
+
minus 7. In other words, an array (type 11) will be stored with a 0 for the
|
429
|
+
type in the first byte and a 4 in the second.
|
430
|
+
|
431
|
+
Here is an example of how the control byte may combine with the next byte to
|
432
|
+
tell us the type:
|
433
|
+
|
434
|
+
001XXXXX pointer
|
435
|
+
010XXXXX UTF-8 string
|
436
|
+
110XXXXX unsigned 32-bit int (ASCII)
|
437
|
+
000XXXXX 00000011 unsigned 128-bit int (binary)
|
438
|
+
000XXXXX 00000100 array
|
439
|
+
000XXXXX 00000110 end marker
|
440
|
+
|
441
|
+
#### Payload Size
|
442
|
+
|
443
|
+
The next five bits in the control byte tell you how long the data field's
|
444
|
+
payload is, except for maps and pointers. Maps and pointers use this size
|
445
|
+
information a bit differently. See below.
|
446
|
+
|
447
|
+
If the five bits are smaller than 29, then those bits are the payload size in
|
448
|
+
bytes. For example:
|
449
|
+
|
450
|
+
01000010 UTF-8 string - 2 bytes long
|
451
|
+
01011100 UTF-8 string - 28 bytes long
|
452
|
+
11000001 unsigned 32-bit int - 1 byte long
|
453
|
+
00000011 00000011 unsigned 128-bit int - 3 bytes long
|
454
|
+
|
455
|
+
If the five bits are equal to 29, 30, or 31, then use the following algorithm
|
456
|
+
to calculate the payload size.
|
457
|
+
|
458
|
+
If the value is 29, then the size is 29 + *the next byte after the type
|
459
|
+
specifying bytes as an unsigned integer*.
|
460
|
+
|
461
|
+
If the value is 30, then the size is 285 + *the next two bytes after the type
|
462
|
+
specifying bytes as a single unsigned integer*.
|
463
|
+
|
464
|
+
If the value is 31, then the size is 65,821 + *the next three bytes after the
|
465
|
+
type specifying bytes as a single unsigned integer*.
|
466
|
+
|
467
|
+
Some examples:
|
468
|
+
|
469
|
+
01011101 00110011 UTF-8 string - 80 bytes long
|
470
|
+
|
471
|
+
In this case, the last five bits of the control byte equal 29. We treat the
|
472
|
+
next byte as an unsigned integer. The next byte is 51, so the total size is
|
473
|
+
(29 + 51) = 80.
|
474
|
+
|
475
|
+
01011110 00110011 00110011 UTF-8 string - 13,392 bytes long
|
476
|
+
|
477
|
+
The last five bits of the control byte equal 30. We treat the next two bytes
|
478
|
+
as a single unsigned integer. The next two bytes equal 13,107, so the total
|
479
|
+
size is (285 + 13,107) = 13,392.
|
480
|
+
|
481
|
+
01011111 00110011 00110011 00110011 UTF-8 string - 3,421,264 bytes long
|
482
|
+
|
483
|
+
The last five bits of the control byte equal 31. We treat the next three bytes
|
484
|
+
as a single unsigned integer. The next three bytes equal 3,355,443, so the
|
485
|
+
total size is (65,821 + 3,355,443) = 3,421,264.
|
486
|
+
|
487
|
+
This means that the maximum payload size for a single field is 16,843,036
|
488
|
+
bytes.
|
489
|
+
|
490
|
+
The binary number types always have a known size, but for consistency's sake,
|
491
|
+
the control byte will always specify the correct size for these types.
|
492
|
+
|
493
|
+
#### Maps
|
494
|
+
|
495
|
+
Maps use the size in the control byte (and any following bytes) to indicate
|
496
|
+
the number of key/value pairs in the map, not the size of the payload in
|
497
|
+
bytes.
|
498
|
+
|
499
|
+
This means that the maximum number of pairs for a single map is 16,843,036.
|
500
|
+
|
501
|
+
Maps are laid out with each key followed by its value, followed by the next
|
502
|
+
pair, etc.
|
503
|
+
|
504
|
+
The keys are **always** UTF-8 strings. The values may be any data type,
|
505
|
+
including maps or pointers.
|
506
|
+
|
507
|
+
Once we know the number of pairs, we can look at each pair in turn to
|
508
|
+
determine the size of the key and the key name, as well as the value's type
|
509
|
+
and payload.
|
510
|
+
|
511
|
+
#### Pointers
|
512
|
+
|
513
|
+
Pointers use the last five bits in the control byte to calculate the pointer
|
514
|
+
value.
|
515
|
+
|
516
|
+
To calculate the pointer value, we start by subdividing the five bits into two
|
517
|
+
groups. The first two bits indicate the size, and the next three bits are part
|
518
|
+
of the value, so we end up with a control byte breaking down like this:
|
519
|
+
001SSVVV.
|
520
|
+
|
521
|
+
The size can be 0, 1, 2, or 3.
|
522
|
+
|
523
|
+
If the size is 0, the pointer is built by appending the next byte to the last
|
524
|
+
three bits to produce an 11-bit value.
|
525
|
+
|
526
|
+
If the size is 1, the pointer is built by appending the next two bytes to the
|
527
|
+
last three bits to produce a 19-bit value + 2048.
|
528
|
+
|
529
|
+
If the size is 2, the pointer is built by appending the next three bytes to the
|
530
|
+
last three bits to produce a 27-bit value + 526336.
|
531
|
+
|
532
|
+
Finally, if the size is 3, the pointer's value is contained in the next four
|
533
|
+
bytes as a 32-bit value. In this case, the last three bits of the control byte
|
534
|
+
are ignored.
|
535
|
+
|
536
|
+
This means that we are limited to 4GB of address space for pointers, so the
|
537
|
+
data section size for the database is limited to 4GB.
|
538
|
+
|
539
|
+
## Reference Implementations
|
540
|
+
|
541
|
+
### Writer
|
542
|
+
|
543
|
+
* [Perl](https://github.com/maxmind/MaxMind-DB-Writer-perl)
|
544
|
+
|
545
|
+
### Reader
|
546
|
+
|
547
|
+
* [C](https://github.com/maxmind/libmaxminddb)
|
548
|
+
* [C#](https://github.com/maxmind/MaxMind-DB-Reader-dotnet)
|
549
|
+
* [Java](https://github.com/maxmind/MaxMind-DB-Reader-java)
|
550
|
+
* [Perl](https://github.com/maxmind/MaxMind-DB-Reader-perl)
|
551
|
+
* [PHP](https://github.com/maxmind/MaxMind-DB-Reader-php)
|
552
|
+
* [Python](https://github.com/maxmind/MaxMind-DB-Reader-python)
|
553
|
+
* [Ruby](https://github.com/maxmind/MaxMind-DB-Reader-ruby)
|
554
|
+
|
555
|
+
## Authors
|
556
|
+
|
557
|
+
This specification was created by the following authors:
|
558
|
+
|
559
|
+
* Greg Oschwald \<goschwald@maxmind.com\>
|
560
|
+
* Dave Rolsky \<drolsky@maxmind.com\>
|
561
|
+
* Boris Zentner \<bzentner@maxmind.com\>
|
562
|
+
|
563
|
+
## License
|
564
|
+
|
565
|
+
This work is licensed under the Creative Commons Attribution-ShareAlike 3.0
|
566
|
+
Unported License. To view a copy of this license, visit
|
567
|
+
[http://creativecommons.org/licenses/by-sa/3.0/](http://creativecommons.org/licenses/by-sa/3.0/)
|
568
|
+
or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain
|
569
|
+
View, California, 94041, USA
|
570
|
+
|