maxmind-db 1.1.0 → 1.1.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (78) hide show
  1. checksums.yaml +5 -5
  2. data/CHANGELOG.md +5 -0
  3. data/Gemfile +1 -4
  4. data/README.dev.md +6 -2
  5. data/README.md +1 -9
  6. data/lib/maxmind/db.rb +38 -34
  7. data/lib/maxmind/db/decoder.rb +4 -2
  8. data/lib/maxmind/db/errors.rb +1 -1
  9. data/lib/maxmind/db/file_reader.rb +3 -2
  10. data/lib/maxmind/db/memory_reader.rb +4 -4
  11. data/lib/maxmind/db/metadata.rb +28 -2
  12. data/maxmind-db.gemspec +7 -2
  13. data/test/data/LICENSE +4 -0
  14. data/test/data/MaxMind-DB-spec.md +570 -0
  15. data/test/data/MaxMind-DB-test-metadata-pointers.mmdb +0 -0
  16. data/test/data/README.md +4 -0
  17. data/test/data/bad-data/README.md +7 -0
  18. data/test/data/bad-data/libmaxminddb/libmaxminddb-offset-integer-overflow.mmdb +0 -0
  19. data/test/data/bad-data/maxminddb-golang/cyclic-data-structure.mmdb +0 -0
  20. data/test/data/bad-data/maxminddb-golang/invalid-bytes-length.mmdb +1 -0
  21. data/test/data/bad-data/maxminddb-golang/invalid-data-record-offset.mmdb +0 -0
  22. data/test/data/bad-data/maxminddb-golang/invalid-map-key-length.mmdb +0 -0
  23. data/test/data/bad-data/maxminddb-golang/invalid-string-length.mmdb +1 -0
  24. data/test/data/bad-data/maxminddb-golang/metadata-is-an-uint128.mmdb +1 -0
  25. data/test/data/bad-data/maxminddb-golang/unexpected-bytes.mmdb +0 -0
  26. data/test/data/perltidyrc +12 -0
  27. data/test/data/source-data/GeoIP2-Anonymous-IP-Test.json +48 -0
  28. data/test/data/source-data/GeoIP2-City-Test.json +12852 -0
  29. data/test/data/source-data/GeoIP2-Connection-Type-Test.json +102 -0
  30. data/test/data/source-data/GeoIP2-Country-Test.json +15916 -0
  31. data/test/data/source-data/GeoIP2-DensityIncome-Test.json +14 -0
  32. data/test/data/source-data/GeoIP2-Domain-Test.json +452 -0
  33. data/test/data/source-data/GeoIP2-Enterprise-Test.json +673 -0
  34. data/test/data/source-data/GeoIP2-ISP-Test.json +12593 -0
  35. data/test/data/source-data/GeoIP2-Precision-Enterprise-Test.json +2018 -0
  36. data/test/data/source-data/GeoIP2-Static-IP-Score-Test.json +2132 -0
  37. data/test/data/source-data/GeoIP2-User-Count-Test.json +2837 -0
  38. data/test/data/source-data/GeoLite2-ASN-Test.json +37 -0
  39. data/test/data/source-data/README +15 -0
  40. data/test/data/test-data/GeoIP2-Anonymous-IP-Test.mmdb +0 -0
  41. data/test/data/test-data/GeoIP2-City-Test-Broken-Double-Format.mmdb +0 -0
  42. data/test/data/test-data/GeoIP2-City-Test-Invalid-Node-Count.mmdb +0 -0
  43. data/test/data/test-data/GeoIP2-City-Test.mmdb +0 -0
  44. data/test/data/test-data/GeoIP2-Connection-Type-Test.mmdb +0 -0
  45. data/test/data/test-data/GeoIP2-Country-Test.mmdb +0 -0
  46. data/test/data/test-data/GeoIP2-DensityIncome-Test.mmdb +0 -0
  47. data/test/data/test-data/GeoIP2-Domain-Test.mmdb +0 -0
  48. data/test/data/test-data/GeoIP2-Enterprise-Test.mmdb +0 -0
  49. data/test/data/test-data/GeoIP2-ISP-Test.mmdb +0 -0
  50. data/test/data/test-data/GeoIP2-Precision-Enterprise-Test.mmdb +0 -0
  51. data/test/data/test-data/GeoIP2-Static-IP-Score-Test.mmdb +0 -0
  52. data/test/data/test-data/GeoIP2-User-Count-Test.mmdb +0 -0
  53. data/test/data/test-data/GeoLite2-ASN-Test.mmdb +0 -0
  54. data/test/data/test-data/MaxMind-DB-no-ipv4-search-tree.mmdb +0 -0
  55. data/test/data/test-data/MaxMind-DB-string-value-entries.mmdb +0 -0
  56. data/test/data/test-data/MaxMind-DB-test-broken-pointers-24.mmdb +0 -0
  57. data/test/data/test-data/MaxMind-DB-test-broken-search-tree-24.mmdb +0 -0
  58. data/test/data/test-data/MaxMind-DB-test-decoder.mmdb +0 -0
  59. data/test/data/test-data/MaxMind-DB-test-ipv4-24.mmdb +0 -0
  60. data/test/data/test-data/MaxMind-DB-test-ipv4-28.mmdb +0 -0
  61. data/test/data/test-data/MaxMind-DB-test-ipv4-32.mmdb +0 -0
  62. data/test/data/test-data/MaxMind-DB-test-ipv6-24.mmdb +0 -0
  63. data/test/data/test-data/MaxMind-DB-test-ipv6-28.mmdb +0 -0
  64. data/test/data/test-data/MaxMind-DB-test-ipv6-32.mmdb +0 -0
  65. data/test/data/test-data/MaxMind-DB-test-metadata-pointers.mmdb +0 -0
  66. data/test/data/test-data/MaxMind-DB-test-mixed-24.mmdb +0 -0
  67. data/test/data/test-data/MaxMind-DB-test-mixed-28.mmdb +0 -0
  68. data/test/data/test-data/MaxMind-DB-test-mixed-32.mmdb +0 -0
  69. data/test/data/test-data/MaxMind-DB-test-nested.mmdb +0 -0
  70. data/test/data/test-data/README.md +26 -0
  71. data/test/data/test-data/maps-with-pointers.raw +0 -0
  72. data/test/data/test-data/write-test-data.pl +640 -0
  73. data/test/data/tidyall.ini +5 -0
  74. data/test/mmdb_util.rb +1 -1
  75. data/test/test_decoder.rb +1 -1
  76. data/test/test_reader.rb +14 -1
  77. metadata +122 -6
  78. data/Gemfile.lock +0 -34
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: e8404a64573590d7a01fd9199ae1be5bb1892407
4
- data.tar.gz: cfdec2ef12cae84fa914784b90c386d1006fdf47
2
+ SHA256:
3
+ metadata.gz: 485d10049eaba35df9d49f7aa4637857a3fe84f38dec3e2a163c675fc8f2e474
4
+ data.tar.gz: 964008df8d8bc5bc8c558fb004cbb7eda6843295cb3abd5fea3fc193da58981d
5
5
  SHA512:
6
- metadata.gz: 263eff4235eadc529e14266772c3db95bc17950ebc7d73fbb3f5d20ee76f3b2f50035c97ad2e39454e5dbd65dd6093fc56f3a6dd3b8d91a69037b8e396986366
7
- data.tar.gz: 8fe05ac4631539293ee2d1485fe7c1908858ef952d76e60e75d9f601f356263bb47928bc8dd670c2989df4437cb0d2a876869c662b1f4aab36d0e8a81ce19552
6
+ metadata.gz: 92181b893b438f63a874d915817f4e325aa4a20fb12e8e373e6d6f4885235472353391db4508f313f7d4b5b9ab33e6f09cd05a5b88e718b59dc352dc18f415be
7
+ data.tar.gz: a03201ffac72cac0228545ce2f1a3a5e3d84ad807d33fac18bfffd3da4a5be69d0bceee96c5c87429b394410ed3e3efc17f20893700b7811558c7a383d4773e4
@@ -1,5 +1,10 @@
1
1
  # Changelog
2
2
 
3
+ ## 1.1.1 (2020-06-23)
4
+
5
+ * Fixed the memory reader's inspect method to no longer attempt to modify a
6
+ frozen string. Pull request by Tietew. GitHub #35.
7
+
3
8
  ## 1.1.0 (2020-01-08)
4
9
 
5
10
  * The method `get_with_prefix_length` was added. This method returns both
data/Gemfile CHANGED
@@ -2,7 +2,4 @@
2
2
 
3
3
  source 'https://rubygems.org'
4
4
 
5
- gem 'minitest', group: :development
6
- gem 'rake', group: :development
7
- gem 'rubocop', group: :development
8
- gem 'rubocop-performance', group: :development
5
+ gemspec
@@ -1,11 +1,15 @@
1
1
  # How to release
2
- * Update changelog and set release date
3
- * Bump version in `maxmind-db.gemspec`
2
+ * Ensure tests pass: `rake`
3
+ * Update changelog: Set version and release date
4
+ * Set version in `maxmind-db.gemspec`
5
+ * Add them: `git add -p`
4
6
  * Commit: `git commit -m v1.0.0`
5
7
  * Tag: `git tag -a v1.0.0 -m v1.0.0`
6
8
  * Clean up to be sure nothing stray gets into gem: `git clean -dxff`
7
9
  * Create `.gem` file: `gem build maxmind-db.gemspec`
8
10
  * Complete prerequisites (see below)
11
+ * You only need to do this if `~/.gem/credentials` is missing
12
+ `:rubygems_api_key`.
9
13
  * Upload to rubygems.org: `gem push maxmind-db-1.0.0.gem`
10
14
  * Push: `git push`
11
15
  * Push tag: `git push --tags`
data/README.md CHANGED
@@ -13,13 +13,6 @@ file format that stores data indexed by IP address subnets (IPv4 or IPv6).
13
13
  gem install maxmind-db
14
14
  ```
15
15
 
16
- Or from source:
17
-
18
- ```
19
- gem build maxmind-db.spec
20
- gem install ./maxmind-db-xxx.gem
21
- ```
22
-
23
16
  ## Usage
24
17
 
25
18
  ```ruby
@@ -43,8 +36,7 @@ For more information see the
43
36
 
44
37
  ## Requirements
45
38
 
46
- This code requires Ruby version 2.4 or higher. Older versions may work, but
47
- are not supported.
39
+ This code requires Ruby version 2.4 or higher.
48
40
 
49
41
  ## Contributing
50
42
 
@@ -7,7 +7,7 @@ require 'maxmind/db/file_reader.rb'
7
7
  require 'maxmind/db/memory_reader.rb'
8
8
  require 'maxmind/db/metadata.rb'
9
9
 
10
- module MaxMind # :nodoc:
10
+ module MaxMind
11
11
  # DB provides a way to read {MaxMind DB
12
12
  # files}[https://maxmind.github.io/MaxMind-DB/].
13
13
  #
@@ -31,11 +31,6 @@ module MaxMind # :nodoc:
31
31
  # end
32
32
  #
33
33
  # reader.close
34
- #
35
- # == Exceptions
36
- #
37
- # DB raises an InvalidDatabaseError if the database is corrupt or invalid. It
38
- # can raise other exceptions, such as ArgumentError, if other errors occur.
39
34
  class DB
40
35
  # Choose the default method to open the database. Currently the default is
41
36
  # MODE_FILE.
@@ -47,7 +42,9 @@ module MaxMind # :nodoc:
47
42
  MODE_MEMORY = :MODE_MEMORY
48
43
  # Treat the database parameter as containing a database already read into
49
44
  # memory. It must be a binary string. This primarily exists for testing.
50
- MODE_PARAM_IS_BUFFER = :MODE_PARAM_IS_BUFFER # :nodoc:
45
+ #
46
+ # @!visibility private
47
+ MODE_PARAM_IS_BUFFER = :MODE_PARAM_IS_BUFFER
51
48
 
52
49
  DATA_SECTION_SEPARATOR_SIZE = 16
53
50
  private_constant :DATA_SECTION_SEPARATOR_SIZE
@@ -59,7 +56,9 @@ module MaxMind # :nodoc:
59
56
  private_constant :METADATA_MAX_SIZE
60
57
 
61
58
  # Return the metadata associated with the {MaxMind
62
- # DB}[https://maxmind.github.io/MaxMind-DB/] as a Metadata object.
59
+ # DB}[https://maxmind.github.io/MaxMind-DB/]
60
+ #
61
+ # @return [MaxMind::DB::Metadata]
63
62
  attr_reader :metadata
64
63
 
65
64
  # Create a DB. A DB provides a way to read {MaxMind DB
@@ -70,20 +69,20 @@ module MaxMind # :nodoc:
70
69
  # is safe to use after forking only if you use MODE_MEMORY or if your
71
70
  # version of Ruby supports IO#pread.
72
71
  #
73
- # Creating the DB may raise an exception if initialization fails.
72
+ # @param database [String] a path to a {MaxMind
73
+ # DB}[https://maxmind.github.io/MaxMind-DB/].
74
74
  #
75
- # +database+ is a path to a {MaxMind
76
- # DB}[https://maxmind.github.io/MaxMind-DB/].
75
+ # @param options [Hash<Symbol, Symbol>] options controlling the behavior of
76
+ # the DB.
77
77
  #
78
- # +options+ is an option hash where each key is a symbol. The options
79
- # control the behaviour of the DB.
78
+ # @option options [Symbol] :mode Defines how to open the database. It may
79
+ # be one of MODE_AUTO, MODE_FILE, or MODE_MEMORY. If you don't provide
80
+ # one, DB uses MODE_AUTO. Refer to the definition of those constants for
81
+ # an explanation of their meaning.
80
82
  #
81
- # The available options are:
83
+ # @raise [InvalidDatabaseError] if the database is corrupt or invalid.
82
84
  #
83
- # [+:mode+] defines how to open the database. It may be one of MODE_AUTO,
84
- # MODE_FILE, or MODE_MEMORY. If you don't provide one, DB uses
85
- # MODE_AUTO. Refer to the definition of those constants for an
86
- # explanation of their meaning.
85
+ # @raise [ArgumentError] if the mode is invalid.
87
86
  def initialize(database, options = {})
88
87
  options[:mode] = MODE_AUTO unless options.key?(:mode)
89
88
 
@@ -125,35 +124,44 @@ module MaxMind # :nodoc:
125
124
  end
126
125
  end
127
126
 
128
- # Return the record for the +ip_address+ in the {MaxMind
127
+ # Return the record for the IP address in the {MaxMind
129
128
  # DB}[https://maxmind.github.io/MaxMind-DB/]. The record can be one of
130
129
  # several types and depends on the contents of the database.
131
130
  #
132
- # If no record is found for +ip_address+, +get+ returns +nil+.
131
+ # If no record is found for the IP address, +get+ returns +nil+.
132
+ #
133
+ # @param ip_address [String] a string in the standard notation. It may be
134
+ # IPv4 or IPv6.
133
135
  #
134
- # +get+ raises an exception if there is an error performing the lookup.
136
+ # @raise [ArgumentError] if you attempt to look up an IPv6 address in an
137
+ # IPv4-only database.
135
138
  #
136
- # +ip_address+ is a string in the standard notation. It may be IPv4 or
137
- # IPv6.
139
+ # @raise [InvalidDatabaseError] if the database is corrupt or invalid.
140
+ #
141
+ # @return [Object, nil]
138
142
  def get(ip_address)
139
143
  record, = get_with_prefix_length(ip_address)
140
144
 
141
145
  record
142
146
  end
143
147
 
144
- # Return an array containing the record for the +ip_address+ in the
148
+ # Return an array containing the record for the IP address in the
145
149
  # {MaxMind DB}[https://maxmind.github.io/MaxMind-DB/] and its associated
146
150
  # network prefix length. The record can be one of several types and
147
151
  # depends on the contents of the database.
148
152
  #
149
- # If no record is found for +ip_address+, the record will be +nil+ and
153
+ # If no record is found for the IP address, the record will be +nil+ and
150
154
  # the prefix length will be the value for the missing network.
151
155
  #
152
- # +get_with_prefix_length+ raises an exception if there is an error
153
- # performing the lookup.
156
+ # @param ip_address [String] a string in the standard notation. It may be
157
+ # IPv4 or IPv6.
158
+ #
159
+ # @raise [ArgumentError] if you attempt to look up an IPv6 address in an
160
+ # IPv4-only database.
154
161
  #
155
- # +ip_address+ is a string in the standard notation. It may be IPv4 or
156
- # IPv6.
162
+ # @raise [InvalidDatabaseError] if the database is corrupt or invalid.
163
+ #
164
+ # @return [Array<(Object, Integer)>]
157
165
  def get_with_prefix_length(ip_address)
158
166
  ip = IPAddr.new(ip_address)
159
167
  # We could check the IP has the correct prefix (32 or 128) but I do not
@@ -221,8 +229,6 @@ module MaxMind # :nodoc:
221
229
 
222
230
  # Read a record from the indicated node. Index indicates whether it's the
223
231
  # left (0) or right (1) record.
224
- #
225
- # rubocop:disable Metrics/CyclomaticComplexity
226
232
  def read_node(node_number, index)
227
233
  base_offset = node_number * @node_byte_size
228
234
 
@@ -253,7 +259,6 @@ module MaxMind # :nodoc:
253
259
 
254
260
  raise InvalidDatabaseError, "Unsupported record size: #{@record_size}"
255
261
  end
256
- # rubocop:enable Metrics/CyclomaticComplexity
257
262
 
258
263
  def resolve_data_pointer(pointer)
259
264
  offset_in_file = pointer - @node_count + @search_tree_size
@@ -290,8 +295,7 @@ module MaxMind # :nodoc:
290
295
 
291
296
  # Close the DB and return resources to the system.
292
297
  #
293
- # There is no useful return value. #close raises an exception if there is
294
- # an error.
298
+ # @return [void]
295
299
  def close
296
300
  @io.close
297
301
  end
@@ -2,14 +2,16 @@
2
2
 
3
3
  require 'maxmind/db/errors'
4
4
 
5
- module MaxMind # :nodoc:
5
+ module MaxMind
6
6
  class DB
7
7
  # +Decoder+ decodes a {MaxMind DB}[https://maxmind.github.io/MaxMind-DB/]
8
8
  # data section.
9
9
  #
10
10
  # Typically you will interact with this class through a Reader rather than
11
11
  # directly.
12
- class Decoder # :nodoc:
12
+ #
13
+ # @!visibility private
14
+ class Decoder
13
15
  # Create a +Decoder+.
14
16
  #
15
17
  # +io+ is the DB. It must provide a +read+ method. It must be opened in
@@ -1,6 +1,6 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- module MaxMind # :nodoc:
3
+ module MaxMind
4
4
  class DB
5
5
  # An InvalidDatabaseError means the {MaxMind
6
6
  # DB}[https://maxmind.github.io/MaxMind-DB/] file is corrupt or invalid.
@@ -2,9 +2,10 @@
2
2
 
3
3
  require 'maxmind/db/errors'
4
4
 
5
- module MaxMind # :nodoc:
5
+ module MaxMind
6
6
  class DB
7
- class FileReader # :nodoc:
7
+ # @!visibility private
8
+ class FileReader
8
9
  def initialize(filename)
9
10
  @fh = File.new(filename, 'rb')
10
11
  @size = @fh.size
@@ -1,8 +1,9 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- module MaxMind # :nodoc:
3
+ module MaxMind
4
4
  class DB
5
- class MemoryReader # :nodoc:
5
+ # @!visibility private
6
+ class MemoryReader
6
7
  def initialize(filename, options = {})
7
8
  if options[:is_buffer]
8
9
  @buf = filename
@@ -18,8 +19,7 @@ module MaxMind # :nodoc:
18
19
 
19
20
  # Override to not show @buf in inspect to avoid showing it in irb.
20
21
  def inspect
21
- s = "#<#{self.class.name}:0x#{self.class.object_id.to_s(16)} "
22
- s << '@size=' << @size.inspect << '>'
22
+ "#<#{self.class.name}:0x#{self.class.object_id.to_s(16)}, @size=#{@size.inspect}>"
23
23
  end
24
24
 
25
25
  def close; end
@@ -1,42 +1,64 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- module MaxMind # :nodoc:
3
+ module MaxMind
4
4
  class DB
5
5
  # Metadata holds metadata about a {MaxMind
6
- # DB}[https://maxmind.github.io/MaxMind-DB/] file.
6
+ # DB}[https://maxmind.github.io/MaxMind-DB/] file. See
7
+ # https://maxmind.github.io/MaxMind-DB/#database-metadata for the
8
+ # specification.
7
9
  class Metadata
8
10
  # The number of nodes in the database.
11
+ #
12
+ # @return [Integer]
9
13
  attr_reader :node_count
10
14
 
11
15
  # The bit size of a record in the search tree.
16
+ #
17
+ # @return [Integer]
12
18
  attr_reader :record_size
13
19
 
14
20
  # The IP version of the data in the database. A value of 4 means the
15
21
  # database only supports IPv4. A database with a value of 6 may support
16
22
  # both IPv4 and IPv6 lookups.
23
+ #
24
+ # @return [Integer]
17
25
  attr_reader :ip_version
18
26
 
19
27
  # A string identifying the database type. e.g., "GeoIP2-City".
28
+ #
29
+ # @return [String]
20
30
  attr_reader :database_type
21
31
 
22
32
  # An array of locale codes supported by the database.
33
+ #
34
+ # @return [Array<String>]
23
35
  attr_reader :languages
24
36
 
25
37
  # The major version number of the binary format used when creating the
26
38
  # database.
39
+ #
40
+ # @return [Integer]
27
41
  attr_reader :binary_format_major_version
28
42
 
29
43
  # The minor version number of the binary format used when creating the
30
44
  # database.
45
+ #
46
+ # @return [Integer]
31
47
  attr_reader :binary_format_minor_version
32
48
 
33
49
  # The Unix epoch for the build time of the database.
50
+ #
51
+ # @return [Integer]
34
52
  attr_reader :build_epoch
35
53
 
36
54
  # A hash from locales to text descriptions of the database.
55
+ #
56
+ # @return [Hash<String, String>]
37
57
  attr_reader :description
38
58
 
39
59
  # +m+ is a hash representing the metadata map.
60
+ #
61
+ # @!visibility private
40
62
  def initialize(map)
41
63
  @node_count = map['node_count']
42
64
  @record_size = map['record_size']
@@ -50,11 +72,15 @@ module MaxMind # :nodoc:
50
72
  end
51
73
 
52
74
  # The size of a node in bytes.
75
+ #
76
+ # @return [Integer]
53
77
  def node_byte_size
54
78
  @record_size / 4
55
79
  end
56
80
 
57
81
  # The size of the search tree in bytes.
82
+ #
83
+ # @return [Integer]
58
84
  def search_tree_size
59
85
  @node_count * node_byte_size
60
86
  end
@@ -5,7 +5,7 @@ Gem::Specification.new do |s|
5
5
  s.files = Dir['**/*']
6
6
  s.name = 'maxmind-db'
7
7
  s.summary = 'A gem for reading MaxMind DB files.'
8
- s.version = '1.1.0'
8
+ s.version = '1.1.1'
9
9
 
10
10
  s.description = 'A gem for reading MaxMind DB files. MaxMind DB is a binary file format that stores data indexed by IP address subnets (IPv4 or IPv6).'
11
11
  s.email = 'support@maxmind.com'
@@ -14,9 +14,14 @@ Gem::Specification.new do |s|
14
14
  s.metadata = {
15
15
  'bug_tracker_uri' => 'https://github.com/maxmind/MaxMind-DB-Reader-ruby/issues',
16
16
  'changelog_uri' => 'https://github.com/maxmind/MaxMind-DB-Reader-ruby/blob/master/CHANGELOG.md',
17
- 'documentation_uri' => 'https://github.com/maxmind/MaxMind-DB-Reader-ruby',
17
+ 'documentation_uri' => 'https://www.rubydoc.info/gems/maxmind-db',
18
18
  'homepage_uri' => 'https://github.com/maxmind/MaxMind-DB-Reader-ruby',
19
19
  'source_code_uri' => 'https://github.com/maxmind/MaxMind-DB-Reader-ruby',
20
20
  }
21
21
  s.required_ruby_version = '>= 2.4.0'
22
+
23
+ s.add_development_dependency 'minitest'
24
+ s.add_development_dependency 'rake'
25
+ s.add_development_dependency 'rubocop'
26
+ s.add_development_dependency 'rubocop-performance'
22
27
  end
@@ -0,0 +1,4 @@
1
+ This work is licensed under the Creative Commons Attribution-ShareAlike 3.0
2
+ Unported License. To view a copy of this license, visit
3
+ http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative
4
+ Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
@@ -0,0 +1,570 @@
1
+ ---
2
+ layout: default
3
+ title: MaxMind DB File Format Specification
4
+ version: v2.0
5
+ ---
6
+ # MaxMind DB File Format Specification
7
+
8
+ ## Description
9
+
10
+ The MaxMind DB file format is a database format that maps IPv4 and IPv6
11
+ addresses to data records using an efficient binary search tree.
12
+
13
+ ## Version
14
+
15
+ This spec documents **version 2.0** of the MaxMind DB binary format.
16
+
17
+ The version number consists of separate major and minor version numbers. It
18
+ should not be considered a decimal number. In other words, version 2.10 comes
19
+ after version 2.9.
20
+
21
+ Code which is capable of reading a given major version of the format should
22
+ not be broken by minor version changes to the format.
23
+
24
+ ## Overview
25
+
26
+ The binary database is split into three parts:
27
+
28
+ 1. The binary search tree. Each level of the tree corresponds to a single bit
29
+ in the 128 bit representation of an IPv6 address.
30
+ 2. The data section. These are the values returned to the client for a
31
+ specific IP address, e.g. "US", "New York", or a more complex map type made up
32
+ of multiple fields.
33
+ 3. Database metadata. Information about the database itself.
34
+
35
+ ## Database Metadata
36
+
37
+ This portion of the database is stored at the end of the file. It is
38
+ documented first because understanding some of the metadata is key to
39
+ understanding how the other sections work.
40
+
41
+ This section can be found by looking for a binary sequence matching
42
+ "\xab\xcd\xefMaxMind.com". The *last* occurrence of this string in the file
43
+ marks the end of the data section and the beginning of the metadata. Since we
44
+ allow for arbitrary binary data in the data section, some other piece of data
45
+ could contain these values. This is why you need to find the last occurrence
46
+ of this sequence.
47
+
48
+ The maximum allowable size for the metadata section, including the marker that
49
+ starts the metadata, is 128KiB.
50
+
51
+ The metadata is stored as a map data structure. This structure is described
52
+ later in the spec. Changing a key's data type or removing a key would
53
+ constitute a major version change for this spec.
54
+
55
+ Except where otherwise specified, each key listed is required for the database
56
+ to be considered valid.
57
+
58
+ Adding a key constitutes a minor version change. Removing a key or changing
59
+ its type constitutes a major version change.
60
+
61
+ The list of known keys for the current version of the format is as follows:
62
+
63
+ ### node\_count
64
+
65
+ This is an unsigned 32-bit integer indicating the number of nodes in the
66
+ search tree.
67
+
68
+ ### record\_size
69
+
70
+ This is an unsigned 16-bit integer. It indicates the number of bits in a
71
+ record in the search tree. Note that each node consists of *two* records.
72
+
73
+ ### ip\_version
74
+
75
+ This is an unsigned 16-bit integer which is always 4 or 6. It indicates
76
+ whether the database contains IPv4 or IPv6 address data.
77
+
78
+ ### database\_type
79
+
80
+ This is a string that indicates the structure of each data record associated
81
+ with an IP address. The actual definition of these structures is left up to
82
+ the database creator.
83
+
84
+ Names starting with "GeoIP" are reserved for use by MaxMind (and "GeoIP" is a
85
+ trademark anyway).
86
+
87
+ ### languages
88
+
89
+ An array of strings, each of which is a locale code. A given record may
90
+ contain data items that have been localized to some or all of these
91
+ locales. Records should not contain localized data for locales not included in
92
+ this array.
93
+
94
+ This is an optional key, as this may not be relevant for all types of data.
95
+
96
+ ### binary\_format\_major\_version
97
+
98
+ This is an unsigned 16-bit integer indicating the major version number for the
99
+ database's binary format.
100
+
101
+ ### binary\_format\_minor\_version
102
+
103
+ This is an unsigned 16-bit integer indicating the minor version number for the
104
+ database's binary format.
105
+
106
+ ### build\_epoch
107
+
108
+ This is an unsigned 64-bit integer that contains the database build timestamp
109
+ as a Unix epoch value.
110
+
111
+ ### description
112
+
113
+ This key will always point to a map. The keys of that map will be language
114
+ codes, and the values will be a description in that language as a UTF-8
115
+ string.
116
+
117
+ The codes may include additional information such as script or country
118
+ identifiers, like "zh-TW" or "mn-Cyrl-MN". The additional identifiers will be
119
+ separated by a dash character ("-").
120
+
121
+ This key is optional. However, creators of databases are strongly
122
+ encouraged to include a description in at least one language.
123
+
124
+ ### Calculating the Search Tree Section Size
125
+
126
+ The formula for calculating the search tree section size *in bytes* is as
127
+ follows:
128
+
129
+ ( ( $record_size * 2 ) / 8 ) * $number_of_nodes
130
+
131
+ The end of the search tree marks the beginning of the data section.
132
+
133
+ ## Binary Search Tree Section
134
+
135
+ The database file starts with a binary search tree. The number of nodes in the
136
+ tree is dependent on how many unique netblocks are needed for the particular
137
+ database. For example, the city database needs many more small netblocks than
138
+ the country database.
139
+
140
+ The top most node is always located at the beginning of the search tree
141
+ section's address space. The top node is node 0.
142
+
143
+ Each node consists of two records, each of which is a pointer to an address in
144
+ the file.
145
+
146
+ The pointers can point to one of three things. First, it may point to another
147
+ node in the search tree address space. These pointers are followed as part of
148
+ the IP address search algorithm, described below.
149
+
150
+ The pointer can point to a value equal to `$number_of_nodes`. If this is the
151
+ case, it means that the IP address we are searching for is not in the
152
+ database.
153
+
154
+ Finally, it may point to an address in the data section. This is the data
155
+ relevant to the given netblock.
156
+
157
+ ### Node Layout
158
+
159
+ Each node in the search tree consists of two records, each of which is a
160
+ pointer. The record size varies by database, but inside a single database node
161
+ records are always the same size. A record may be anywhere from 24 to 128 bits
162
+ long, depending on the number of nodes in the tree. These pointers are
163
+ stored in big-endian format (most significant byte first).
164
+
165
+ Here are some examples of how the records are laid out in a node for 24, 28,
166
+ and 32 bit records. Larger record sizes follow this same pattern.
167
+
168
+ #### 24 bits (small database), one node is 6 bytes
169
+
170
+ | <------------- node --------------->|
171
+ | 23 .. 0 | 23 .. 0 |
172
+
173
+ #### 28 bits (medium database), one node is 7 bytes
174
+
175
+ | <------------- node --------------->|
176
+ | 23 .. 0 | 27..24 | 27..24 | 23 .. 0 |
177
+
178
+ Note 4 bits of each pointer are combined into the middle byte. For both
179
+ records, they are prepended and end up in the most significant position.
180
+
181
+ #### 32 bits (large database), one node is 8 bytes
182
+
183
+ | <------------- node --------------->|
184
+ | 31 .. 0 | 31 .. 0 |
185
+
186
+ ### Search Lookup Algorithm
187
+
188
+ The first step is to convert the IP address to its big-endian binary
189
+ representation. For an IPv4 address, this becomes 32 bits. For IPv6 you get
190
+ 128 bits.
191
+
192
+ The leftmost bit corresponds to the first node in the search tree. For each
193
+ bit, a value of 0 means we choose the left record in a node, and a value of 1
194
+ means we choose the right record.
195
+
196
+ The record value is always interpreted as an unsigned integer. The maximum
197
+ size of the integer is dependent on the number of bits in a record (24, 28, or
198
+ 32).
199
+
200
+ If the record value is a number that is less than the *number of nodes* (not
201
+ in bytes, but the actual node count) in the search tree (this is stored in the
202
+ database metadata), then the value is a node number. In this case, we find
203
+ that node in the search tree and repeat the lookup algorithm from there.
204
+
205
+ If the record value is equal to the number of nodes, that means that we do not
206
+ have any data for the IP address, and the search ends here.
207
+
208
+ If the record value is *greater* than the number of nodes in the search tree,
209
+ then it is an actual pointer value pointing into the data section. The value
210
+ of the pointer is relative to the start of the data section, *not* the
211
+ start of the file.
212
+
213
+ In order to determine where in the data section we should start looking, we use
214
+ the following formula:
215
+
216
+ $data_section_offset = ( $record_value - $node_count ) - 16
217
+
218
+ The 16 is the size of the data section separator. We subtract it because we
219
+ want to permit pointing to the first byte of the data section. Recall that
220
+ the record value cannot equal the node count as that means there is no
221
+ data. Instead, we choose to start values that go to the data section at
222
+ `$node_count + 16`. (This has the side effect that record values
223
+ `$node_count + 1` through `$node_count + 15` inclusive are not valid).
224
+
225
+ This is best demonstrated by an example:
226
+
227
+ Let's assume we have a 24-bit tree with 1,000 nodes. Each node contains 48
228
+ bits, or 6 bytes. The size of the tree is 6,000 bytes.
229
+
230
+ When a record in the tree contains a number that is less than 1,000, this
231
+ is a *node number*, and we look up that node. If a record contains a value
232
+ greater than or equal to 1,016, we know that it is a data section value. We
233
+ subtract the node count (1,000) and then subtract 16 for the data section
234
+ separator, giving us the number 0, the first byte of the data section.
235
+
236
+ If a record contained the value 6,000, this formula would give us an offset of
237
+ 4,984 into the data section.
238
+
239
+ In order to determine where in the file this offset really points to, we also
240
+ need to know where the data section starts. This can be calculated by
241
+ determining the size of the search tree in bytes and then adding an additional
242
+ 16 bytes for the data section separator:
243
+
244
+ $offset_in_file = $data_section_offset
245
+ + $search_tree_size_in_bytes
246
+ + 16
247
+
248
+ Since we subtract and then add 16, the final formula to determine the
249
+ offset in the file can be simplified to:
250
+
251
+ $offset_in_file = ( $record_value - $node_count )
252
+ + $search_tree_size_in_bytes
253
+
254
+ ### IPv4 addresses in an IPv6 tree
255
+
256
+ When storing IPv4 addresses in an IPv6 tree, they are stored as-is, so they
257
+ occupy the first 32-bits of the address space (from 0 to 2**32 - 1).
258
+
259
+ Creators of databases should decide on a strategy for handling the various
260
+ mappings between IPv4 and IPv6.
261
+
262
+ The strategy that MaxMind uses for its GeoIP databases is to include a pointer
263
+ from the `::ffff:0:0/96` subnet to the root node of the IPv4 address space in
264
+ the tree. This accounts for the
265
+ [IPv4-mapped IPv6 address](http://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses).
266
+
267
+ MaxMind also includes a pointer from the `2002::/16` subnet to the root node
268
+ of the IPv4 address space in the tree. This accounts for the
269
+ [6to4 mapping](http://en.wikipedia.org/wiki/6to4) subnet.
270
+
271
+ Database creators are encouraged to document whether they are doing something
272
+ similar for their databases.
273
+
274
+ The Teredo subnet cannot be accounted for in the tree. Instead, code that
275
+ searches the tree can offer to decode the IPv4 portion of a Teredo address and
276
+ look that up.
277
+
278
+ ## Data Section Separator
279
+
280
+ There are 16 bytes of NULLs in between the search tree and the data
281
+ section. This separator exists in order to make it possible for a verification
282
+ tool to distinguish between the two sections.
283
+
284
+ This separator is not considered part of the data section itself. In other
285
+ words, the data section starts at `$size_of_search_tree + 16` bytes in the
286
+ file.
287
+
288
+ ## Output Data Section
289
+
290
+ Each output data field has an associated type, and that type is encoded as a
291
+ number that begins the data field. Some types are variable length. In those
292
+ cases, the type indicator is also followed by a length. The data payload
293
+ always comes at the end of the field.
294
+
295
+ All binary data is stored in big-endian format.
296
+
297
+ Note that the *interpretation* of a given data type's meaning is decided by
298
+ higher-level APIs, not by the binary format itself.
299
+
300
+ ### pointer - 1
301
+
302
+ A pointer to another part of the data section's address space. The pointer
303
+ will point to the beginning of a field. It is illegal for a pointer to point
304
+ to another pointer.
305
+
306
+ Pointer values start from the beginning of the data section, *not* the
307
+ beginning of the file.
308
+
309
+ ### UTF-8 string - 2
310
+
311
+ A variable length byte sequence that contains valid utf8. If the length is
312
+ zero then this is an empty string.
313
+
314
+ ### double - 3
315
+
316
+ This is stored as an IEEE-754 double (binary64) in big-endian format. The
317
+ length of a double is always 8 bytes.
318
+
319
+ ### bytes - 4
320
+
321
+ A variable length byte sequence containing any sort of binary data. If the
322
+ length is zero then this a zero-length byte sequence.
323
+
324
+ This is not currently used but may be used in the future to embed non-text
325
+ data (images, etc.).
326
+
327
+ ### integer formats
328
+
329
+ Integers are stored in variable length binary fields.
330
+
331
+ We support 16-bit, 32-bit, 64-bit, and 128-bit unsigned integers. We also
332
+ support 32-bit signed integers.
333
+
334
+ A 128-bit integer can use up to 16 bytes, but may use fewer. Similarly, a
335
+ 32-bit integer may use from 0-4 bytes. The number of bytes used is determined
336
+ by the length specifier in the control byte. See below for details.
337
+
338
+ A length of zero always indicates the number 0.
339
+
340
+ When storing a signed integer, the left-most bit is the sign. A 1 is negative
341
+ and a 0 is positive.
342
+
343
+ The type numbers for our integer types are:
344
+
345
+ * unsigned 16-bit int - 5
346
+ * unsigned 32-bit int - 6
347
+ * signed 32-bit int - 8
348
+ * unsigned 64-bit int - 9
349
+ * unsigned 128-bit int - 10
350
+
351
+ The unsigned 32-bit and 128-bit types may be used to store IPv4 and IPv6
352
+ addresses, respectively.
353
+
354
+ The signed 32-bit integers are stored using the 2's complement representation.
355
+
356
+ ### map - 7
357
+
358
+ A map data type contains a set of key/value pairs. Unlike other data types,
359
+ the length information for maps indicates how many key/value pairs it
360
+ contains, not its length in bytes. This size can be zero.
361
+
362
+ See below for the algorithm used to determine the number of pairs in the
363
+ hash. This algorithm is also used to determine the length of a field's
364
+ payload.
365
+
366
+ ### array - 11
367
+
368
+ An array type contains a set of ordered values. The length information for
369
+ arrays indicates how many values it contains, not its length in bytes. This
370
+ size can be zero.
371
+
372
+ This type uses the same algorithm as maps for determining the length of a
373
+ field's payload.
374
+
375
+ ### data cache container - 12
376
+
377
+ This is a special data type that marks a container used to cache repeated
378
+ data. For example, instead of repeating the string "United States" over and
379
+ over in the database, we store it in the cache container and use pointers
380
+ *into* this container instead.
381
+
382
+ Nothing in the database will ever contain a pointer to this field
383
+ itself. Instead, various fields will point into the container.
384
+
385
+ The primary reason for making this a separate data type versus simply inlining
386
+ the cached data is so that a database dumper tool can skip this cache when
387
+ dumping the data section. The cache contents will end up being dumped as
388
+ pointers into it are followed.
389
+
390
+ ### end marker - 13
391
+
392
+ The end marker marks the end of the data section. It is not strictly
393
+ necessary, but including this marker allows a data section deserializer to
394
+ process a stream of input, rather than having to find the end of the section
395
+ before beginning the deserialization.
396
+
397
+ This data type is not followed by a payload, and its size is always zero.
398
+
399
+ ### boolean - 14
400
+
401
+ A true or false value. The length information for a boolean type will always
402
+ be 0 or 1, indicating the value. There is no payload for this field.
403
+
404
+ ### float - 15
405
+
406
+ This is stored as an IEEE-754 float (binary32) in big-endian format. The
407
+ length of a float is always 4 bytes.
408
+
409
+ This type is provided primarily for completeness. Because of the way floating
410
+ point numbers are stored, this type can easily lose precision when serialized
411
+ and then deserialized. If this is an issue for you, consider using a double
412
+ instead.
413
+
414
+ ### Data Field Format
415
+
416
+ Each field starts with a control byte. This control byte provides information
417
+ about the field's data type and payload size.
418
+
419
+ The first three bits of the control byte tell you what type the field is. If
420
+ these bits are all 0, then this is an "extended" type, which means that the
421
+ *next* byte contains the actual type. Otherwise, the first three bits will
422
+ contain a number from 1 to 7, the actual type for the field.
423
+
424
+ We've tried to assign the most commonly used types as numbers 1-7 as an
425
+ optimization.
426
+
427
+ With an extended type, the type number in the second byte is the number
428
+ minus 7. In other words, an array (type 11) will be stored with a 0 for the
429
+ type in the first byte and a 4 in the second.
430
+
431
+ Here is an example of how the control byte may combine with the next byte to
432
+ tell us the type:
433
+
434
+ 001XXXXX pointer
435
+ 010XXXXX UTF-8 string
436
+ 110XXXXX unsigned 32-bit int (ASCII)
437
+ 000XXXXX 00000011 unsigned 128-bit int (binary)
438
+ 000XXXXX 00000100 array
439
+ 000XXXXX 00000110 end marker
440
+
441
+ #### Payload Size
442
+
443
+ The next five bits in the control byte tell you how long the data field's
444
+ payload is, except for maps and pointers. Maps and pointers use this size
445
+ information a bit differently. See below.
446
+
447
+ If the five bits are smaller than 29, then those bits are the payload size in
448
+ bytes. For example:
449
+
450
+ 01000010 UTF-8 string - 2 bytes long
451
+ 01011100 UTF-8 string - 28 bytes long
452
+ 11000001 unsigned 32-bit int - 1 byte long
453
+ 00000011 00000011 unsigned 128-bit int - 3 bytes long
454
+
455
+ If the five bits are equal to 29, 30, or 31, then use the following algorithm
456
+ to calculate the payload size.
457
+
458
+ If the value is 29, then the size is 29 + *the next byte after the type
459
+ specifying bytes as an unsigned integer*.
460
+
461
+ If the value is 30, then the size is 285 + *the next two bytes after the type
462
+ specifying bytes as a single unsigned integer*.
463
+
464
+ If the value is 31, then the size is 65,821 + *the next three bytes after the
465
+ type specifying bytes as a single unsigned integer*.
466
+
467
+ Some examples:
468
+
469
+ 01011101 00110011 UTF-8 string - 80 bytes long
470
+
471
+ In this case, the last five bits of the control byte equal 29. We treat the
472
+ next byte as an unsigned integer. The next byte is 51, so the total size is
473
+ (29 + 51) = 80.
474
+
475
+ 01011110 00110011 00110011 UTF-8 string - 13,392 bytes long
476
+
477
+ The last five bits of the control byte equal 30. We treat the next two bytes
478
+ as a single unsigned integer. The next two bytes equal 13,107, so the total
479
+ size is (285 + 13,107) = 13,392.
480
+
481
+ 01011111 00110011 00110011 00110011 UTF-8 string - 3,421,264 bytes long
482
+
483
+ The last five bits of the control byte equal 31. We treat the next three bytes
484
+ as a single unsigned integer. The next three bytes equal 3,355,443, so the
485
+ total size is (65,821 + 3,355,443) = 3,421,264.
486
+
487
+ This means that the maximum payload size for a single field is 16,843,036
488
+ bytes.
489
+
490
+ The binary number types always have a known size, but for consistency's sake,
491
+ the control byte will always specify the correct size for these types.
492
+
493
+ #### Maps
494
+
495
+ Maps use the size in the control byte (and any following bytes) to indicate
496
+ the number of key/value pairs in the map, not the size of the payload in
497
+ bytes.
498
+
499
+ This means that the maximum number of pairs for a single map is 16,843,036.
500
+
501
+ Maps are laid out with each key followed by its value, followed by the next
502
+ pair, etc.
503
+
504
+ The keys are **always** UTF-8 strings. The values may be any data type,
505
+ including maps or pointers.
506
+
507
+ Once we know the number of pairs, we can look at each pair in turn to
508
+ determine the size of the key and the key name, as well as the value's type
509
+ and payload.
510
+
511
+ #### Pointers
512
+
513
+ Pointers use the last five bits in the control byte to calculate the pointer
514
+ value.
515
+
516
+ To calculate the pointer value, we start by subdividing the five bits into two
517
+ groups. The first two bits indicate the size, and the next three bits are part
518
+ of the value, so we end up with a control byte breaking down like this:
519
+ 001SSVVV.
520
+
521
+ The size can be 0, 1, 2, or 3.
522
+
523
+ If the size is 0, the pointer is built by appending the next byte to the last
524
+ three bits to produce an 11-bit value.
525
+
526
+ If the size is 1, the pointer is built by appending the next two bytes to the
527
+ last three bits to produce a 19-bit value + 2048.
528
+
529
+ If the size is 2, the pointer is built by appending the next three bytes to the
530
+ last three bits to produce a 27-bit value + 526336.
531
+
532
+ Finally, if the size is 3, the pointer's value is contained in the next four
533
+ bytes as a 32-bit value. In this case, the last three bits of the control byte
534
+ are ignored.
535
+
536
+ This means that we are limited to 4GB of address space for pointers, so the
537
+ data section size for the database is limited to 4GB.
538
+
539
+ ## Reference Implementations
540
+
541
+ ### Writer
542
+
543
+ * [Perl](https://github.com/maxmind/MaxMind-DB-Writer-perl)
544
+
545
+ ### Reader
546
+
547
+ * [C](https://github.com/maxmind/libmaxminddb)
548
+ * [C#](https://github.com/maxmind/MaxMind-DB-Reader-dotnet)
549
+ * [Java](https://github.com/maxmind/MaxMind-DB-Reader-java)
550
+ * [Perl](https://github.com/maxmind/MaxMind-DB-Reader-perl)
551
+ * [PHP](https://github.com/maxmind/MaxMind-DB-Reader-php)
552
+ * [Python](https://github.com/maxmind/MaxMind-DB-Reader-python)
553
+ * [Ruby](https://github.com/maxmind/MaxMind-DB-Reader-ruby)
554
+
555
+ ## Authors
556
+
557
+ This specification was created by the following authors:
558
+
559
+ * Greg Oschwald \<goschwald@maxmind.com\>
560
+ * Dave Rolsky \<drolsky@maxmind.com\>
561
+ * Boris Zentner \<bzentner@maxmind.com\>
562
+
563
+ ## License
564
+
565
+ This work is licensed under the Creative Commons Attribution-ShareAlike 3.0
566
+ Unported License. To view a copy of this license, visit
567
+ [http://creativecommons.org/licenses/by-sa/3.0/](http://creativecommons.org/licenses/by-sa/3.0/)
568
+ or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain
569
+ View, California, 94041, USA
570
+