iodef 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -0
- data/README.md +37 -18
- data/SECURITY.md +1 -1
- data/data/enums.yml +1 -1
- data/data/types.yml +1 -1
- data/lib/iodef/binary_cbor.rb +285 -0
- data/lib/iodef/cbor.rb +16 -23
- data/lib/iodef/cbor_guard.rb +9 -110
- data/lib/iodef/version.rb +1 -1
- data/lib/iodef/xml/codec.rb +363 -0
- data/lib/iodef/xml/tree.rb +229 -0
- data/lib/iodef/xml.rb +24 -0
- data/sig/iodef.rbs +2 -0
- metadata +28 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: abb43889b6b8b21b5b16be80fd2ed74c92042692b725fce72e60e7ec3f6c8628
|
|
4
|
+
data.tar.gz: 96c490a291b68bfbf8d527eba06c6b6f528600e797ac05372750fe29f6258c68
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: cd6942a46620066e97795ce412dde97d7c41cf278c81eacd971c357b1f99b0b131020d01ddac53e8d7f09800149c0d349a8c0256316c0cd94ab84a862444a37c
|
|
7
|
+
data.tar.gz: fa4ffc07109c725c824cff336e86cc86321746472e5d9e1116dbaea15483b44e40485904612a975bc2e43969d04ae32545f26d153bc5fd689d46f4b76b10522f
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.2.0 - 2026-10-12
|
|
4
|
+
|
|
5
|
+
- Read and write RFC 7970 XML incident reports with `IODEF.parse_xml` and `Document#to_xml`, including namespace-aware extensions and multilingual text.
|
|
6
|
+
- Bound XML parsing and output, reject DTDs, and report unsupported mappings instead of discarding data.
|
|
7
|
+
- Support CBOR across CRuby, JRuby, and TruffleRuby without a separate `cbor` gem.
|
|
8
|
+
|
|
3
9
|
## 0.1.0 - 2026-10-12
|
|
4
10
|
|
|
5
11
|
Initial release.
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# IODEF
|
|
2
2
|
|
|
3
|
-
Read, validate, build, and exchange incident reports
|
|
3
|
+
Read, validate, build, and exchange IODEF v2 incident reports as JSON, CBOR, or XML ([RFC 8727](https://www.rfc-editor.org/rfc/rfc8727.html) and [RFC 7970](https://www.rfc-editor.org/rfc/rfc7970.html)).
|
|
4
4
|
|
|
5
5
|
```ruby
|
|
6
6
|
require "iodef"
|
|
@@ -15,7 +15,7 @@ issues = IODEF.validate(json)
|
|
|
15
15
|
issues.each { |issue| puts "#{issue.pointer}: #{issue.rule}: #{issue.message}" }
|
|
16
16
|
```
|
|
17
17
|
|
|
18
|
-
Install with `gem install iodef`. Ruby 3.3 or newer, JRuby, and TruffleRuby are supported.
|
|
18
|
+
Install with `gem install iodef`. Ruby 3.3 or newer, JRuby, and TruffleRuby are supported. Runtime dependencies are the Ruby standard libraries `json` (2.19.4 or newer, including 3.x) and `rexml` (3.4.4 or newer, below 4); no other gem from this project is required. `IODEF` and `Iodef` refer to the same module.
|
|
19
19
|
|
|
20
20
|
## Build and edit a report
|
|
21
21
|
|
|
@@ -50,15 +50,15 @@ Required properties, types, nonempty collections, enum values, extension selecto
|
|
|
50
50
|
| `max_input_size` | 16 MiB of input/output; also bounds copied string data |
|
|
51
51
|
| `max_depth` | 32 nested levels |
|
|
52
52
|
| `max_collection_size` | 10,000 members per array or object |
|
|
53
|
-
| `max_nodes` | 200,000 JSON/CBOR values |
|
|
53
|
+
| `max_nodes` | 200,000 JSON/CBOR values or XML parsing nodes |
|
|
54
54
|
| `max_binary_size` | 1 MiB per decoded binary value |
|
|
55
55
|
| `max_violations` | 1,000 diagnostics |
|
|
56
56
|
|
|
57
|
-
Pass overrides to `parse`, `validate`, `parse_cbor`, or model constructors. Limits are positive integers. Resource-limit failures raise `LimitError` even in inspection mode. Malformed/duplicate-key JSON raises `ParseError`; cycles, nonfinite numbers, invalid string encodings, and non-JSON Ruby objects are rejected. Diagnostics identify properties and rules without including incident values.
|
|
57
|
+
Pass overrides to `parse`, `validate`, `parse_cbor`, `parse_xml`, or model constructors. Limits are positive integers. Resource-limit failures raise `LimitError` even in inspection mode. Malformed/duplicate-key JSON raises `ParseError`; cycles, nonfinite numbers, invalid string encodings, and non-JSON Ruby objects are rejected. Diagnostics identify properties and rules without including incident values.
|
|
58
58
|
|
|
59
|
-
##
|
|
59
|
+
## CBOR
|
|
60
60
|
|
|
61
|
-
|
|
61
|
+
Load the built-in Ruby codec:
|
|
62
62
|
|
|
63
63
|
```ruby
|
|
64
64
|
require "iodef/cbor"
|
|
@@ -68,25 +68,46 @@ copy = IODEF.parse_cbor(bytes)
|
|
|
68
68
|
|
|
69
69
|
All 207 numeric map keys are supported. The encoder uses date tag 0, base64 byte-string tag 22, hexadecimal byte-string tag 23, and URI tag 32. Dates and URI spelling are preserved. Typed byte content becomes binary; hex output uses uppercase. The decoder also accepts contextually untagged strings/bytes, as used in the RFC's own CBOR examples. Integer bignums are preserved. IODEF real values are encoded as float32; values that would round or overflow raise `ConversionError`. Unknown JSON floats keep their original precision.
|
|
70
70
|
|
|
71
|
-
Unknown CBOR tags, unassigned integer keys, and opaque byte strings without an IODEF type cannot be represented unambiguously in JSON and raise `ConversionError`. Duplicate/colliding keys, excessive nesting, large collections, and trailing data are rejected
|
|
71
|
+
Unknown CBOR tags, unassigned integer keys, and opaque byte strings without an IODEF type cannot be represented unambiguously in JSON and raise `ConversionError`. Duplicate/colliding keys, invalid UTF-8, excessive nesting, large collections, and trailing data are rejected during bounded decoding. JSON and CBOR are tested on CRuby, JRuby, and TruffleRuby, including CRuby development builds. No separate CBOR gem is required.
|
|
72
|
+
|
|
73
|
+
## XML
|
|
74
|
+
|
|
75
|
+
```ruby
|
|
76
|
+
require "iodef/xml"
|
|
77
|
+
|
|
78
|
+
document = IODEF.parse_xml(xml)
|
|
79
|
+
json = document.to_json
|
|
80
|
+
xml = IODEF.parse(json).to_xml
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
XML conversion accepts UTF-8 input and uses the RFC 7970 namespace `urn:ietf:params:xml:ns:iodef-2.0`. It maps attributes, element text, repeated children, and the wrapper elements removed by the JSON binding. Namespace prefixes are resolved by URI. Inherited `xml:lang` and multilingual text are preserved; unqualified JSON multilingual strings mean English. XML output uses version `2.00`, including when the input JSON uses `2.0`.
|
|
84
|
+
|
|
85
|
+
`parse_xml` accepts the same `strict: false` and `validate: false` inspection options as JSON parsing. Serialization validates again. Input size, depth, node count, and collection limits apply during XML parsing; serialized output is also bounded. DTDs and entity declarations are rejected, and no external resource is loaded.
|
|
86
|
+
|
|
87
|
+
Round trips preserve supported IODEF data, not XML formatting, namespace prefix choices, comments, or the original grouping of wrapper elements. XML-valued extensions preserve their namespace-qualified content, but their XML strings and base64-encoded XML bytes may be reserialized. `RawData` supports an embedded XML document with `dtype="xml"`; attributes without a JSON counterpart and other raw encodings cannot be converted. XML Signature/X509 structures and unknown JSON properties without an unambiguous XML mapping raise `ConversionError`. Conversion does not verify or preserve digital signatures.
|
|
88
|
+
|
|
89
|
+
Structured-information `Reference` values are not converted because RFC 7203 references the IODEF v1 XML namespace, while this API exchanges IODEF v2. An `AttackPattern` containing `Scoring` is also rejected because that child is not permitted by RFC 7203. These values remain available in JSON and CBOR.
|
|
90
|
+
|
|
91
|
+
Numeric literals use Ruby integers and floats. Other consumers that apply XML Schema validation may round fields defined as `xs:float`.
|
|
72
92
|
|
|
73
93
|
## Scope
|
|
74
94
|
|
|
75
|
-
| Supported in 0.
|
|
95
|
+
| Supported in 0.2.0 | Outside 0.2.0 |
|
|
76
96
|
| --- | --- |
|
|
77
|
-
| JSON binding, all RFC 8727 class/property definitions |
|
|
97
|
+
| JSON binding, all RFC 8727 class/property definitions | Ambiguous binary XML structures and vendor extension semantics |
|
|
78
98
|
| Typed core models, multilingual text, builders | RID or other exchange transports |
|
|
79
|
-
| IODEF enum extensions
|
|
80
|
-
|
|
|
99
|
+
| RFC 7970 XML conversion and IODEF enum extensions | XML Schema validation and XML Signature conversion |
|
|
100
|
+
| Preserved unknown JSON fields in JSON/CBOR | Automatic XML mapping of unknown JSON fields |
|
|
101
|
+
| CBOR, numeric keys, typed binary values | Signing, encryption, threat correlation |
|
|
81
102
|
| RBS signatures and bounded validation | Authenticating senders or verifying referenced evidence |
|
|
82
103
|
|
|
83
|
-
The
|
|
104
|
+
The property metadata in `data/` describes the RFC field names, types, cardinalities, and protocol identifiers. CDDL, JSON Schema, RFC examples, and IANA source downloads are not embedded in the package. Tests inspect the downloaded CDDL as reference data; no third-party CDDL or CBOR implementation is loaded.
|
|
84
105
|
|
|
85
106
|
### Specification differences
|
|
86
107
|
|
|
87
|
-
RFC 8727's CDDL has a verified missing-brace correction ([erratum 7334](https://www.rfc-editor.org/errata/eid7334)). The
|
|
108
|
+
RFC 8727's CDDL has a verified missing-brace correction ([erratum 7334](https://www.rfc-editor.org/errata/eid7334)). The temporary reference schema applies it. Its section 4.2 JSON example needs presentation-wrapped strings joined; the original text is retained beside the corrected fixture. Both complete JSON and CBOR examples are tested.
|
|
88
109
|
|
|
89
|
-
The JSON examples use version `2.0`; the XML information model uses `2.00`.
|
|
110
|
+
The JSON examples use version `2.0`; the XML information model uses `2.00`. JSON and CBOR accept and preserve both spellings; XML output uses `2.00`. Required nonempty child groups and conditional values follow RFC 7970: nonnumeric `Confidence` omits `value`, numeric confidence requires it; a node can include both addresses and domain data; `HashData` supports `ext-scope`. `RelatedActivity.Description` accepts multilingual strings. These constraints are not all accurately expressed in the RFC 8727 CDDL. The section 2.2.5 illustration uses a CWE `SpecID` absent from its normative enum: use `private` with `ext-SpecID` for that extension.
|
|
90
111
|
|
|
91
112
|
The pinned IANA registry and RFC have differing spellings for `breach-propietary` / `breach-proprietary`, `anonymization` / `anomyzation`, and `ntp-stamp` / `ntpstamp`; both are accepted. RFC 8727 additionally defines the `json` extension data type. This library validates format and structure, not whether a reported address, software identifier, or incident fact is authentic.
|
|
92
113
|
|
|
@@ -96,16 +117,14 @@ The pinned IANA registry and RFC have differing spellings for `breach-propietary
|
|
|
96
117
|
bundle install
|
|
97
118
|
bundle exec rake test
|
|
98
119
|
COVERAGE=1 bundle exec rake test
|
|
99
|
-
bundle exec rubocop
|
|
100
120
|
bundle exec rbs -I sig validate
|
|
101
|
-
bundle exec ruby script/interop.rb
|
|
102
121
|
bundle exec ruby script/benchmark.rb 5000
|
|
103
122
|
```
|
|
104
123
|
|
|
105
|
-
The fixture task verifies SHA-256-pinned RFC/IANA sources, retains source notices, and regenerates examples under ignored `tmp/rfc/`.
|
|
124
|
+
The fixture task verifies SHA-256-pinned RFC/IANA sources, retains source notices, and regenerates examples under ignored `tmp/rfc/`. Tests check both complete RFC JSON and CBOR examples, both complete RFC 7970 XML documents, every CDDL object property, and the numeric map assignments. The printed XML campaign example uses the unregistered `fqdn` value; its derived fixture uses `domain-name` as described in [reported erratum 6163](https://www.rfc-editor.org/errata/eid6163), while the original remains a negative test. Original wire-format tests cover integer widths, floating-point precision, UTF-8 chunk boundaries, duplicate keys, tags, malformed values, limits, and randomized round trips. XML tests cover namespaces, language inheritance, escaping, DTD rejection, and bounded parsing. Reference data keeps its own license and notices even though it is not distributed with the gem.
|
|
106
125
|
|
|
107
126
|
`script/update_registry.rb` audits the pinned IANA metadata. `--refresh` downloads the current registry, rejects unexplained differences, and updates its checksum only after the audit succeeds. Review RFC errata and registry changes before refreshing. No downloaded reference material is tracked or included in the gem.
|
|
108
127
|
|
|
109
128
|
## License
|
|
110
129
|
|
|
111
|
-
MIT
|
|
130
|
+
Project code is MIT licensed; see [LICENSE.txt](LICENSE.txt). Added library dependencies are limited to MIT-licensed libraries. Ruby implementations and standard/default libraries (including JSON and REXML) are exceptions and retain their own licenses. Downloaded standards material is test reference data under its source terms and is neither committed nor packaged. These statements do not relicense third-party material.
|
data/SECURITY.md
CHANGED
|
@@ -4,4 +4,4 @@ Report suspected vulnerabilities privately through [GitHub's vulnerability repor
|
|
|
4
4
|
|
|
5
5
|
IODEF documents contain untrusted, potentially sensitive content. This library validates representation and selected information-model rules; it does not authenticate reports or enforce disclosure restrictions. Apply authorization, transport encryption, retention policies, and trusted source selection in the calling application. Do not execute embedded scripts, XML, regexes, or extension payloads. The library itself never fetches document-supplied URLs or schemas.
|
|
6
6
|
|
|
7
|
-
Size, depth, node, collection, binary, and diagnostic limits apply by default. Increase them only for known workloads. The
|
|
7
|
+
Size, depth, node, collection, binary, and diagnostic limits apply by default. Increase them only for known workloads. The Ruby CBOR decoder checks bounds while reading and accepts only the IODEF tag set. It never evaluates tags as application code. Invalid or ambiguous CBOR values fail instead of silently dropping information.
|
data/data/enums.yml
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Protocol identifiers
|
|
1
|
+
# Protocol identifiers used by the information model.
|
|
2
2
|
restriction: [public, partner, need-to-know, private, default, white, green, amber, red, ext-value]
|
|
3
3
|
purpose: [traceback, mitigation, reporting, watch, other, ext-value]
|
|
4
4
|
status: [new, in-progress, forwarded, resolved, future, ext-value]
|
data/data/types.yml
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Implementation metadata. Sections refer to RFC 7970;
|
|
2
2
|
# JSON cardinalities and property spellings follow RFC 8727 sections 2 and 3.
|
|
3
3
|
# A trailing ! requires a property; [] denotes a nonempty list when present.
|
|
4
4
|
groups:
|
|
@@ -0,0 +1,285 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Iodef
|
|
4
|
+
# Internal wire codec. RFC 8949 sections 3.1–3.4 define the byte format;
|
|
5
|
+
# application-specific tag meanings belong to the caller.
|
|
6
|
+
module BinaryCBOR
|
|
7
|
+
class Error < StandardError; end
|
|
8
|
+
class LimitError < Error; end
|
|
9
|
+
class UnsupportedError < Error; end
|
|
10
|
+
Tagged = Struct.new(:tag, :value)
|
|
11
|
+
Simple = Struct.new(:value)
|
|
12
|
+
Float32 = Struct.new(:value)
|
|
13
|
+
|
|
14
|
+
def self.decode(bytes, **options)
|
|
15
|
+
Decoder.new(bytes, **options).read
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def self.encode(value, **options)
|
|
19
|
+
Encoder.new(**options).write(value)
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
class Decoder
|
|
23
|
+
DEFAULTS = { max_input_size: 16_777_216, max_depth: 32, max_nodes: 1_000_000, max_collection_size: 200_000,
|
|
24
|
+
max_string_size: 16_777_216, max_binary_size: 16_777_216, allowed_tags: nil, allow_indefinite_strings: true,
|
|
25
|
+
max_records: nil, max_record_bytes: nil }.freeze
|
|
26
|
+
|
|
27
|
+
def initialize(bytes, **options)
|
|
28
|
+
@limits = DEFAULTS.merge(options)
|
|
29
|
+
raise Error, "CBOR input must be a String" unless bytes.is_a?(String)
|
|
30
|
+
raise LimitError, "CBOR exceeds max_input_size" if bytes.bytesize > @limits[:max_input_size]
|
|
31
|
+
|
|
32
|
+
@bytes = bytes
|
|
33
|
+
@offset = 0
|
|
34
|
+
@nodes = 0
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def read
|
|
38
|
+
value = read_value(0)
|
|
39
|
+
raise Error, "trailing CBOR input" unless @offset == @bytes.bytesize
|
|
40
|
+
|
|
41
|
+
value
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
private
|
|
45
|
+
|
|
46
|
+
def take(size)
|
|
47
|
+
raise Error, "truncated CBOR input" if size > @bytes.bytesize - @offset
|
|
48
|
+
if @record_start && @limits[:max_record_bytes] && @offset + size - @record_start > @limits[:max_record_bytes]
|
|
49
|
+
raise LimitError, "CBOR record exceeds max_record_bytes"
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
result = @bytes.byteslice(@offset, size)
|
|
53
|
+
@offset += size
|
|
54
|
+
result
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
def argument(info)
|
|
58
|
+
return info if info < 24
|
|
59
|
+
return nil if info == 31
|
|
60
|
+
|
|
61
|
+
format = { 24 => [1, "C"], 25 => [2, "n"], 26 => [4, "N"], 27 => [8, "Q>"] }[info]
|
|
62
|
+
raise Error, "reserved CBOR additional information" unless format
|
|
63
|
+
|
|
64
|
+
take(format[0]).unpack1(format[1])
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def read_value(depth)
|
|
68
|
+
@nodes += 1
|
|
69
|
+
raise LimitError, "CBOR exceeds max_nodes" if @nodes > @limits[:max_nodes]
|
|
70
|
+
raise LimitError, "CBOR exceeds max_depth" if depth > @limits[:max_depth]
|
|
71
|
+
|
|
72
|
+
initial = take(1).getbyte(0)
|
|
73
|
+
major, info = initial.divmod(32)
|
|
74
|
+
return read_simple(info) if major == 7
|
|
75
|
+
|
|
76
|
+
size = argument(info)
|
|
77
|
+
raise Error, "indefinite CBOR integer or tag" if size.nil? && [0, 1, 6].include?(major)
|
|
78
|
+
|
|
79
|
+
case major
|
|
80
|
+
when 0 then size
|
|
81
|
+
when 1 then -1 - size
|
|
82
|
+
when 2, 3 then read_string(major, size, depth)
|
|
83
|
+
when 4, 5 then read_collection(major, size, depth)
|
|
84
|
+
when 6
|
|
85
|
+
raise UnsupportedError, "unsupported CBOR tag" if @limits[:allowed_tags] && !@limits[:allowed_tags].include?(size)
|
|
86
|
+
|
|
87
|
+
Tagged.new(size, read_value(depth + 1))
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def read_string(major, size, depth)
|
|
92
|
+
limit = @limits[:max_string_size]
|
|
93
|
+
limit = [limit, @limits[:max_binary_size]].min if major == 2
|
|
94
|
+
if size
|
|
95
|
+
raise LimitError, "CBOR string exceeds #{major == 2 ? "max_binary_size" : "max_string_size"}" if size > limit
|
|
96
|
+
|
|
97
|
+
value = take(size).dup
|
|
98
|
+
value.force_encoding(major == 2 ? Encoding::BINARY : Encoding::UTF_8)
|
|
99
|
+
raise Error, "invalid CBOR UTF-8 text" unless value.valid_encoding?
|
|
100
|
+
|
|
101
|
+
return value
|
|
102
|
+
end
|
|
103
|
+
raise Error, "CBOR strings must have definite length" unless @limits[:allow_indefinite_strings]
|
|
104
|
+
|
|
105
|
+
result = String.new(encoding: major == 2 ? Encoding::BINARY : Encoding::UTF_8)
|
|
106
|
+
until @bytes.getbyte(@offset) == 255
|
|
107
|
+
initial = @bytes.getbyte(@offset)
|
|
108
|
+
raise Error, "invalid indefinite CBOR string chunk" unless initial && initial.div(32) == major && (initial & 31) < 28
|
|
109
|
+
|
|
110
|
+
chunk = read_value(depth + 1)
|
|
111
|
+
raise LimitError, "CBOR string exceeds byte limit" if result.bytesize + chunk.bytesize > limit
|
|
112
|
+
|
|
113
|
+
result << chunk
|
|
114
|
+
end
|
|
115
|
+
take(1)
|
|
116
|
+
result
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
def read_collection(major, size, depth)
|
|
120
|
+
limit = @limits[:max_collection_size]
|
|
121
|
+
limit = [limit, @limits[:max_records]].min if depth.zero? && @limits[:max_records]
|
|
122
|
+
raise LimitError, "CBOR exceeds collection limit" if size && size > limit
|
|
123
|
+
|
|
124
|
+
result = major == 4 ? [] : {}
|
|
125
|
+
count = 0
|
|
126
|
+
until size ? count == size : @bytes.getbyte(@offset) == 255
|
|
127
|
+
raise LimitError, "CBOR exceeds collection limit" if count >= limit
|
|
128
|
+
|
|
129
|
+
@record_start = @offset if depth.zero? && @limits[:max_record_bytes]
|
|
130
|
+
if major == 4
|
|
131
|
+
result << read_value(depth + 1)
|
|
132
|
+
else
|
|
133
|
+
key = read_value(depth + 1)
|
|
134
|
+
unless key.is_a?(Integer) || (key.is_a?(String) && key.encoding == Encoding::UTF_8)
|
|
135
|
+
raise UnsupportedError, "CBOR map keys must be text or integers"
|
|
136
|
+
end
|
|
137
|
+
raise Error, "duplicate CBOR map key" if result.key?(key)
|
|
138
|
+
|
|
139
|
+
result[key] = read_value(depth + 1)
|
|
140
|
+
end
|
|
141
|
+
@record_start = nil if depth.zero?
|
|
142
|
+
count += 1
|
|
143
|
+
end
|
|
144
|
+
take(1) unless size
|
|
145
|
+
result
|
|
146
|
+
end
|
|
147
|
+
|
|
148
|
+
def read_simple(info)
|
|
149
|
+
case info
|
|
150
|
+
when 20 then false
|
|
151
|
+
when 21 then true
|
|
152
|
+
when 22 then nil
|
|
153
|
+
when 0..19, 23 then Simple.new(info)
|
|
154
|
+
when 24
|
|
155
|
+
value = take(1).getbyte(0)
|
|
156
|
+
raise Error, "reserved CBOR simple value" if value < 32
|
|
157
|
+
|
|
158
|
+
Simple.new(value)
|
|
159
|
+
when 25 then half(take(2).unpack1("n"))
|
|
160
|
+
when 26 then take(4).unpack1("g")
|
|
161
|
+
when 27 then take(8).unpack1("G")
|
|
162
|
+
else raise Error, "reserved CBOR value or unexpected break"
|
|
163
|
+
end
|
|
164
|
+
end
|
|
165
|
+
|
|
166
|
+
def half(bits)
|
|
167
|
+
exponent = (bits >> 10) & 31
|
|
168
|
+
fraction = bits & 1023
|
|
169
|
+
magnitude = if exponent == 31
|
|
170
|
+
fraction.zero? ? Float::INFINITY : Float::NAN
|
|
171
|
+
elsif exponent.zero?
|
|
172
|
+
Math.ldexp(fraction.to_f, -24)
|
|
173
|
+
else
|
|
174
|
+
Math.ldexp((1024 + fraction).to_f, exponent - 25)
|
|
175
|
+
end
|
|
176
|
+
bits[15].zero? ? magnitude : -magnitude
|
|
177
|
+
end
|
|
178
|
+
end
|
|
179
|
+
|
|
180
|
+
class Encoder
|
|
181
|
+
def initialize(max_output_size: 16_777_216, max_depth: 128)
|
|
182
|
+
@max_output_size = max_output_size
|
|
183
|
+
@max_depth = max_depth
|
|
184
|
+
@buffer = String.new(encoding: Encoding::BINARY)
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
def write(value)
|
|
188
|
+
append_value(value, 0)
|
|
189
|
+
@buffer
|
|
190
|
+
end
|
|
191
|
+
|
|
192
|
+
private
|
|
193
|
+
|
|
194
|
+
def append(bytes)
|
|
195
|
+
raise LimitError, "CBOR output exceeds max_input_size" if @buffer.bytesize + bytes.bytesize > @max_output_size
|
|
196
|
+
|
|
197
|
+
@buffer << bytes.b
|
|
198
|
+
end
|
|
199
|
+
|
|
200
|
+
def header(major, number)
|
|
201
|
+
raise Error, "CBOR argument outside unsigned 64-bit range" unless number.is_a?(Integer) && number.between?(0, (2**64) - 1)
|
|
202
|
+
|
|
203
|
+
case number
|
|
204
|
+
when 0...24 then append([(major << 5) | number].pack("C"))
|
|
205
|
+
when 24..255 then append([(major << 5) | 24, number].pack("CC"))
|
|
206
|
+
when 256..65_535 then append([(major << 5) | 25, number].pack("Cn"))
|
|
207
|
+
when 65_536..4_294_967_295 then append([(major << 5) | 26, number].pack("CN"))
|
|
208
|
+
else append([(major << 5) | 27, number].pack("CQ>"))
|
|
209
|
+
end
|
|
210
|
+
end
|
|
211
|
+
|
|
212
|
+
def append_value(value, depth)
|
|
213
|
+
raise LimitError, "CBOR output exceeds max_depth" if depth > @max_depth
|
|
214
|
+
|
|
215
|
+
case value
|
|
216
|
+
when Integer then append_integer(value)
|
|
217
|
+
when Float then append_float(value)
|
|
218
|
+
when Float32 then append("\xfa".b + [value.value].pack("g"))
|
|
219
|
+
when String then append_string(value)
|
|
220
|
+
when Array
|
|
221
|
+
header(4, value.length)
|
|
222
|
+
value.each { |item| append_value(item, depth + 1) }
|
|
223
|
+
when Hash
|
|
224
|
+
header(5, value.length)
|
|
225
|
+
value.each { |key, item| append_value(key, depth + 1); append_value(item, depth + 1) }
|
|
226
|
+
when Tagged
|
|
227
|
+
header(6, value.tag)
|
|
228
|
+
append_value(value.value, depth + 1)
|
|
229
|
+
when Simple
|
|
230
|
+
raise Error, "invalid CBOR simple value" unless value.value.is_a?(Integer) && (value.value.between?(0, 23) || value.value.between?(32, 255))
|
|
231
|
+
|
|
232
|
+
header(7, value.value)
|
|
233
|
+
when TrueClass then append("\xf5".b)
|
|
234
|
+
when FalseClass then append("\xf4".b)
|
|
235
|
+
when NilClass then append("\xf6".b)
|
|
236
|
+
else raise Error, "unsupported CBOR value type"
|
|
237
|
+
end
|
|
238
|
+
end
|
|
239
|
+
|
|
240
|
+
def append_integer(value)
|
|
241
|
+
magnitude = value.negative? ? -1 - value : value
|
|
242
|
+
return header(value.negative? ? 1 : 0, magnitude) if magnitude < 2**64
|
|
243
|
+
|
|
244
|
+
header(6, value.negative? ? 3 : 2)
|
|
245
|
+
hex = magnitude.to_s(16)
|
|
246
|
+
hex = "0#{hex}" if hex.length.odd?
|
|
247
|
+
append_string([hex].pack("H*"))
|
|
248
|
+
end
|
|
249
|
+
|
|
250
|
+
def append_float(value)
|
|
251
|
+
half = half_bits(value)
|
|
252
|
+
return append("\xf9".b + [half].pack("n")) if half
|
|
253
|
+
|
|
254
|
+
single = [value].pack("g")
|
|
255
|
+
append(single.unpack1("g") == value ? "\xfa".b + single : "\xfb".b + [value].pack("G"))
|
|
256
|
+
end
|
|
257
|
+
|
|
258
|
+
def half_bits(value)
|
|
259
|
+
return 0x7e00 if value.nan?
|
|
260
|
+
|
|
261
|
+
sign = value.negative? || (value.zero? && (1.0 / value).negative?) ? 0x8000 : 0
|
|
262
|
+
return sign | 0x7c00 if value.infinite?
|
|
263
|
+
return sign if value.zero?
|
|
264
|
+
|
|
265
|
+
significand, exponent = Math.frexp(value.abs)
|
|
266
|
+
if exponent < -13
|
|
267
|
+
fraction = Math.ldexp(value.abs, 24)
|
|
268
|
+
return sign | fraction.to_i if fraction == fraction.to_i
|
|
269
|
+
elsif exponent <= 16
|
|
270
|
+
fraction = significand * 2048 - 1024
|
|
271
|
+
return sign | ((exponent + 14) << 10) | fraction.to_i if fraction == fraction.to_i
|
|
272
|
+
end
|
|
273
|
+
nil
|
|
274
|
+
end
|
|
275
|
+
|
|
276
|
+
def append_string(value)
|
|
277
|
+
binary = value.encoding == Encoding::BINARY
|
|
278
|
+
raise Error, "invalid CBOR UTF-8 text" unless binary || (value.valid_encoding? && (value.encoding == Encoding::UTF_8 || value.ascii_only?))
|
|
279
|
+
|
|
280
|
+
header(binary ? 2 : 3, value.bytesize)
|
|
281
|
+
append(value)
|
|
282
|
+
end
|
|
283
|
+
end
|
|
284
|
+
end
|
|
285
|
+
end
|
data/lib/iodef/cbor.rb
CHANGED
|
@@ -1,21 +1,13 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require "iodef"
|
|
4
|
-
|
|
4
|
+
require_relative "binary_cbor"
|
|
5
5
|
require_relative "cbor_guard"
|
|
6
6
|
|
|
7
7
|
module Iodef
|
|
8
8
|
module CBORCodec
|
|
9
9
|
module_function
|
|
10
10
|
|
|
11
|
-
Float32 = Struct.new(:value) do
|
|
12
|
-
def to_cbor(packer)
|
|
13
|
-
packer.buffer.write("\xfa".b + [value].pack("g"))
|
|
14
|
-
packer
|
|
15
|
-
end
|
|
16
|
-
end
|
|
17
|
-
private_constant :Float32
|
|
18
|
-
|
|
19
11
|
def encode(value, type = "Document")
|
|
20
12
|
return value.map { |item| encode(item, type.delete_suffix("[]")) } if type.end_with?("[]")
|
|
21
13
|
|
|
@@ -28,16 +20,16 @@ module Iodef
|
|
|
28
20
|
end
|
|
29
21
|
end
|
|
30
22
|
return value.map { |item| encode(item, "any") } if value.is_a?(Array)
|
|
31
|
-
return
|
|
32
|
-
return
|
|
33
|
-
return
|
|
34
|
-
return
|
|
23
|
+
return BinaryCBOR::Tagged.new(22, value.unpack1("m0").b) if type == "binary"
|
|
24
|
+
return BinaryCBOR::Tagged.new(23, [value].pack("H*").b) if type == "hex"
|
|
25
|
+
return BinaryCBOR::Tagged.new(0, value) if type == "datetime"
|
|
26
|
+
return BinaryCBOR::Tagged.new(32, value) if type == "uri"
|
|
35
27
|
|
|
36
28
|
if %w[number positive].include?(type)
|
|
37
29
|
rounded = [value].pack("g").unpack1("g")
|
|
38
30
|
raise ConversionError, "IODEF CBOR real values must be exactly representable as float32" unless rounded.finite? && rounded == value
|
|
39
31
|
|
|
40
|
-
return Float32.new(rounded)
|
|
32
|
+
return BinaryCBOR::Float32.new(rounded)
|
|
41
33
|
end
|
|
42
34
|
value
|
|
43
35
|
end
|
|
@@ -49,11 +41,11 @@ module Iodef
|
|
|
49
41
|
return decode_object(value, type, path) if value.is_a?(Hash)
|
|
50
42
|
return value.each_with_index.map { |item, index| decode(item, "any", Iodef.pointer(path, index)) } if value.is_a?(Array)
|
|
51
43
|
|
|
52
|
-
if value.is_a?(
|
|
44
|
+
if value.is_a?(BinaryCBOR::Tagged) && [2, 3].include?(value.tag)
|
|
53
45
|
raise ConversionError, "bignum tag requires a byte string at #{path}" unless value.value.is_a?(String) && value.value.encoding == Encoding::BINARY
|
|
54
46
|
|
|
55
47
|
magnitude = value.value.empty? ? 0 : value.value.unpack1("H*").to_i(16)
|
|
56
|
-
value = value.tag ==
|
|
48
|
+
value = value.tag == 3 ? -1 - magnitude : magnitude
|
|
57
49
|
end
|
|
58
50
|
case type
|
|
59
51
|
when "binary" then decode_tag(value, 22, path) { |bytes| [bytes].pack("m0") }
|
|
@@ -61,7 +53,7 @@ module Iodef
|
|
|
61
53
|
when "datetime" then decode_tag(value, 0, path) { |text| text }
|
|
62
54
|
when "uri" then decode_tag(value, 32, path) { |text| text }
|
|
63
55
|
else
|
|
64
|
-
if value.is_a?(
|
|
56
|
+
if value.is_a?(BinaryCBOR::Tagged) || value.is_a?(BinaryCBOR::Simple) || (value.is_a?(String) && value.encoding == Encoding::BINARY)
|
|
65
57
|
raise ConversionError, "CBOR value has no unambiguous JSON representation at #{path}"
|
|
66
58
|
end
|
|
67
59
|
|
|
@@ -97,7 +89,7 @@ module Iodef
|
|
|
97
89
|
binary = [22, 23].include?(tag)
|
|
98
90
|
return yield(value) if value.is_a?(String) && (value.encoding == Encoding::BINARY) == binary
|
|
99
91
|
|
|
100
|
-
valid = value.is_a?(
|
|
92
|
+
valid = value.is_a?(BinaryCBOR::Tagged) && value.tag == tag && value.value.is_a?(String)
|
|
101
93
|
valid &&= (value.value.encoding == Encoding::BINARY) == [22, 23].include?(tag)
|
|
102
94
|
raise ConversionError, "expected CBOR tag #{tag} at #{path}" unless valid
|
|
103
95
|
|
|
@@ -111,21 +103,22 @@ module Iodef
|
|
|
111
103
|
raise ParseError, "CBOR input must be a String" unless bytes.is_a?(String)
|
|
112
104
|
raise LimitError, "input exceeds max_input_size" if bytes.bytesize > options[:max_input_size]
|
|
113
105
|
|
|
114
|
-
|
|
115
|
-
data = CBORCodec.decode(CBOR.decode(guarded))
|
|
106
|
+
data = CBORCodec.decode(CBORGuard.new(bytes, options).validate!)
|
|
116
107
|
Document.new(data, strict: strict && validate, **options)
|
|
117
|
-
rescue CBOR::UnpackError, CBOR::TypeError, EOFError, EncodingError
|
|
118
|
-
raise ParseError, "invalid CBOR input"
|
|
119
108
|
end
|
|
120
109
|
end
|
|
121
110
|
|
|
122
111
|
class Document
|
|
123
112
|
def to_cbor
|
|
124
113
|
validate!
|
|
125
|
-
bytes =
|
|
114
|
+
bytes = BinaryCBOR.encode(CBORCodec.encode(to_h), max_output_size: limits[:max_input_size])
|
|
126
115
|
raise LimitError, "output exceeds max_input_size" if bytes.bytesize > limits[:max_input_size]
|
|
127
116
|
|
|
128
117
|
bytes
|
|
118
|
+
rescue BinaryCBOR::LimitError => e
|
|
119
|
+
raise LimitError, e.message
|
|
120
|
+
rescue BinaryCBOR::Error => e
|
|
121
|
+
raise ConversionError, e.message
|
|
129
122
|
end
|
|
130
123
|
end
|
|
131
124
|
end
|