toon-fu 0.0.1 → 4.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +126 -5
- data/lib/toon_fu/decimal_literal.rb +19 -0
- data/lib/toon_fu/encodable.rb +20 -0
- data/lib/toon_fu/encoder.rb +52 -0
- data/lib/toon_fu/fields.rb +34 -0
- data/lib/toon_fu/float_literal.rb +52 -0
- data/lib/toon_fu/normalizer.rb +90 -0
- data/lib/toon_fu/string_literal.rb +36 -0
- data/lib/toon_fu/version.rb +1 -1
- data/lib/toon_fu/writer.rb +128 -0
- data/lib/toon_fu.rb +32 -0
- metadata +10 -3
- data/CHANGELOG.md +0 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 6eda50e6a13e61ee805fbaf15f417176c2463648bc226af942aaa383ee50cc5c
|
|
4
|
+
data.tar.gz: 65406f2ccd9d8e643c9f26109d16d52d3361c92e200323e235e52ec572669461
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ef2e69c9a75e69600957f6ab92e139ee522d960cafc2793aa7621854fa9c5bf667b7c197f2b32820b361af3070b68769dfc86c8e639a428c8bec8ad61ee6ee99
|
|
7
|
+
data.tar.gz: 8d5ced2be98d6bba1e423a75aac5eb22992bbfa8e76b1cf5cc2a33f09a0b1eea96590c4a8b33970768cd9885d30a39e0fc8db8522c263c60d93130d1c7043112
|
data/README.md
CHANGED
|
@@ -1,19 +1,140 @@
|
|
|
1
1
|
# toon-fu
|
|
2
2
|
|
|
3
|
-
[
|
|
3
|
+
[](https://github.com/hoblin/toon-fu/actions/workflows/ci.yml)
|
|
4
|
+
[](https://github.com/hoblin/toon-fu/actions/workflows/spec-drift.yml)
|
|
5
|
+
[](https://rubygems.org/gems/toon-fu)
|
|
4
6
|
|
|
5
|
-
|
|
7
|
+
[TOON](https://toonformat.dev/) (Token-Oriented Object Notation) encoder for Ruby, versioned by the spec it implements: the gem's `MAJOR.MINOR` is the TOON spec version it speaks.
|
|
8
|
+
|
|
9
|
+
## What is this?
|
|
10
|
+
|
|
11
|
+
TOON is a compact, readable encoding of the JSON data model for LLM prompts: indentation instead of braces, quotes only where needed, and tables for arrays of uniform objects. toon-fu is written from the [specification](https://github.com/toon-format/spec) and runs the spec's reference fixtures as its conformance suite — every encode fixture of the spec version it implements passes.
|
|
12
|
+
|
|
13
|
+
## Why?
|
|
14
|
+
|
|
15
|
+
Every Ruby TOON gem on RubyGems was published in late 2025 and stopped at spec 1.2, three major versions behind. They leave strings starting with `#` or `+` unquoted, which a current reader takes for a comment or a number, and shift dates by a day east of Greenwich. toon-fu tracks the spec: its version is the spec version, and a daily check flags a newer spec.
|
|
16
|
+
|
|
17
|
+
## Getting started
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
gem install toon-fu
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
In a Gemfile, pin the spec line you speak — see [Versioning](#versioning).
|
|
24
|
+
|
|
25
|
+
```ruby
|
|
26
|
+
require "toon_fu"
|
|
27
|
+
|
|
28
|
+
ToonFu.encode({users: [{id: 1, name: "Ada", role: "admin"}, {id: 2, name: "Bob", role: "user"}]})
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
users[2]{id,name,role}:
|
|
33
|
+
1,Ada,admin
|
|
34
|
+
2,Bob,user
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Arrays of uniform objects become tables that declare their fields once. Everything else reads like YAML:
|
|
38
|
+
|
|
39
|
+
```ruby
|
|
40
|
+
{order: {id: 7, tags: ["new", "paid"], customer: {name: "Ada"}}}.to_toon
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
```
|
|
44
|
+
order:
|
|
45
|
+
id: 7
|
|
46
|
+
tags[2]: new,paid
|
|
47
|
+
customer:
|
|
48
|
+
name: Ada
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
`to_toon` is available on Hash, Array, String, Symbol, Integer, Float, `true`, `false`, `nil`, Set, Time and Date once `toon_fu` is loaded.
|
|
52
|
+
|
|
53
|
+
### Options
|
|
54
|
+
|
|
55
|
+
- `delimiter:` — `","` (default), `"\t"` or `"|"`. Tab usually costs the fewest tokens.
|
|
56
|
+
- `indent_size:` — spaces per nesting level, default `2`.
|
|
57
|
+
|
|
58
|
+
```ruby
|
|
59
|
+
ToonFu.encode(data, delimiter: "\t")
|
|
60
|
+
encoder = ToonFu::Encoder.new(delimiter: "|") # reuse for many documents
|
|
61
|
+
encoder.encode(data)
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
### Your own objects
|
|
65
|
+
|
|
66
|
+
Define `as_toon` to return plain data; it wins over every built-in mapping:
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
Money = Struct.new(:cents, :currency) do
|
|
70
|
+
include ToonFu::Encodable # optional: adds #to_toon
|
|
71
|
+
|
|
72
|
+
def as_toon = {amount: cents / 100.0, currency:}
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
ToonFu.encode({price: Money.new(1999, "EUR")})
|
|
76
|
+
# price:
|
|
77
|
+
# amount: 19.99
|
|
78
|
+
# currency: EUR
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Anything toon-fu does not know raises `ToonFu::Error` instead of guessing:
|
|
82
|
+
|
|
83
|
+
```ruby
|
|
84
|
+
ToonFu.encode({at: Object.new})
|
|
85
|
+
# ToonFu::Error: cannot encode Object; convert it first or define #as_toon
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
For ActiveRecord models, pass `record.as_json` (or define `as_toon`).
|
|
89
|
+
|
|
90
|
+
## What it accepts
|
|
91
|
+
|
|
92
|
+
| Ruby value | TOON |
|
|
93
|
+
|---|---|
|
|
94
|
+
| `Hash` with String, Symbol or Integer keys | object; keys become strings, keys that collide raise |
|
|
95
|
+
| `Array`, `Set` | array: inline, table, or list form |
|
|
96
|
+
| `String` | string, quoted only when needed; other encodings are transcoded to UTF-8, invalid UTF-8 raises |
|
|
97
|
+
| `Symbol` | its name |
|
|
98
|
+
| `Integer` | its exact digits, any size |
|
|
99
|
+
| `Float`, `BigDecimal` | canonical number; NaN and infinities become `null` |
|
|
100
|
+
| `true`, `false`, `nil` | `true`, `false`, `null` |
|
|
101
|
+
| `Date` | `2026-05-31` |
|
|
102
|
+
| `Time`, `DateTime` | ISO 8601 with its offset: `2026-05-31T10:00:05.25Z` |
|
|
103
|
+
| objects with `to_hash`, `to_ary`, `to_str` | the value they convert to |
|
|
104
|
+
| objects with `as_toon` | whatever `as_toon` returns, encoded in turn |
|
|
105
|
+
|
|
106
|
+
Everything else raises `ToonFu::Error` — including a `Struct` or `Data` without `as_toon`, and circular references.
|
|
6
107
|
|
|
7
108
|
## Versioning
|
|
8
109
|
|
|
9
110
|
The gem version tracks the TOON specification it implements:
|
|
10
111
|
|
|
11
|
-
- `MAJOR.MINOR` is the spec version
|
|
112
|
+
- `MAJOR.MINOR` is the spec version: `X.Y.Z` speaks TOON `X.Y`.
|
|
12
113
|
- `PATCH` is the gem's own: fixes and improvements that do not change the dialect.
|
|
13
114
|
|
|
14
|
-
Pin
|
|
115
|
+
Pin the spec line, not just the major: `gem "toon-fu", "~> X.Y.0"` takes our fixes and never moves you to a new dialect. The gem badge above shows the current release; the spec-drift badge turns red when a newer spec is released and toon-fu has not caught up yet.
|
|
116
|
+
|
|
117
|
+
Release notes: [GitHub releases](https://github.com/hoblin/toon-fu/releases).
|
|
118
|
+
|
|
119
|
+
## Development
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
git clone --recurse-submodules git@github.com:hoblin/toon-fu.git
|
|
123
|
+
cd toon-fu
|
|
124
|
+
bundle install
|
|
125
|
+
bundle exec rspec # unit specs + the spec's conformance fixtures
|
|
126
|
+
bundle exec standardrb # lint
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
The TOON spec is a git submodule at `spec/toon-spec`, pinned to its release tag; the fixtures run from there.
|
|
130
|
+
|
|
131
|
+
## Releasing
|
|
132
|
+
|
|
133
|
+
1. Bump `lib/toon_fu/version.rb` in a pull request and merge it.
|
|
134
|
+
2. `git tag vX.Y.Z && git push origin vX.Y.Z` on `main`.
|
|
135
|
+
3. Approve the `release` deployment in Actions.
|
|
15
136
|
|
|
16
|
-
|
|
137
|
+
The [release workflow](.github/workflows/release.yml) checks the tag matches the version, runs CI, and publishes to RubyGems via [trusted publishing](https://guides.rubygems.org/trusted-publishing/).
|
|
17
138
|
|
|
18
139
|
## License
|
|
19
140
|
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ToonFu
|
|
4
|
+
class DecimalLiteral
|
|
5
|
+
def initialize(value)
|
|
6
|
+
@value = value
|
|
7
|
+
end
|
|
8
|
+
|
|
9
|
+
def to_s
|
|
10
|
+
return "null" unless @value.finite?
|
|
11
|
+
return "0" if @value.zero?
|
|
12
|
+
return @value.to_s("F").delete_suffix(".0") if FloatLiteral::DECIMAL_RANGE.cover?(@value.abs)
|
|
13
|
+
|
|
14
|
+
sign, digits, _base, exponent = @value.split
|
|
15
|
+
mantissa = (digits.size > 1) ? "#{digits[0]}.#{digits[1..]}" : digits
|
|
16
|
+
"#{"-" if sign.negative?}#{mantissa}e#{format("%+d", exponent - 1)}"
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ToonFu
|
|
4
|
+
# Adds +to_toon+ to the core and standard-library classes TOON encodes.
|
|
5
|
+
# +BigDecimal+ is left out: it is a bundled gem the encoder accepts but
|
|
6
|
+
# does not load. Include this module into a class that defines +as_toon+
|
|
7
|
+
# to give that class +to_toon+ as well.
|
|
8
|
+
module Encodable
|
|
9
|
+
# @param options [Hash] see {Encoder#initialize}
|
|
10
|
+
# @return [String] the receiver encoded as TOON, as {ToonFu.encode} would
|
|
11
|
+
# @raise [Error] see {Encoder#encode}
|
|
12
|
+
def to_toon(**options)
|
|
13
|
+
ToonFu.encode(self, **options)
|
|
14
|
+
end
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
[Hash, Array, Set, String, Symbol, Integer, Float, TrueClass, FalseClass, NilClass, Time, Date].each do |klass|
|
|
18
|
+
klass.include(Encodable)
|
|
19
|
+
end
|
|
20
|
+
end
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ToonFu
|
|
4
|
+
# Encodes values with one set of options, for callers that encode many.
|
|
5
|
+
class Encoder
|
|
6
|
+
DELIMITERS = [",", "\t", "|"].freeze
|
|
7
|
+
|
|
8
|
+
# @param delimiter [String] the document delimiter, one of {DELIMITERS};
|
|
9
|
+
# strings containing it are quoted
|
|
10
|
+
# @param indent_size [Integer] spaces per nesting level
|
|
11
|
+
# @raise [ArgumentError] when the delimiter is not one of {DELIMITERS}
|
|
12
|
+
# or indent_size is not a positive Integer
|
|
13
|
+
def initialize(delimiter: ",", indent_size: 2)
|
|
14
|
+
raise ArgumentError, "delimiter must be one of #{DELIMITERS.inspect}, got #{delimiter.inspect}" unless DELIMITERS.include?(delimiter)
|
|
15
|
+
raise ArgumentError, "indent_size must be a positive Integer, got #{indent_size.inspect}" unless indent_size.is_a?(Integer) && indent_size.positive?
|
|
16
|
+
|
|
17
|
+
@delimiter = delimiter
|
|
18
|
+
@indent = " " * indent_size
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
# Encodes a value as TOON.
|
|
22
|
+
#
|
|
23
|
+
# Accepts the JSON data model plus these Ruby types, nested in any
|
|
24
|
+
# combination:
|
|
25
|
+
#
|
|
26
|
+
# - +Hash+ with String, Symbol or Integer keys, which become strings;
|
|
27
|
+
# +Array+; +Set+ as an array
|
|
28
|
+
# - +String+ in UTF-8, in an encoding that transcodes to it, or binary
|
|
29
|
+
# bytes that are valid UTF-8; +Symbol+ as its name
|
|
30
|
+
# - +Integer+ of any size as its exact digits; +Float+ and +BigDecimal+,
|
|
31
|
+
# with NaN and infinities as +null+, +BigDecimal+ as its exact digits;
|
|
32
|
+
# +true+, +false+, +nil+
|
|
33
|
+
# - +Date+ as an ISO 8601 date; +Time+ and +DateTime+ as ISO 8601
|
|
34
|
+
# timestamps keeping their offset, fraction digits up to the last
|
|
35
|
+
# non-zero one
|
|
36
|
+
# - objects that declare themselves a Hash, Array or String through
|
|
37
|
+
# Ruby's implicit conversions: +to_hash+, +to_ary+, +to_str+
|
|
38
|
+
# - any object responding to +as_toon+: its result is encoded instead,
|
|
39
|
+
# ahead of the mappings above
|
|
40
|
+
#
|
|
41
|
+
# @param value [Object] one of the types above
|
|
42
|
+
# @return [String] UTF-8
|
|
43
|
+
# @raise [Error] for any other value type; for keys other than String,
|
|
44
|
+
# Symbol or Integer, and keys that collide once converted to strings;
|
|
45
|
+
# for strings that are not valid UTF-8; for circular references,
|
|
46
|
+
# including an +as_toon+ or implicit conversion that leads back to its
|
|
47
|
+
# own object
|
|
48
|
+
def encode(value)
|
|
49
|
+
Writer.new(@delimiter, @indent).write(Normalizer.new.call(value))
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
end
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ToonFu
|
|
4
|
+
class Fields
|
|
5
|
+
attr_reader :columns
|
|
6
|
+
|
|
7
|
+
def self.of(rows)
|
|
8
|
+
return if rows.empty? || !rows.all?(Hash)
|
|
9
|
+
|
|
10
|
+
keys = rows.first.keys
|
|
11
|
+
return if keys.empty?
|
|
12
|
+
return unless rows.all? { |row| row.size == keys.size && keys.all? { |key| row.key?(key) } }
|
|
13
|
+
|
|
14
|
+
columns = keys.to_h do |key|
|
|
15
|
+
values = rows.map { |row| row[key] }
|
|
16
|
+
next [key, nil] if values.none? { |value| value.is_a?(Hash) || value.is_a?(Array) }
|
|
17
|
+
|
|
18
|
+
nested = of(values)
|
|
19
|
+
return nil unless nested
|
|
20
|
+
|
|
21
|
+
[key, nested]
|
|
22
|
+
end
|
|
23
|
+
new(columns)
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def initialize(columns)
|
|
27
|
+
@columns = columns
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def cells(row)
|
|
31
|
+
@columns.flat_map { |key, nested| nested ? nested.cells(row[key]) : [row[key]] }
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ToonFu
|
|
4
|
+
class FloatLiteral
|
|
5
|
+
DECIMAL_RANGE = (1e-6...1e21)
|
|
6
|
+
|
|
7
|
+
def initialize(value)
|
|
8
|
+
@value = value
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def to_s
|
|
12
|
+
return "null" unless @value.finite?
|
|
13
|
+
return "0" if @value.zero?
|
|
14
|
+
|
|
15
|
+
DECIMAL_RANGE.cover?(@value.abs) ? decimal : exponential
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
private
|
|
19
|
+
|
|
20
|
+
def decimal
|
|
21
|
+
plain = @value.to_s
|
|
22
|
+
return plain.delete_suffix(".0") unless plain.include?("e")
|
|
23
|
+
|
|
24
|
+
digits, point = significand
|
|
25
|
+
text =
|
|
26
|
+
if point <= 0
|
|
27
|
+
"0.#{"0" * -point}#{digits}"
|
|
28
|
+
elsif point >= digits.length
|
|
29
|
+
digits.ljust(point, "0")
|
|
30
|
+
else
|
|
31
|
+
"#{digits[0, point]}.#{digits[point..]}"
|
|
32
|
+
end
|
|
33
|
+
sign + text
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def exponential
|
|
37
|
+
mantissa, exponent = @value.abs.to_s.split("e")
|
|
38
|
+
"#{sign}#{mantissa.delete_suffix(".0")}e#{format("%+d", exponent.to_i)}"
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def significand
|
|
42
|
+
mantissa, exponent = @value.abs.to_s.split("e")
|
|
43
|
+
whole, fraction = mantissa.split(".")
|
|
44
|
+
fraction = "" if fraction == "0"
|
|
45
|
+
[whole + fraction, whole.length + exponent.to_i]
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def sign
|
|
49
|
+
@value.negative? ? "-" : ""
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
end
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ToonFu
|
|
4
|
+
class Normalizer
|
|
5
|
+
TRAILING_FRACTION_ZEROS = /\.?0+\z/
|
|
6
|
+
|
|
7
|
+
def initialize
|
|
8
|
+
@path = {}.compare_by_identity
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def call(value)
|
|
12
|
+
raise Error, "cannot encode a BasicObject" unless Kernel === value
|
|
13
|
+
return core(value) unless value.respond_to?(:as_toon)
|
|
14
|
+
|
|
15
|
+
within(value) { call(value.as_toon) }
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
private
|
|
19
|
+
|
|
20
|
+
def core(value)
|
|
21
|
+
case value
|
|
22
|
+
when nil, true, false, Integer, Float then value
|
|
23
|
+
when String then utf8(value)
|
|
24
|
+
when Symbol then utf8(value.name)
|
|
25
|
+
when Hash then within(value) { object(value) }
|
|
26
|
+
when Array, Set then within(value) { value.map { |element| call(element) } }
|
|
27
|
+
when Time then timestamp(value)
|
|
28
|
+
when DateTime then date_time(value)
|
|
29
|
+
when Date then value.iso8601
|
|
30
|
+
else convert(value)
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def convert(value)
|
|
35
|
+
if defined?(BigDecimal) && value.is_a?(BigDecimal) then DecimalLiteral.new(value)
|
|
36
|
+
elsif value.respond_to?(:to_hash) then within(value) { call(value.to_hash) }
|
|
37
|
+
elsif value.respond_to?(:to_ary) then within(value) { call(value.to_ary) }
|
|
38
|
+
elsif value.respond_to?(:to_str) then within(value) { call(value.to_str) }
|
|
39
|
+
else raise Error, "cannot encode #{value.class}; convert it first or define #as_toon"
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def object(hash)
|
|
44
|
+
hash.each_with_object({}) do |(key, value), result|
|
|
45
|
+
name = key_name(key)
|
|
46
|
+
raise Error, "duplicate key #{name.inspect} after converting keys to strings" if result.key?(name)
|
|
47
|
+
|
|
48
|
+
result[name] = call(value)
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def key_name(key)
|
|
53
|
+
case key
|
|
54
|
+
when String then utf8(key)
|
|
55
|
+
when Symbol then utf8(key.name)
|
|
56
|
+
when Integer then key.to_s
|
|
57
|
+
else raise Error, "cannot encode #{key.class} keys; use String, Symbol or Integer keys"
|
|
58
|
+
end
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def within(container)
|
|
62
|
+
raise Error, "cannot encode a circular reference through #{container.class}" if @path.key?(container)
|
|
63
|
+
|
|
64
|
+
@path[container] = true
|
|
65
|
+
result = yield
|
|
66
|
+
@path.delete(container)
|
|
67
|
+
result
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def timestamp(time)
|
|
71
|
+
moment = time.strftime("%Y-%m-%dT%H:%M:%S.%9N").sub(TRAILING_FRACTION_ZEROS, "")
|
|
72
|
+
"#{moment}#{time.utc? ? "Z" : time.strftime("%:z")}"
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def date_time(value)
|
|
76
|
+
moment, offset = value.iso8601(9).split(/(?=[+-]\d\d:\d\d\z)/)
|
|
77
|
+
"#{moment.sub(TRAILING_FRACTION_ZEROS, "")}#{offset}"
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
def utf8(string)
|
|
81
|
+
string = string.dup.force_encoding(Encoding::UTF_8) if string.encoding == Encoding::BINARY
|
|
82
|
+
string = string.encode(Encoding::UTF_8) unless string.encoding == Encoding::UTF_8
|
|
83
|
+
raise Error, "cannot encode a string that is not valid UTF-8: #{string.inspect}" unless string.valid_encoding?
|
|
84
|
+
|
|
85
|
+
string
|
|
86
|
+
rescue EncodingError => error
|
|
87
|
+
raise Error, "cannot encode a string as UTF-8: #{error.message}"
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
end
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ToonFu
|
|
4
|
+
class StringLiteral
|
|
5
|
+
READS_AS_LITERAL = /\A(?:true|false|null|[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?)\z/
|
|
6
|
+
UNSAFE = Encoder::DELIMITERS.to_h do |delimiter|
|
|
7
|
+
[delimiter, /\A[ \t#-]|[ \t]\z|[:"\\\[\]{}\x00-\x1f#{Regexp.escape(delimiter)}]/]
|
|
8
|
+
end.freeze
|
|
9
|
+
BARE_KEY = /\A[A-Za-z_][A-Za-z0-9_.]*\z/
|
|
10
|
+
ESCAPABLE = /["\\\x00-\x1f]/
|
|
11
|
+
ESCAPES = (0x00..0x1f).to_h { |code| [code.chr, format("\\u%04x", code)] }
|
|
12
|
+
.merge("\\" => "\\\\", '"' => '\\"', "\n" => "\\n", "\r" => "\\r", "\t" => "\\t").freeze
|
|
13
|
+
|
|
14
|
+
def initialize(delimiter)
|
|
15
|
+
@unsafe = UNSAFE.fetch(delimiter)
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def encode(string)
|
|
19
|
+
quote?(string) ? quote(string) : string
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def key(string)
|
|
23
|
+
BARE_KEY.match?(string) ? string : quote(string)
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
private
|
|
27
|
+
|
|
28
|
+
def quote(string)
|
|
29
|
+
%("#{string.gsub(ESCAPABLE, ESCAPES)}")
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def quote?(string)
|
|
33
|
+
string.empty? || READS_AS_LITERAL.match?(string) || @unsafe.match?(string)
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|
data/lib/toon_fu/version.rb
CHANGED
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ToonFu
|
|
4
|
+
class Writer
|
|
5
|
+
def initialize(delimiter, indent)
|
|
6
|
+
@delimiter = delimiter
|
|
7
|
+
@marker = (delimiter == ",") ? "" : delimiter
|
|
8
|
+
@strings = StringLiteral.new(delimiter)
|
|
9
|
+
@unit = indent
|
|
10
|
+
@indent = ""
|
|
11
|
+
@hyphen = nil
|
|
12
|
+
@lines = []
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def write(value)
|
|
16
|
+
value(value)
|
|
17
|
+
@lines.join("\n").force_encoding(Encoding::UTF_8)
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
private
|
|
21
|
+
|
|
22
|
+
def value(value)
|
|
23
|
+
case value
|
|
24
|
+
when Hash then mapping("", value)
|
|
25
|
+
when Array then array("", value)
|
|
26
|
+
else line(scalar(value))
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def object(hash)
|
|
31
|
+
hash.each do |key, value|
|
|
32
|
+
name = @strings.key(key)
|
|
33
|
+
case value
|
|
34
|
+
when Hash then mapping(name, value)
|
|
35
|
+
when Array then array(name, value)
|
|
36
|
+
else line("#{name}: #{scalar(value)}")
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def mapping(name, hash)
|
|
42
|
+
if hash.size >= 2 && (fields = Fields.of(hash.values))
|
|
43
|
+
line("#{name}#{header(hash.size, fields, keyed: true)}")
|
|
44
|
+
nested { hash.each { |key, entry| line("#{@strings.key(key)}: #{row(fields.cells(entry))}") } }
|
|
45
|
+
elsif name.empty?
|
|
46
|
+
object(hash)
|
|
47
|
+
else
|
|
48
|
+
line("#{name}:")
|
|
49
|
+
nested { object(hash) }
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def array(name, values)
|
|
54
|
+
listed = name.empty? && @hyphen
|
|
55
|
+
if values.empty? && !listed
|
|
56
|
+
line(name.empty? ? "[]" : "#{name}: []")
|
|
57
|
+
elsif !listed && (fields = Fields.of(values))
|
|
58
|
+
line("#{name}#{header(values.size, fields)}")
|
|
59
|
+
nested { values.each { |element| line(row(fields.cells(element))) } }
|
|
60
|
+
elsif values.none? { |value| value.is_a?(Hash) || value.is_a?(Array) }
|
|
61
|
+
line("#{name}#{inline(values)}")
|
|
62
|
+
else
|
|
63
|
+
line("#{name}#{header(values.size)}")
|
|
64
|
+
nested { values.each { |element| item(element) } }
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def item(element)
|
|
69
|
+
@hyphen = @indent
|
|
70
|
+
case element
|
|
71
|
+
when Array then array("", element)
|
|
72
|
+
when Hash then nested { object(element) }
|
|
73
|
+
else line(scalar(element))
|
|
74
|
+
end
|
|
75
|
+
return unless @hyphen
|
|
76
|
+
|
|
77
|
+
@lines << "#{@hyphen}-"
|
|
78
|
+
@hyphen = nil
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
def inline(values)
|
|
82
|
+
return header(values.size) if values.empty?
|
|
83
|
+
|
|
84
|
+
"#{header(values.size)} #{row(values)}"
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
def row(values)
|
|
88
|
+
values.map { |value| scalar(value) }.join(@delimiter)
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def header(count, fields = nil, keyed: false)
|
|
92
|
+
"[#{count}#{":" if keyed}#{@marker}]#{field_list(fields) if fields}:"
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
def field_list(fields)
|
|
96
|
+
names = fields.columns.map { |key, nested| "#{@strings.key(key)}#{field_list(nested) if nested}" }
|
|
97
|
+
"{#{names.join(@delimiter)}}"
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def line(text)
|
|
101
|
+
if @hyphen
|
|
102
|
+
@lines << "#{@hyphen}- #{text}"
|
|
103
|
+
@hyphen = nil
|
|
104
|
+
else
|
|
105
|
+
@lines << "#{@indent}#{text}"
|
|
106
|
+
end
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
def nested
|
|
110
|
+
outer = @indent
|
|
111
|
+
@indent += @unit
|
|
112
|
+
yield
|
|
113
|
+
ensure
|
|
114
|
+
@indent = outer
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
def scalar(value)
|
|
118
|
+
case value
|
|
119
|
+
when nil then "null"
|
|
120
|
+
when true, false, Integer then value.to_s
|
|
121
|
+
when Float then FloatLiteral.new(value).to_s
|
|
122
|
+
when DecimalLiteral then value.to_s
|
|
123
|
+
when String then @strings.encode(value)
|
|
124
|
+
else raise Error, "cannot encode #{value.class}"
|
|
125
|
+
end
|
|
126
|
+
end
|
|
127
|
+
end
|
|
128
|
+
end
|
data/lib/toon_fu.rb
CHANGED
|
@@ -1,8 +1,40 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "date"
|
|
4
|
+
|
|
3
5
|
require_relative "toon_fu/version"
|
|
6
|
+
require_relative "toon_fu/encoder"
|
|
7
|
+
require_relative "toon_fu/float_literal"
|
|
8
|
+
require_relative "toon_fu/decimal_literal"
|
|
9
|
+
require_relative "toon_fu/string_literal"
|
|
10
|
+
require_relative "toon_fu/normalizer"
|
|
11
|
+
require_relative "toon_fu/fields"
|
|
12
|
+
require_relative "toon_fu/writer"
|
|
13
|
+
require_relative "toon_fu/encodable"
|
|
4
14
|
|
|
5
15
|
# TOON (Token-Oriented Object Notation) for Ruby.
|
|
6
16
|
module ToonFu
|
|
17
|
+
# Raised when a value has no TOON representation; see {Encoder#encode}.
|
|
7
18
|
class Error < StandardError; end
|
|
19
|
+
|
|
20
|
+
private_constant :FloatLiteral, :DecimalLiteral, :StringLiteral, :Normalizer, :Fields, :Writer
|
|
21
|
+
|
|
22
|
+
# Encodes a value as TOON.
|
|
23
|
+
#
|
|
24
|
+
# ToonFu.encode("hello") # => "hello"
|
|
25
|
+
# ToonFu.encode("a,b") # => "\"a,b\""
|
|
26
|
+
# ToonFu.encode("a,b", delimiter: "|") # => "a,b"
|
|
27
|
+
# ToonFu.encode(1e-7) # => "1e-7"
|
|
28
|
+
# ToonFu.encode({user: {id: 1}}) # => "user:\n id: 1"
|
|
29
|
+
# ToonFu.encode({tags: ["a", "b"]}) # => "tags[2]: a,b"
|
|
30
|
+
# ToonFu.encode([{id: 1}, {id: 2}]) # => "[2]{id}:\n 1\n 2"
|
|
31
|
+
# ToonFu.encode({a: {x: 1}, b: {x: 2}}) # => "[2:]{x}:\n a: 1\n b: 2"
|
|
32
|
+
#
|
|
33
|
+
# @param value [Object] see {Encoder#encode} for the accepted types
|
|
34
|
+
# @param options [Hash] see {Encoder#initialize}
|
|
35
|
+
# @return [String]
|
|
36
|
+
# @raise [Error] see {Encoder#encode}
|
|
37
|
+
def self.encode(value, **options)
|
|
38
|
+
Encoder.new(**options).encode(value)
|
|
39
|
+
end
|
|
8
40
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: toon-fu
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version:
|
|
4
|
+
version: 4.1.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Yevhenii Hurin
|
|
@@ -18,17 +18,24 @@ executables: []
|
|
|
18
18
|
extensions: []
|
|
19
19
|
extra_rdoc_files: []
|
|
20
20
|
files:
|
|
21
|
-
- CHANGELOG.md
|
|
22
21
|
- LICENSE
|
|
23
22
|
- README.md
|
|
24
23
|
- lib/toon_fu.rb
|
|
24
|
+
- lib/toon_fu/decimal_literal.rb
|
|
25
|
+
- lib/toon_fu/encodable.rb
|
|
26
|
+
- lib/toon_fu/encoder.rb
|
|
27
|
+
- lib/toon_fu/fields.rb
|
|
28
|
+
- lib/toon_fu/float_literal.rb
|
|
29
|
+
- lib/toon_fu/normalizer.rb
|
|
30
|
+
- lib/toon_fu/string_literal.rb
|
|
25
31
|
- lib/toon_fu/version.rb
|
|
32
|
+
- lib/toon_fu/writer.rb
|
|
26
33
|
homepage: https://github.com/hoblin/toon-fu
|
|
27
34
|
licenses:
|
|
28
35
|
- MIT
|
|
29
36
|
metadata:
|
|
30
37
|
source_code_uri: https://github.com/hoblin/toon-fu
|
|
31
|
-
changelog_uri: https://github.com/hoblin/toon-fu/
|
|
38
|
+
changelog_uri: https://github.com/hoblin/toon-fu/releases
|
|
32
39
|
rubygems_mfa_required: 'true'
|
|
33
40
|
rdoc_options: []
|
|
34
41
|
require_paths:
|