wtf8 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1cd5f73f969ea12de1b2d7083a4095699985edf7452d434e231603a76ea78491
-  data.tar.gz: 1a8ac861c756ba15117208e7e3b6117792176d6a7b7ffa0c844b54fbfaa6ef2a
+  metadata.gz: e7ce5d331e0d651f4c9459fed8f598edeb395f7a1edcecba524db6c08bfebfb1
+  data.tar.gz: b859dc0c6c5642d8c802486318e6b71c8e660cf716a4a3b29936c7ccfe5dc410
 SHA512:
-  metadata.gz: 571982f037b24f6e546917f1320d66ed3dd9c74dfdbaa2ddf310a2a1d964cc9548cd8d0f89aed5b58f0d2582d1ab7e04fd4e1295c342a1ee71050461601655b3
-  data.tar.gz: 5d37234457c20489781fad90ba694bc5a0225f0371adb3ae11d5c95efbbbd28669b9e0d4e0f6d91af41a14e2538c782ad312f4ff2f6499c585fe6634f1d4492d
+  metadata.gz: 956122b0ec988b861939aaf643ed7b5b62bbdf9fb7802216784414316d89ca70ef0811b4ed6225216f137f84ea2c2483816933ab77112ec2375f5ffbcfb5da68
+  data.tar.gz: 5bce8715f5fb0c813876a26f67e6f03b3062030daebd59cdc906ac76e2d37906a50b99fa4a10a8cf3adbfb6ac32f50fd779b13f52fa5963dba7c053172ece099

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Marco Roth
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -1,39 +1,148 @@
-# Wtf8
+<h2 align="center">🤨 WTF-8 for Ruby</h2>
 
-TODO: Delete this and the text below, and describe your gem
+<h4 align="center">A superset of UTF-8 that encodes surrogate code points if they are not in a pair.</h4>
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/wtf8`. To experiment with that code, run `bin/console` for an interactive prompt.
+<div align="center">Ruby implementation of <a href="https://wtf-8.codeberg.page">WTF-8</a>, the encoding for potentially ill-formed UTF-16.</div><br/>
 
-## Installation
+<p align="center">
+  <a href="https://rubygems.org/gems/wtf8"><img alt="Gem Version" src="https://img.shields.io/gem/v/wtf8"></a>
+  <a href="https://wtf-8.codeberg.page"><img alt="Specification" src="https://img.shields.io/badge/wtf--8.codeberg.page-specification-green"></a>
+  <a href="https://github.com/marcoroth/wtf8-ruby/blob/main/LICENSE.txt"><img alt="License" src="https://img.shields.io/github/license/marcoroth/wtf8-ruby"></a>
+  <a href="https://github.com/marcoroth/wtf8-ruby/issues"><img alt="Issues" src="https://img.shields.io/github/issues/marcoroth/wtf8-ruby"></a>
+</p>
 
-TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
+<br/>
 
-Install the gem and add to the application's Gemfile by executing:
+### What is WTF-8 for Ruby?
 
-```bash
-bundle add UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
+Windows filenames, JavaScript strings, Java strings, and JSON `\uD800` escapes are all sequences of 16-bit code units that are only *potentially* UTF-16. An unpaired surrogate is a legal value in all of them, but it isn't a Unicode scalar value, so UTF-8 can't encode it. Ruby refuses to convert one:
+
+```ruby
+[0xD800].pack("v").force_encoding("UTF-16LE").encode("UTF-8")
+# Encoding::InvalidByteSequenceError: incomplete "\x00\xD8" on UTF-16LE
+```
+
+WTF-8 fills that gap. It encodes a surrogate the same way UTF-8 encodes any other code point, and encodes a surrogate *pair* as the single four-byte sequence for the code point it represents. That second rule is what makes conversion to and from potentially ill-formed UTF-16 lossless in both directions.
+
+This gem implements [the WTF-8 specification](https://wtf-8.codeberg.page) by Simon Sapin. Rust uses WTF-8 internally to represent Windows paths, and there is a [Rust implementation](https://crates.io/crates/wtf8) by the same author.
+
+```ruby
+WTF8.from_utf16le([0xD800].pack("v")).code_points
+#=> [55296]
 ```
 
-If bundler is not being used to manage dependencies, install the gem by executing:
+### Installation
 
 ```bash
-gem install UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
+bundle add wtf8
 ```
 
-## Usage
+Pure Ruby, no dependencies and no native extension. Requires Ruby 3.2 or later.
+
+### Usage
+
+#### Reading UTF-16
+
+```ruby
+WTF8.from_utf16le(bytes)            # also from_utf16be and from_utf16
+WTF8.from_utf16([0xD83D, 0xDE00])   #=> #<WTF8::String "😀">
+WTF8.from_utf16([0xD83D])           #=> #<WTF8::String "\u{D83D}">
+```
+
+UTF-8 is a subset of WTF-8, so reading a UTF-8 string costs nothing:
+
+```ruby
+WTF8.from_utf8("héllo 😀")
+```
+
+#### Converting back
+
+`to_utf16` is exact. `to_utf8` isn't, since UTF-8 has no encoding for a surrogate, so each one is replaced with U+FFFD.
+
+```ruby
+string = WTF8.from_code_points([0x61, 0xD800, 0x62])
+
+string.to_utf16                     #=> [97, 55296, 98]
+string.to_utf8                      #=> "a�b"
+string.to_utf8(replacement: "?")    #=> "a?b"
+```
+
+One surrogate costs one replacement character. Ruby's `String#scrub` sees three invalid bytes rather than one invalid code point, and spends three:
+
+```ruby
+"\xED\xA0\x80".dup.force_encoding("UTF-8").scrub("?")   #=> "???"
+```
+
+#### WTF8::String
+
+A Ruby String can store these bytes, since it's bytes plus an encoding tag, but it can't interpret them. Tagged as UTF-8, three bytes of lone surrogate have a `length` of 3 and raise from `codepoints`.
 
-TODO: Write usage instructions here
+```ruby
+string = WTF8.from_code_points([0x61, 0xD83D])
 
-## Development
+string.length         #=> 2
+string.code_points    #=> [97, 55357]
+string.well_formed?   #=> false
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+string.raw.dup.force_encoding("UTF-8").length   #=> 4
+```
+
+Instances are frozen, validated on construction, and compare and hash by their bytes.
+
+#### Concatenation
+
+Concatenating WTF-8 isn't the same as concatenating bytes. If the left side ends with a lead surrogate and the right side starts with a trail surrogate, the two form a pair, and a pair has to be encoded as one four-byte sequence. Six bytes become four, and the result is one code point shorter than its halves were:
+
+```ruby
+left = WTF8.from_code_points([0x61, 0xD83D])
+right = WTF8.from_code_points([0xDE00, 0x62])
+
+left.length + right.length   #=> 4
+(left + right).length        #=> 3
+(left + right).to_s          #=> "a😀b"
+```
+
+Slicing is the same thing in reverse. It's defined in UTF-16 code units, because that's the only index space where the halves of a supplementary code point can be addressed:
+
+```ruby
+emoji = WTF8.from_utf8("a😀b")
+
+emoji.slice_utf16(0..1)   #=> #<WTF8::String "a\u{D83D}">
+emoji.slice_utf16(2..3)   #=> #<WTF8::String "\u{DE00}b">
+
+emoji.slice_utf16(0..1) + emoji.slice_utf16(2..3) == emoji   #=> true
+```
+
+#### CESU-8 and Modified UTF-8
+
+CESU-8 inverts the rule WTF-8 uses for supplementary code points. It always encodes them as two surrogates of three bytes each, where WTF-8 always encodes them as one four-byte sequence. Java's Modified UTF-8 is CESU-8 with U+0000 encoded as `C0 80`, so that an encoded string never contains a NUL byte. `DataOutputStream.writeUTF`, `.class` constant pools, dex files and JNI all use it.
+
+```ruby
+WTF8.from_cesu8(bytes)
+WTF8.from_modified_utf8(bytes)
+
+WTF8.from_utf8("😀").to_cesu8.bytes   #=> [237, 160, 189, 237, 184, 128]
+```
+
+Ruby has a built-in CESU-8 transcoder, and for well-formed input this produces identical bytes. Ruby's rejects lone surrogates in both directions; this one doesn't.
+
+#### Working with bytes
+
+`WTF8::Codec` offers the same operations on plain binary strings, without the wrapper object:
+
+```ruby
+WTF8::Codec.decode("\xED\xA0\x80")   #=> [55296]
+WTF8::Codec.encode([0xD800]).bytes   #=> [237, 160, 128]
+WTF8::Codec.concat(left, right)
+WTF8::Codec.valid?("\xED\xA0\xBD\xED\xB8\x80")   #=> false
+```
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+### Don't use it for interchange
 
-## Contributing
+WTF-8 is an internal representation. Reading UTF-8 as WTF-8 is safe, but WTF-8 is not UTF-8 and must not be sent as though it were. The byte sequences it adds are exactly the ones a UTF-8 decoder is required to reject, and inconsistent handling of them across a pipeline is a known validation-bypass vector.
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/marcoroth/wtf8. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/marcoroth/wtf8/blob/main/CODE_OF_CONDUCT.md).
+Convert with `to_utf8` at the boundary. Don't serve it as `charset=utf-8`, don't write it to a file another program will read as UTF-8, and don't store it in a UTF-8 column.
 
-## Code of Conduct
+### License
 
-Everyone interacting in the Wtf8 project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/marcoroth/wtf8/blob/main/CODE_OF_CONDUCT.md).
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/wtf8/cesu8.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module WTF8
+  module CESU8
+    #: (::String) -> ::String
+    def self.encode(bytes)
+      cesu8 = +"".b
+
+      Codec.decode(bytes).each do |code_point|
+        if code_point > 0xFFFF
+          Surrogates.split(code_point).each { |unit| cesu8 << [unit].pack("U").b }
+        else
+          cesu8 << [code_point].pack("U").b
+        end
+      end
+
+      cesu8
+    end
+
+    #: (::String, ?strict: bool) -> ::String
+    def self.decode(bytes, strict: true)
+      code_points = Codec.unpack(bytes)
+
+      wtf8 = +"".b
+      index = 0
+
+      while index < code_points.length
+        code_point = code_points[index]
+        following = code_points[index + 1]
+
+        if code_point > Codec::MAX_CODE_POINT
+          raise InvalidCodePointError, format("U+%04X is above U+10FFFF", code_point)
+        elsif code_point > 0xFFFF
+          raise InvalidByteSequenceError, "a four-byte sequence is not CESU-8" if strict
+
+          wtf8 << [code_point].pack("U").b
+          index += 1
+        elsif Surrogates.lead?(code_point) && following && Surrogates.trail?(following)
+          wtf8 << [Surrogates.combine(code_point, following)].pack("U").b
+          index += 2
+        else
+          wtf8 << [code_point].pack("U").b
+          index += 1
+        end
+      end
+
+      wtf8
+    end
+  end
+end

data/lib/wtf8/codec.rb

@@ -0,0 +1,170 @@
+# frozen_string_literal: true
+
+module WTF8
+  module Codec
+    MAX_CODE_POINT = 0x10FFFF
+    REPLACEMENT_CHARACTER = "\u{FFFD}"
+
+    #: (::String) -> Array[Integer]
+    def self.decode(bytes)
+      code_points = unpack(bytes)
+
+      code_points.each do |code_point|
+        next unless code_point > MAX_CODE_POINT
+
+        raise InvalidCodePointError, format("U+%04X is above U+10FFFF", code_point)
+      end
+
+      code_points.each_cons(2) do |pair|
+        lead = pair[0]
+        trail = pair[1]
+
+        next unless Surrogates.lead?(lead) && Surrogates.trail?(trail)
+
+        raise InvalidByteSequenceError, format("U+%<lead>04X U+%<trail>04X is an encoded surrogate pair, which WTF-8 writes as one four-byte sequence", lead: lead, trail: trail)
+      end
+
+      code_points
+    end
+
+    #: (::String) -> Array[Integer]
+    def self.unpack(bytes)
+      bytes.b.unpack("U*") #: Array[Integer]
+    rescue ArgumentError => e
+      raise InvalidByteSequenceError, e.message
+    end
+
+    #: (Array[Integer]) -> ::String
+    def self.encode(code_points)
+      code_points.each do |code_point|
+        next if code_point.is_a?(Integer) && code_point >= 0 && code_point <= MAX_CODE_POINT
+
+        raise InvalidCodePointError, "#{code_point.inspect} is not a code point in U+0000..U+10FFFF"
+      end
+
+      code_points.each_cons(2) do |pair|
+        next unless Surrogates.lead?(pair[0]) && Surrogates.trail?(pair[1])
+
+        raise InvalidCodePointError, "a surrogate pair has to be combined before it is encoded, which from_utf16 does"
+      end
+
+      code_points.each_with_object(+"".b) { |code_point, bytes| bytes << [code_point].pack("U").b }
+    end
+
+    #: (::String) -> bool
+    def self.valid?(bytes)
+      decode(bytes)
+      true
+    rescue Error
+      false
+    end
+
+    #: (::String) -> bool
+    def self.well_formed?(bytes)
+      bytes.b.dup.force_encoding(Encoding::UTF_8).valid_encoding?
+    end
+
+    #: (::String) -> ::String
+    def self.from_utf8(string)
+      string = string.encode(Encoding::UTF_8) unless string.encoding == Encoding::UTF_8 || string.ascii_only?
+
+      raise InvalidByteSequenceError, "not valid UTF-8" unless string.valid_encoding?
+
+      string.b
+    end
+
+    #: (::String, ?replacement: ::String) -> ::String
+    def self.to_utf8(bytes, replacement: REPLACEMENT_CHARACTER)
+      utf8 = +""
+
+      decode(bytes).each do |code_point|
+        utf8 << (Surrogates.surrogate?(code_point) ? replacement : [code_point].pack("U"))
+      end
+
+      utf8.force_encoding(Encoding::UTF_8)
+    end
+
+    #: (Array[Integer]) -> ::String
+    def self.from_utf16(units)
+      bytes = +"".b
+      index = 0
+
+      while index < units.length
+        unit = units[index]
+
+        unless unit.is_a?(Integer) && unit >= 0 && unit <= 0xFFFF
+          raise InvalidCodePointError, "#{unit.inspect} is not a UTF-16 code unit"
+        end
+
+        following = units[index + 1]
+
+        if Surrogates.lead?(unit) && following && Surrogates.trail?(following)
+          bytes << [Surrogates.combine(unit, following)].pack("U").b
+          index += 2
+        else
+          bytes << [unit].pack("U").b
+          index += 1
+        end
+      end
+
+      bytes
+    end
+
+    #: (::String) -> Array[Integer]
+    def self.to_utf16(bytes)
+      decode(bytes).flat_map { |code_point| code_point > 0xFFFF ? Surrogates.split(code_point) : code_point }
+    end
+
+    #: (::String) -> ::String
+    def self.from_utf16le(bytes)
+      units = bytes.b.unpack("v*") #: Array[Integer]
+
+      from_utf16(units)
+    end
+
+    #: (::String) -> ::String
+    def self.from_utf16be(bytes)
+      units = bytes.b.unpack("n*") #: Array[Integer]
+
+      from_utf16(units)
+    end
+
+    #: (::String) -> ::String
+    def self.to_utf16le(bytes)
+      to_utf16(bytes).pack("v*")
+    end
+
+    #: (::String) -> ::String
+    def self.to_utf16be(bytes)
+      to_utf16(bytes).pack("n*")
+    end
+
+    #: (::String, ::String) -> ::String
+    def self.concat(left, right)
+      left = left.b
+      right = right.b
+
+      return left + right unless ends_with_lead_surrogate?(left) && starts_with_trail_surrogate?(right)
+
+      lead_bytes = left.byteslice(-3, 3) #: ::String
+      trail_bytes = right.byteslice(0, 3) #: ::String
+      head = left.byteslice(0, left.bytesize - 3) #: ::String
+      tail = right.byteslice(3, right.bytesize - 3) #: ::String
+
+      lead = lead_bytes.unpack1("U") #: Integer
+      trail = trail_bytes.unpack1("U") #: Integer
+
+      head + [Surrogates.combine(lead, trail)].pack("U").b + tail
+    end
+
+    #: (::String) -> bool
+    def self.ends_with_lead_surrogate?(bytes)
+      bytes.bytesize >= 3 && bytes.getbyte(-3) == Surrogates::PREFIX_BYTE && Surrogates::LEAD_SECOND_BYTE.cover?(bytes.getbyte(-2))
+    end
+
+    #: (::String) -> bool
+    def self.starts_with_trail_surrogate?(bytes)
+      bytes.bytesize >= 3 && bytes.getbyte(0) == Surrogates::PREFIX_BYTE && Surrogates::TRAIL_SECOND_BYTE.cover?(bytes.getbyte(1))
+    end
+  end
+end

data/lib/wtf8/errors.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module WTF8
+  class Error < StandardError; end
+  class InvalidByteSequenceError < Error; end
+  class InvalidCodePointError < Error; end
+end

data/lib/wtf8/modified_utf8.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module WTF8
+  module ModifiedUTF8
+    NUL = "\x00"
+    ENCODED_NUL = "\xC0\x80"
+
+    #: (::String) -> ::String
+    def self.encode(bytes)
+      CESU8.encode(bytes).gsub(NUL.b, ENCODED_NUL.b)
+    end
+
+    #: (::String, ?strict: bool) -> ::String
+    def self.decode(bytes, strict: true)
+      CESU8.decode(bytes.b.gsub(ENCODED_NUL.b, NUL.b), strict: strict)
+    end
+  end
+end

data/lib/wtf8/string.rb

@@ -0,0 +1,148 @@
+# frozen_string_literal: true
+
+module WTF8
+  class String
+    include Comparable
+
+    #: (::String) -> WTF8::String
+    def self.from_utf8(string) = new(Codec.from_utf8(string))
+
+    #: (Array[Integer]) -> WTF8::String
+    def self.from_utf16(units) = new(Codec.from_utf16(units))
+
+    #: (::String) -> WTF8::String
+    def self.from_utf16le(bytes) = new(Codec.from_utf16le(bytes))
+
+    #: (::String) -> WTF8::String
+    def self.from_utf16be(bytes) = new(Codec.from_utf16be(bytes))
+
+    #: (Array[Integer]) -> WTF8::String
+    def self.from_code_points(code_points) = new(Codec.encode(code_points))
+
+    #: (::String, ?strict: bool) -> WTF8::String
+    def self.from_cesu8(bytes, strict: true) = new(CESU8.decode(bytes, strict: strict))
+
+    #: (::String, ?strict: bool) -> WTF8::String
+    def self.from_modified_utf8(bytes, strict: true) = new(ModifiedUTF8.decode(bytes, strict: strict))
+
+    attr_reader :raw #: ::String
+    attr_reader :code_points #: Array[Integer]
+
+    #: (::String) -> void
+    def initialize(bytes)
+      @raw = bytes.b.freeze
+      @code_points = Codec.decode(@raw).freeze
+
+      freeze
+    end
+
+    #: () -> Integer
+    def length = code_points.length
+
+    alias size length
+
+    #: () -> Integer
+    def bytesize = @raw.bytesize
+
+    #: () -> bool
+    def empty? = @raw.empty?
+
+    #: () -> bool
+    def well_formed? = Codec.well_formed?(@raw)
+
+    #: () { (Integer) -> void } -> WTF8::String
+    #: () -> Enumerator[Integer, WTF8::String]
+    def each_code_point(&block)
+      return enum_for(:each_code_point) unless block
+
+      code_points.each(&block)
+
+      self
+    end
+
+    #: (WTF8::String | ::String) -> WTF8::String
+    def +(other)
+      self.class.new(Codec.concat(@raw, other.is_a?(WTF8::String) ? other.raw : Codec.from_utf8(other)))
+    end
+
+    #: (Integer) -> WTF8::String?
+    #: (Range[Integer]) -> WTF8::String?
+    def [](index)
+      case index
+      when Range
+        sliced = code_points[index]
+
+        sliced && self.class.from_code_points(sliced)
+      else
+        code_point = code_points[index]
+
+        code_point && self.class.from_code_points([code_point])
+      end
+    end
+
+    alias slice []
+
+    #: (Range[Integer]) -> WTF8::String
+    def slice_utf16(range)
+      self.class.from_utf16(to_utf16[range] || [])
+    end
+
+    #: (WTF8::String | ::String) -> bool
+    def start_with?(prefix)
+      @raw.start_with?(prefix.is_a?(WTF8::String) ? prefix.raw : Codec.from_utf8(prefix))
+    end
+
+    #: (WTF8::String | ::String) -> bool
+    def end_with?(suffix)
+      @raw.end_with?(suffix.is_a?(WTF8::String) ? suffix.raw : Codec.from_utf8(suffix))
+    end
+
+    #: (?replacement: ::String) -> ::String
+    def to_utf8(replacement: Codec::REPLACEMENT_CHARACTER) = Codec.to_utf8(@raw, replacement: replacement)
+
+    alias to_s to_utf8
+
+    #: () -> Array[Integer]
+    def to_utf16 = Codec.to_utf16(@raw)
+
+    #: () -> ::String
+    def to_utf16le = Codec.to_utf16le(@raw)
+
+    #: () -> ::String
+    def to_utf16be = Codec.to_utf16be(@raw)
+
+    #: () -> ::String
+    def to_cesu8 = CESU8.encode(@raw)
+
+    #: () -> ::String
+    def to_modified_utf8 = ModifiedUTF8.encode(@raw)
+
+    #: (untyped) -> bool
+    def ==(other)
+      other.is_a?(WTF8::String) && other.raw == @raw
+    end
+
+    alias eql? ==
+
+    #: () -> Integer
+    def hash = [self.class, @raw].hash
+
+    #: (untyped) -> Integer?
+    def <=>(other)
+      other.is_a?(WTF8::String) ? (@raw <=> other.raw) : nil
+    end
+
+    #: () -> ::String
+    def inspect
+      shown = code_points.map { |code_point|
+        if Surrogates.surrogate?(code_point)
+          format("\\u{%04X}", code_point)
+        else
+          [code_point].pack("U").inspect[1..-2]
+        end
+      }.join
+
+      %(#<#{self.class.name} "#{shown}">)
+    end
+  end
+end

data/lib/wtf8/surrogates.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module WTF8
+  module Surrogates
+    RANGE = 0xD800..0xDFFF
+    LEAD_RANGE = 0xD800..0xDBFF
+    TRAIL_RANGE = 0xDC00..0xDFFF
+
+    PREFIX_BYTE = 0xED
+    LEAD_SECOND_BYTE = 0xA0..0xAF
+    TRAIL_SECOND_BYTE = 0xB0..0xBF
+
+    #: (Integer) -> bool
+    def self.surrogate?(code_point)
+      RANGE.cover?(code_point)
+    end
+
+    #: (Integer) -> bool
+    def self.lead?(code_point)
+      LEAD_RANGE.cover?(code_point)
+    end
+
+    #: (Integer) -> bool
+    def self.trail?(code_point)
+      TRAIL_RANGE.cover?(code_point)
+    end
+
+    #: (Integer, Integer) -> Integer
+    def self.combine(lead, trail)
+      0x10000 + ((lead - 0xD800) << 10) + (trail - 0xDC00)
+    end
+
+    #: (Integer) -> Array[Integer]
+    def self.split(code_point)
+      remainder = code_point - 0x10000
+
+      [0xD800 + (remainder >> 10), 0xDC00 + (remainder & 0x3FF)]
+    end
+  end
+end

data/lib/wtf8/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module WTF8
-  VERSION = "0.0.1"
+  VERSION = "0.1.0"
 end

data/lib/wtf8.rb

@@ -1,6 +1,38 @@
 # frozen_string_literal: true
 
 require_relative "wtf8/version"
+require_relative "wtf8/errors"
+require_relative "wtf8/surrogates"
+require_relative "wtf8/codec"
+require_relative "wtf8/cesu8"
+require_relative "wtf8/modified_utf8"
+require_relative "wtf8/string"
 
 module WTF8
+  #: (::String) -> WTF8::String
+  def self.from_utf8(string) = String.from_utf8(string)
+
+  #: (Array[Integer]) -> WTF8::String
+  def self.from_utf16(units) = String.from_utf16(units)
+
+  #: (::String) -> WTF8::String
+  def self.from_utf16le(bytes) = String.from_utf16le(bytes)
+
+  #: (::String) -> WTF8::String
+  def self.from_utf16be(bytes) = String.from_utf16be(bytes)
+
+  #: (Array[Integer]) -> WTF8::String
+  def self.from_code_points(code_points) = String.from_code_points(code_points)
+
+  #: (::String, ?strict: bool) -> WTF8::String
+  def self.from_cesu8(bytes, strict: true) = String.from_cesu8(bytes, strict: strict)
+
+  #: (::String, ?strict: bool) -> WTF8::String
+  def self.from_modified_utf8(bytes, strict: true) = String.from_modified_utf8(bytes, strict: strict)
+
+  #: (::String) -> WTF8::String
+  def self.from_wtf8(bytes) = String.new(bytes)
+
+  #: (::String) -> bool
+  def self.valid?(bytes) = Codec.valid?(bytes)
 end

data/sig/wtf8/cesu8.rbs

@@ -0,0 +1,11 @@
+# Generated from lib/wtf8/cesu8.rb with RBS::Inline
+
+module WTF8
+  module CESU8
+    # : (::String) -> ::String
+    def self.encode: (::String) -> ::String
+
+    # : (::String, ?strict: bool) -> ::String
+    def self.decode: (::String, ?strict: bool) -> ::String
+  end
+end

data/sig/wtf8/codec.rbs

@@ -0,0 +1,57 @@
+# Generated from lib/wtf8/codec.rb with RBS::Inline
+
+module WTF8
+  module Codec
+    MAX_CODE_POINT: ::Integer
+
+    REPLACEMENT_CHARACTER: ::String
+
+    # : (::String) -> Array[Integer]
+    def self.decode: (::String) -> Array[Integer]
+
+    # : (::String) -> Array[Integer]
+    def self.unpack: (::String) -> Array[Integer]
+
+    # : (Array[Integer]) -> ::String
+    def self.encode: (Array[Integer]) -> ::String
+
+    # : (::String) -> bool
+    def self.valid?: (::String) -> bool
+
+    # : (::String) -> bool
+    def self.well_formed?: (::String) -> bool
+
+    # : (::String) -> ::String
+    def self.from_utf8: (::String) -> ::String
+
+    # : (::String, ?replacement: ::String) -> ::String
+    def self.to_utf8: (::String, ?replacement: ::String) -> ::String
+
+    # : (Array[Integer]) -> ::String
+    def self.from_utf16: (Array[Integer]) -> ::String
+
+    # : (::String) -> Array[Integer]
+    def self.to_utf16: (::String) -> Array[Integer]
+
+    # : (::String) -> ::String
+    def self.from_utf16le: (::String) -> ::String
+
+    # : (::String) -> ::String
+    def self.from_utf16be: (::String) -> ::String
+
+    # : (::String) -> ::String
+    def self.to_utf16le: (::String) -> ::String
+
+    # : (::String) -> ::String
+    def self.to_utf16be: (::String) -> ::String
+
+    # : (::String, ::String) -> ::String
+    def self.concat: (::String, ::String) -> ::String
+
+    # : (::String) -> bool
+    def self.ends_with_lead_surrogate?: (::String) -> bool
+
+    # : (::String) -> bool
+    def self.starts_with_trail_surrogate?: (::String) -> bool
+  end
+end

data/sig/wtf8/errors.rbs

@@ -0,0 +1,12 @@
+# Generated from lib/wtf8/errors.rb with RBS::Inline
+
+module WTF8
+  class Error < StandardError
+  end
+
+  class InvalidByteSequenceError < Error
+  end
+
+  class InvalidCodePointError < Error
+  end
+end

data/sig/wtf8/modified_utf8.rbs

@@ -0,0 +1,15 @@
+# Generated from lib/wtf8/modified_utf8.rb with RBS::Inline
+
+module WTF8
+  module ModifiedUTF8
+    NUL: ::String
+
+    ENCODED_NUL: ::String
+
+    # : (::String) -> ::String
+    def self.encode: (::String) -> ::String
+
+    # : (::String, ?strict: bool) -> ::String
+    def self.decode: (::String, ?strict: bool) -> ::String
+  end
+end

data/sig/wtf8/string.rbs

@@ -0,0 +1,107 @@
+# Generated from lib/wtf8/string.rb with RBS::Inline
+
+module WTF8
+  class String
+    include Comparable
+
+    # : (::String) -> WTF8::String
+    def self.from_utf8: (::String) -> WTF8::String
+
+    # : (Array[Integer]) -> WTF8::String
+    def self.from_utf16: (Array[Integer]) -> WTF8::String
+
+    # : (::String) -> WTF8::String
+    def self.from_utf16le: (::String) -> WTF8::String
+
+    # : (::String) -> WTF8::String
+    def self.from_utf16be: (::String) -> WTF8::String
+
+    # : (Array[Integer]) -> WTF8::String
+    def self.from_code_points: (Array[Integer]) -> WTF8::String
+
+    # : (::String, ?strict: bool) -> WTF8::String
+    def self.from_cesu8: (::String, ?strict: bool) -> WTF8::String
+
+    # : (::String, ?strict: bool) -> WTF8::String
+    def self.from_modified_utf8: (::String, ?strict: bool) -> WTF8::String
+
+    attr_reader raw: ::String
+
+    attr_reader code_points: Array[Integer]
+
+    # : (::String) -> void
+    def initialize: (::String) -> void
+
+    # : () -> Integer
+    def length: () -> Integer
+
+    alias size length
+
+    # : () -> Integer
+    def bytesize: () -> Integer
+
+    # : () -> bool
+    def empty?: () -> bool
+
+    # : () -> bool
+    def well_formed?: () -> bool
+
+    # : () { (Integer) -> void } -> WTF8::String
+    # : () -> Enumerator[Integer, WTF8::String]
+    def each_code_point: () { (Integer) -> void } -> WTF8::String
+                       | () -> Enumerator[Integer, WTF8::String]
+
+    # : (WTF8::String | ::String) -> WTF8::String
+    def +: (WTF8::String | ::String) -> WTF8::String
+
+    # : (Integer) -> WTF8::String?
+    # : (Range[Integer]) -> WTF8::String?
+    def []: (Integer) -> WTF8::String?
+          | (Range[Integer]) -> WTF8::String?
+
+    alias slice []
+
+    # : (Range[Integer]) -> WTF8::String
+    def slice_utf16: (Range[Integer]) -> WTF8::String
+
+    # : (WTF8::String | ::String) -> bool
+    def start_with?: (WTF8::String | ::String) -> bool
+
+    # : (WTF8::String | ::String) -> bool
+    def end_with?: (WTF8::String | ::String) -> bool
+
+    # : (?replacement: ::String) -> ::String
+    def to_utf8: (?replacement: ::String) -> ::String
+
+    alias to_s to_utf8
+
+    # : () -> Array[Integer]
+    def to_utf16: () -> Array[Integer]
+
+    # : () -> ::String
+    def to_utf16le: () -> ::String
+
+    # : () -> ::String
+    def to_utf16be: () -> ::String
+
+    # : () -> ::String
+    def to_cesu8: () -> ::String
+
+    # : () -> ::String
+    def to_modified_utf8: () -> ::String
+
+    # : (untyped) -> bool
+    def ==: (untyped) -> bool
+
+    alias eql? ==
+
+    # : () -> Integer
+    def hash: () -> Integer
+
+    # : (untyped) -> Integer?
+    def <=>: (untyped) -> Integer?
+
+    # : () -> ::String
+    def inspect: () -> ::String
+  end
+end

data/sig/wtf8/surrogates.rbs

@@ -0,0 +1,32 @@
+# Generated from lib/wtf8/surrogates.rb with RBS::Inline
+
+module WTF8
+  module Surrogates
+    RANGE: untyped
+
+    LEAD_RANGE: untyped
+
+    TRAIL_RANGE: untyped
+
+    PREFIX_BYTE: ::Integer
+
+    LEAD_SECOND_BYTE: untyped
+
+    TRAIL_SECOND_BYTE: untyped
+
+    # : (Integer) -> bool
+    def self.surrogate?: (Integer) -> bool
+
+    # : (Integer) -> bool
+    def self.lead?: (Integer) -> bool
+
+    # : (Integer) -> bool
+    def self.trail?: (Integer) -> bool
+
+    # : (Integer, Integer) -> Integer
+    def self.combine: (Integer, Integer) -> Integer
+
+    # : (Integer) -> Array[Integer]
+    def self.split: (Integer) -> Array[Integer]
+  end
+end

data/sig/wtf8/version.rbs

@@ -0,0 +1,5 @@
+# Generated from lib/wtf8/version.rb with RBS::Inline
+
+module WTF8
+  VERSION: ::String
+end

data/sig/wtf8.rbs

@@ -1,4 +1,30 @@
-module Wtf8
-  VERSION: String
-  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+# Generated from lib/wtf8.rb with RBS::Inline
+
+module WTF8
+  # : (::String) -> WTF8::String
+  def self.from_utf8: (::String) -> WTF8::String
+
+  # : (Array[Integer]) -> WTF8::String
+  def self.from_utf16: (Array[Integer]) -> WTF8::String
+
+  # : (::String) -> WTF8::String
+  def self.from_utf16le: (::String) -> WTF8::String
+
+  # : (::String) -> WTF8::String
+  def self.from_utf16be: (::String) -> WTF8::String
+
+  # : (Array[Integer]) -> WTF8::String
+  def self.from_code_points: (Array[Integer]) -> WTF8::String
+
+  # : (::String, ?strict: bool) -> WTF8::String
+  def self.from_cesu8: (::String, ?strict: bool) -> WTF8::String
+
+  # : (::String, ?strict: bool) -> WTF8::String
+  def self.from_modified_utf8: (::String, ?strict: bool) -> WTF8::String
+
+  # : (::String) -> WTF8::String
+  def self.from_wtf8: (::String) -> WTF8::String
+
+  # : (::String) -> bool
+  def self.valid?: (::String) -> bool
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: wtf8
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.1.0
 platform: ruby
 authors:
 - Marco Roth
@@ -18,10 +18,24 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- LICENSE.txt
 - README.md
 - lib/wtf8.rb
+- lib/wtf8/cesu8.rb
+- lib/wtf8/codec.rb
+- lib/wtf8/errors.rb
+- lib/wtf8/modified_utf8.rb
+- lib/wtf8/string.rb
+- lib/wtf8/surrogates.rb
 - lib/wtf8/version.rb
 - sig/wtf8.rbs
+- sig/wtf8/cesu8.rbs
+- sig/wtf8/codec.rbs
+- sig/wtf8/errors.rbs
+- sig/wtf8/modified_utf8.rbs
+- sig/wtf8/string.rbs
+- sig/wtf8/surrogates.rbs
+- sig/wtf8/version.rbs
 - wtf8.gemspec
 homepage: https://github.com/marcoroth/wtf8-ruby
 licenses:
@@ -45,7 +59,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.16
+rubygems_version: 4.0.6
 specification_version: 4
 summary: Ruby Implementation of the WTF-8 encoding.
 test_files: []