lz_string 0.3.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 751414b97387726a93f8a6a5194f3a775a0ef2954ff1e608816e27a7075de593
-  data.tar.gz: 23b5921756ca7e73b104e8696a379f6e04e14f5ced11806a34b52d2d33f65cf3
+  metadata.gz: 2fbe7383cf742da0cfee289795f4f51e5b686db961520ef3d99fca2212182281
+  data.tar.gz: 0be89c68f03f6253f24571b08c153df6ac7767b70f4190f78688d3a2110abe00
 SHA512:
-  metadata.gz: 82882d474bc980f53126648b1e0005d31a1f5cf478c29d9e3184a13eec52f920be449f472ec2f8ad5c7b584e0c774de3d657648ccd8b00d94f7e429426c7c619
-  data.tar.gz: c90ebbcb83bbde8df69860a180a122d39608c4eb1cf318d68c81877d41df99192ec6d751426c52550d1d8dde894e346fd8ee635c31e3c656f38b09c5ec64d84d
+  metadata.gz: 155134bf111c17d8f874c09450b68d250e4232edb619a89c6fd6b0d14c26d4e73904a5f5f25ad996df170a863d70f8dadc625c08027685560e109cce5aa8e658
+  data.tar.gz: 5071d19377af3302458676e5f74b488b0b63c1fb612bdc5fd374210e30c73054002800a1f211a5cd5a7e811838010783ee46164d9d8d6b86c3559b72a13fd8c2

data/CHANGELOG.md

@@ -0,0 +1,55 @@
+# Changelog
+
+## 1.0.0 (2026-10-16)
+
+Now it gives the same results as JavaScript lz-string for every input, emoji too
+([#3](https://github.com/altivi/lz_string/issues/3)), and it passes the test suite of lz-string itself.
+
+### Fixed
+
+- Emoji and all other characters. Like JS, compress and decompress now work on UTF-16 code units.
+  Before, emoji gave wrong results, `nil` or a `RangeError`.
+- `decompress` always returns UTF-8. Before, it could return a binary string (`"\xF6"` for `"ö"`).
+- Text with `__proto__` in it compresses like in JS.
+- `compress("")` gives what JS gives (`"Q==="` in Base64), not `""`.
+- `LZString.decompress("")` returns `nil`, like JS.
+- `UriSafe.decompress` no longer changes the string you give it, and works with frozen strings.
+- Faster: on 100 KB of text, Base64 compress takes 0.03 s instead of 0.10 s, and decompress 0.008 s instead of 0.12 s.
+
+### Added
+
+- `LZString::Uint8Array`: compress to bytes and back (`compressToUint8Array` in JS).
+- `LZString::Custom`: compress with your own alphabet (`compressToCustom`, new in lz-string 2).
+- The JS function names: `LZString.compress_to_base64`, `LZString.decompress_from_base64` and all the others.
+- `decompress_to_utf16le` for every format: the exact JS result, which can hold a lone surrogate.
+- The `lz_string` command line tool, with the same options as the one of lz-string 2.
+
+### Changed (this can break your code)
+
+- `LZString.compress` (raw) returns a UTF-16LE string. Like in JS, its result often has lone surrogates,
+  and UTF-8 can't hold them. Before, it returned UTF-8 with `"�"` in it, which could not be decompressed.
+  The Base64, URI-safe and UTF-16 results are UTF-8, as before.
+- Data that starts with code 3 (lz-string never writes that) raises `ArgumentError`, like lz-string 2.
+- A binary (ASCII-8BIT) string with bytes above 127 raises `Encoding::UndefinedConversionError`.
+  Before, its bytes were read as Latin-1. Use `force_encoding` to say what the bytes are.
+- It needs Ruby 2.7 or newer.
+
+## 0.3.0 (2025-04-14)
+
+- Better Unicode handling. Emoji still did not work (fixed in 1.0.0).
+
+## 0.2.0 (2023-06-13)
+
+- URI-safe compression: `LZString::UriSafe` ([#2](https://github.com/altivi/lz_string/pull/2)).
+
+## 0.1.2 (2020-01-27)
+
+- Updated dependencies.
+
+## 0.1.1 (2019-12-05)
+
+- UTF-16 and Base64 compression.
+
+## 0.1.0 (2019-12-05)
+
+- First release: raw compression.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Alexander Tishchenko
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -1,67 +1,145 @@
-# lz-string
-Ruby implementation of [LZ-String](https://github.com/pieroxy/lz-string) compression algorithm.
+# lz_string
 
-Supports:
- * Raw compression
- * UTF-16 compression
- * Base64 compression
+[![Gem Version](https://img.shields.io/gem/v/lz_string?logo=rubygems&logoColor=white)](https://rubygems.org/gems/lz_string)
+[![Downloads](https://img.shields.io/gem/dt/lz_string?logo=rubygems&logoColor=white&label=downloads)](https://rubygems.org/gems/lz_string)
+[![Downloads of the latest version](https://img.shields.io/gem/dtv/lz_string?logo=rubygems&logoColor=white&label=downloads%40latest)](https://rubygems.org/gems/lz_string)
+[![Tests](https://github.com/altivi/lz_string/actions/workflows/test.yml/badge.svg)](https://github.com/altivi/lz_string/actions/workflows/test.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
-### Installation
+Ruby port of [lz-string](https://github.com/pieroxy/lz-string), the LZ-based compression for JavaScript.
 
-Install the latest release:
+It gives **the same results as the JavaScript library, for every input**: any text, emoji too, and even broken data.
+So you can compress in the browser and decompress in Ruby, or the other way round.
 
-```
+- **Every format of lz-string:** raw, Base64, URI-safe, UTF-16, Uint8Array and your own alphabet.
+- **Tested against JavaScript:** it passes [the test suite of lz-string itself](https://github.com/pieroxy/lz-string/tree/master/test)
+  (70 of 70 checks), and its specs check it against results of the JavaScript library.
+- **The same command line tool** as lz-string 2, with the same options.
+- No dependencies. Ruby 2.7 or newer.
+
+## Install
+
+```console
 $ gem install lz_string
 ```
 
-In Rails, add it to your Gemfile:
+Or add it to your Gemfile:
 
 ```ruby
-gem 'lz_string'
+gem "lz_string"
 ```
 
-### How to use
+## Use
 
-#### Normal Compression and Decompression:
+```ruby
+require "lz_string"
 
-``` ruby
-  # Compress
-  compressed = LZString.compress("Hello world!")
-  => "҅〶惶@✰Ӏ葀"
-  
-  # Decompress
-  LZString.decompress(compressed)
-  => "Hello world!"
+compressed = LZString.compress_to_base64("Hello, 🎶 world!")
+# => "BIUwNmD2A0AEg8G4Nv3YHdICcwBMCEQ="
+LZString.decompress_from_base64(compressed)
+# => "Hello, 🎶 world!"
 ```
 
-#### UTF-16 Compression and Decompression:
-
-``` ruby
-  some_string = '{"some": "json", "foo": [{"bar": "؋", "key": "؄"}], "ঞᕠ": "൱ඵቜ"}'
-  
-  # Compress
-  compressed = LZString::UTF16.compress(some_string)
-  => "ᯡࡓ䈌\u0B80匰ᜠр\u0AF2Ǹ䀺㈦イ\u0530්C¦¼䒨ᨬිnj〩痐࠸С㸢璑Ч䲤U⋴ҕ䈥㛢ĉ႙  "
-  
-  # Decompress
-  LZString::UTF16.decompress(compressed)
-  => "{\"some\": \"json\", \"foo\": [{\"bar\": \"؋\", \"key\": \"؄\"}], \"ঞᕠ\": \"൱ඵቜ\"}"
+The methods have the same names as in JavaScript:
+
+| JavaScript                                                            | Ruby                                                                                  | Gives                               |
+|-----------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------|
+| `compressToBase64` / `decompressFromBase64`                           | `LZString.compress_to_base64` / `LZString.decompress_from_base64`                     | Base64                              |
+| `compressToEncodedURIComponent` / `decompressFromEncodedURIComponent` | `LZString.compress_to_encoded_uri_component` / `.decompress_from_encoded_uri_component` | text that is safe in a URL          |
+| `compressToUTF16` / `decompressFromUTF16`                             | `LZString.compress_to_utf16` / `.decompress_from_utf16`                               | valid UTF-16 text, for localStorage |
+| `compressToUint8Array` / `decompressFromUint8Array`                   | `LZString.compress_to_uint8_array` / `.decompress_from_uint8_array`                   | bytes                               |
+| `compressToCustom` / `decompressFromCustom`                           | `LZString.compress_to_custom(text, alphabet)` / `.decompress_from_custom(data, alphabet)` | text from your own alphabet         |
+| `compress` / `decompress`                                             | `LZString.compress` / `LZString.decompress`                                           | raw: a UTF-16LE string (see below)  |
+
+Each format also has its own class, as in older versions of this gem: `LZString::Base64`, `LZString::UriSafe`,
+`LZString::UTF16`, `LZString::Uint8Array` and `LZString::Custom`. Each has `compress` and `decompress`.
+
+```ruby
+LZString::UriSafe.compress('{"page": 2, "search": "café ☕"}')
+# => "N4IgDghg5gpiBcACATAGkSAzjCAnAxgBYIb4QBmAl4oKhkIAvkA"
+
+LZString.compress_to_uint8_array("🎶")
+# => "\x8F\x06\xE3o\xDA\x00"
+LZString.decompress_from_uint8_array([143, 6, 227, 111, 218, 0])
+# => "🎶"
+
+LZString.compress_to_custom("Hello world!", "0123456789abcdef")
+# => "0485303660f60040ee90273004c08440"
 ```
 
-#### Base64 Compression and Decompression:
+### Strings in and out
+
+- **In:** any Ruby string. It is read as text in its own encoding (UTF-8, Latin-1, …), like a JavaScript string.
+- **Out of `decompress`:** a UTF-8 string, or `nil` when JavaScript gives `null` (broken data).
+- **Raw `compress`** gives a UTF-16LE string. Like in JavaScript, this result is often not valid UTF-16
+  (it has lone surrogates), and UTF-8 can't hold it. To store or send it, use one of the other formats.
+- **Uint8Array** gives a binary string. `decompress_from_uint8_array` takes a binary string or a list of numbers.
+- A JavaScript string can hold half of an emoji (a lone surrogate). UTF-8 can't, so `decompress` writes `"�"` for it.
+  If you need the result exactly as JavaScript has it, use `decompress_to_utf16le` (every format has it).
+
+## Command line
+
+```console
+$ lz_string -h
+Usage: lz_string [options] [input-file]
+
+Use lz-string to compress or decompress a file
 
-``` ruby		
-  # Compress
-  compressed = LZString::Base64.compress("Hello world!")
-  => "BIUwNmD2AEDukCcwBMCEQ==="
-  
-  # Decompress
-  LZString::Base64.decompress(compressed)
-  => "Hello world!"
+Arguments:
+    input-file                       file to process, if no file then read from stdin
+
+Options:
+    -V, --version                    output the version number
+    -d, --decompress                 if unset then this will compress
+    -e, --encoder TYPE               character encoding to use (choices: base64, custom, encodeduri, raw, uint8array, utf16, default: raw)
+    -c, --custom DICTIONARY          dictionary for custom encoder
+    -v, --verify                     verify before returning
+    -o, --output OUTPUT_FILE         output file, otherwise write to stdout
+    -q, --quiet                      don't print any error messages
+    -h, --help                       display help for command
+
+$ lz_string -e base64 photo.jpg -o photo.lz
+$ lz_string -d -e base64 photo.lz -o photo.jpg
 ```
- 
-### Tests
 
-``` bash
+Like the lz-string 2 tool, it works on the bytes of a file (2 bytes are one UTF-16 code unit), not on text.
+So it can compress any file and give it back the same. It writes the same bytes as the JavaScript tool.
+
+### Check it with the tests of lz-string
+
+```console
+$ git clone https://github.com/pieroxy/lz-string && cd lz-string/test
+$ cat > profiles/lz_string.sh <<'EOF'
+optDescription="The lz_string Ruby gem"
+optCommand=lz_string
+optDecompress="-d"
+optEncoder="-e"
+optEncoders="base64,encodeduri,raw,uint8array,utf16"
+optOutput="-o"
+optQuiet="-q"
+EOF
+$ ./test.sh lz_string
+```
+
+## Upgrade from 0.x
+
+Emoji now work, and all results are the same as in JavaScript. Some things changed, see the [changelog](CHANGELOG.md):
+
+- `LZString.compress` (raw) returns a UTF-16LE string, not UTF-8.
+- `LZString.decompress("")` returns `nil`, like in JavaScript.
+- Data that starts with code 3 (lz-string never writes that) raises `ArgumentError`, like lz-string 2.
+- A binary string with bytes above 127 raises an error. Use `force_encoding` to say what the bytes are.
+
+## Development
+
+```console
+$ bundle install
 $ rake
 ```
+
+The specs compare the gem with results of the JavaScript library, kept in `spec/fixtures/js_results.json`.
+To make that file again, see `spec/fixtures/js_results.js`.
+
+## License
+
+MIT

data/exe/lz_string

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+require "lz_string/cli"
+
+exit LZString::CLI.run(ARGV)

data/lib/lz_string/base.rb

@@ -1,393 +1,206 @@
 module LZString
   # Base compression class.
+  #
+  # A port of lz-string (JavaScript) that gives the same results as lz-string 2, and as 1.5.0 for
+  # all real data. JS strings are made of UTF-16 code units, so the work here is done on code units
+  # too: an emoji like "🎶" is two units.
   class Base
+    # JS keeps the dictionary in a plain object, and an object can't store the key "__proto__".
+    # So JS never finds that word in its dictionary, and we must not find it either.
+    PROTO = "__proto__".encode(Encoding::UTF_16LE).b.freeze
+
+    # @param string [String]
+    # @return [Array<Integer>] the UTF-16 code units, like charCodeAt in JS
+    def self.code_units(string)
+      string.encode(Encoding::UTF_16LE).unpack("v*")
+    end
+
+    # UTF-8 can't hold a lone surrogate. Like JS TextEncoder, we write "�" for it.
+    #
+    # @param string [String, nil] UTF-16LE
+    # @return [String, nil] UTF-8
+    def self.to_utf8(string)
+      string && string.encode(Encoding::UTF_8, invalid: :replace)
+    end
+
     # @param uncompressed      [String]
     # @param bits_per_char     [Integer]
-    # @param get_char_from_int [Integer]
+    # @param get_char_from_int [Proc]
     def self.compress(uncompressed, bits_per_char, get_char_from_int)
-      return "" if uncompressed.nil? || uncompressed.empty?
+      return "" if uncompressed.nil?
 
-      i, value, ii = nil
-      context_dictionary = {}
-      context_dictionary_to_create = {}
-      context_c = ""
-      context_wc = ""
-      context_w = ""
+      dictionary = {}
+      dictionary_to_create = {}
+      w = "".b
       # Compensate for the first entry which should not count
-      context_enlarge_in = 2
-      context_dict_size = 3
-      context_num_bits = 2
-      context_data = []
-      context_data_val = 0
-      context_data_position = 0
-
-      uncompressed.each_char do |c|
-        context_c = c
-
-        if (!context_dictionary.has_key?(context_c))
-          context_dictionary[context_c] = context_dict_size
-          context_dictionary_to_create[context_c] = true
-          context_dict_size += 1
-        end
-
-        context_wc = context_w + context_c
-        if (context_dictionary.has_key?(context_wc))
-          context_w = context_wc
-        else
-          if (context_dictionary_to_create.has_key?(context_w))
-            if (context_w[0].ord < 256)
-              for i in 0...context_num_bits do
-                context_data_val = (context_data_val << 1)
-                if (context_data_position == bits_per_char - 1)
-                  context_data_position = 0
-                  context_data.push(get_char_from_int[context_data_val])
-                  context_data_val = 0
-                else
-                  context_data_position += 1
-                end
-              end
-
-              value = context_w[0].ord
-              for i in 0...8 do
-                context_data_val = (context_data_val << 1) | (value & 1)
-                if (context_data_position == bits_per_char - 1)
-                  context_data_position = 0
-                  context_data.push(get_char_from_int[context_data_val])
-                  context_data_val = 0
-                else
-                  context_data_position += 1
-                end
-                value = value >> 1
-              end
-            else
-              value = 1
-              for i in 0...context_num_bits do
-                context_data_val = (context_data_val << 1) | value
-                if (context_data_position == bits_per_char-1)
-                  context_data_position = 0
-                  context_data.push(get_char_from_int[context_data_val])
-                  context_data_val = 0
-                else
-                  context_data_position += 1
-                end
-                value = 0
-              end
-              value = context_w[0].ord
-              for i in 0...16 do
-                context_data_val = (context_data_val << 1) | (value & 1)
-                if (context_data_position == bits_per_char - 1)
-                  context_data_position = 0
-                  context_data.push(get_char_from_int[context_data_val])
-                  context_data_val = 0
-                else
-                  context_data_position += 1
-                end
-                value = value >> 1
-              end
-            end
-            context_enlarge_in -= 1
-            if (context_enlarge_in == 0)
-              context_enlarge_in = 2**context_num_bits
-              context_num_bits += 1
-            end
-            context_dictionary_to_create.delete(context_w)
+      enlarge_in = 2
+      dict_size = 3
+      num_bits = 2
+      data = []
+      data_val = 0
+      data_position = 0
+
+      # Writes the lowest `count` bits of `value`, lowest bit first.
+      write_bits = lambda do |value, count|
+        count.times do
+          data_val = (data_val << 1) | (value & 1)
+          if data_position == bits_per_char - 1
+            data_position = 0
+            data << get_char_from_int.call(data_val)
+            data_val = 0
           else
-            value = context_dictionary[context_w]
-            for i in 0...context_num_bits do
-              context_data_val = (context_data_val << 1) | (value & 1)
-              if (context_data_position == bits_per_char - 1)
-                context_data_position = 0
-                context_data.push(get_char_from_int[context_data_val])
-                context_data_val = 0
-              else
-                context_data_position += 1
-              end
-              value = value >> 1
-            end
-          end
-
-          context_enlarge_in -= 1
-
-          if (context_enlarge_in == 0)
-            context_enlarge_in = 2**context_num_bits
-            context_num_bits += 1
+            data_position += 1
           end
-
-          # Add wc to the dictionary.
-          context_dictionary[context_wc] = context_dict_size
-          context_dict_size += 1
-          context_w = context_c
+          value >>= 1
         end
       end
 
-      # Output the code for w.
-      if (context_w != "")
-        if (context_dictionary_to_create.has_key?(context_w))
-          if (context_w[0].ord < 256)
-            for i in 0...context_num_bits do
-              context_data_val = (context_data_val << 1)
-              if (context_data_position == bits_per_char-1)
-                context_data_position = 0
-                context_data.push(get_char_from_int[context_data_val])
-                context_data_val = 0
-              else
-                context_data_position += 1
-              end
-            end
-            value = context_w[0].ord
-            for i in 0...8 do
-              context_data_val = (context_data_val << 1) | (value & 1)
-              if (context_data_position == bits_per_char-1)
-                context_data_position = 0
-                context_data.push(get_char_from_int[context_data_val])
-                context_data_val = 0
-              else
-                context_data_position += 1
-              end
-              value = value >> 1
-            end
+      # Writes the code for w: a new character, or its place in the dictionary.
+      write_w = lambda do
+        if dictionary_to_create.key?(w)
+          code_unit = w.unpack1("v")
+          if code_unit < 256
+            write_bits.call(0, num_bits)
+            write_bits.call(code_unit, 8)
           else
-            value = 1
-            for i in 0...context_num_bits do
-              context_data_val = (context_data_val << 1) | value
-              if (context_data_position == bits_per_char-1)
-                context_data_position = 0
-                context_data.push(get_char_from_int[context_data_val])
-                context_data_val = 0
-              else
-                context_data_position += 1
-              end
-              value = 0
-            end
-            value = context_w[0].ord
-            for i in 0...16 do
-              context_data_val = (context_data_val << 1) | (value & 1)
-              if (context_data_position == bits_per_char-1)
-                context_data_position = 0
-                context_data.push(get_char_from_int[context_data_val])
-                context_data_val = 0
-              else
-                context_data_position += 1
-              end
-              value = value >> 1
-            end
+            write_bits.call(1, num_bits)
+            write_bits.call(code_unit, 16)
           end
-          context_enlarge_in -= 1
-          if (context_enlarge_in == 0)
-            context_enlarge_in = 2**context_num_bits
-            context_num_bits += 1
+          enlarge_in -= 1
+          if enlarge_in == 0
+            enlarge_in = 2**num_bits
+            num_bits += 1
           end
-          context_dictionary_to_create.delete(context_w)
+          dictionary_to_create.delete(w)
         else
-          value = context_dictionary[context_w]
-          for i in 0...context_num_bits
-            context_data_val = (context_data_val << 1) | (value & 1)
-            if (context_data_position == bits_per_char-1)
-              context_data_position = 0
-              context_data.push(get_char_from_int[context_data_val])
-              context_data_val = 0
-            else
-              context_data_position += 1
-            end
-            value = value >> 1
-          end
+          write_bits.call(dictionary[w], num_bits)
         end
-
-        context_enlarge_in -= 1
-        if (context_enlarge_in == 0)
-          context_enlarge_in = 2**context_num_bits
-          context_num_bits += 1
+        enlarge_in -= 1
+        if enlarge_in == 0
+          enlarge_in = 2**num_bits
+          num_bits += 1
         end
       end
 
-      # Mark the end of the stream
-      value = 2
-      for i in 0...context_num_bits do
-        context_data_val = (context_data_val << 1) | (value & 1)
-        if (context_data_position == bits_per_char - 1)
-          context_data_position = 0
-          context_data.push(get_char_from_int[context_data_val])
-          context_data_val = 0
+      # Each character is one UTF-16 code unit: 2 bytes, little endian.
+      input = uncompressed.encode(Encoding::UTF_16LE).b
+      0.step(input.bytesize - 1, 2) do |i|
+        c = input.byteslice(i, 2)
+        unless dictionary.key?(c)
+          dictionary[c] = dict_size
+          dict_size += 1
+          dictionary_to_create[c] = true
+        end
+
+        wc = w + c
+        if dictionary.key?(wc)
+          w = wc
         else
-          context_data_position += 1
+          write_w.call
+          # Add wc to the dictionary.
+          dictionary[wc] = dict_size unless wc == PROTO
+          dict_size += 1
+          w = c
         end
-        value = value >> 1
       end
 
+      # Output the code for w.
+      write_w.call unless w.empty?
+
+      # Mark the end of the stream
+      write_bits.call(2, num_bits)
+
       # Flush the last char
-      while (true)
-        context_data_val = (context_data_val << 1)
-        if (context_data_position == bits_per_char-1)
-          context_data.push(get_char_from_int[context_data_val])
+      loop do
+        data_val <<= 1
+        if data_position == bits_per_char - 1
+          data << get_char_from_int.call(data_val)
           break
-        else
-          context_data_position += 1
         end
+        data_position += 1
       end
 
-      return context_data.join("")
+      data.join
     end
 
     # @param length         [Integer]
     # @param reset_value    [Integer]
     # @param get_next_value [Proc]
-    def self.decompress(length, reset_value, get_next_value, encoding = "UTF-8")
+    # @return [String, nil] UTF-16LE, exactly as JS has it (it can hold lone surrogates)
+    def self.decompress(length, reset_value, get_next_value)
       dictionary = [0, 1, 2]
       enlarge_in = 4
       dict_size = 4
       num_bits = 3
-      entry = ""
       result = []
-      data = {
-        val: get_next_value[(0)],
-        position: reset_value,
-        index: 1
-      }
-      bits = 0
-      maxpower = 2**2
-      power = 1
-      i, w, resb, c = nil
+      data_val = get_next_value.call(0)
+      data_position = reset_value
+      data_index = 1
 
-      while (power != maxpower)
-        resb = data[:val] & data[:position]
-        data[:position] >>= 1
-        if (data[:position] == 0)
-          data[:position] = reset_value
-          data[:val] = get_next_value[data[:index]]
-          data[:index] += 1
-        end
-        bits |= (resb > 0 ? 1 : 0) * power
-        power <<= 1
-      end
-
-      case(n = bits)
-      when 0
-        bits = 0
-        maxpower = 2**8
-        power = 1
-        while power != maxpower
-          resb = data[:val] & data[:position]
-          data[:position] >>= 1
-          if (data[:position] == 0)
-            data[:position] = reset_value
-            data[:val] = get_next_value[data[:index]]
-            data[:index] += 1
-          end
-          bits |= (resb > 0 ? 1 : 0) * power
-          power <<= 1
-        end
-        c = bits.chr(encoding)
-      when 1
+      # Reads `count` bits, lowest bit first.
+      read_bits = lambda do |count|
         bits = 0
-        maxpower = 2**16
-        power = 1
-        while (power != maxpower)
-          resb = data[:val] & data[:position]
-          data[:position] >>= 1
-          if (data[:position] == 0)
-              data[:position] = reset_value
-              data[:val] = get_next_value[data[:index]]
-              data[:index] += 1
+        count.times do |i|
+          resb = data_val & data_position
+          data_position >>= 1
+          if data_position == 0
+            data_position = reset_value
+            data_val = get_next_value.call(data_index)
+            data_index += 1
           end
-          bits |= (resb > 0 ? 1 : 0) * power
-          power <<= 1
+          bits |= 1 << i if resb > 0
         end
-        c = bits.chr(encoding)
-      when 2
-        ""
+        bits
       end
 
+      # Each character is one UTF-16 code unit: 2 bytes, little endian.
+      c = case read_bits.call(2)
+          when 0 then [read_bits.call(8)].pack("v")
+          when 1 then [read_bits.call(16)].pack("v")
+          when 2 then return "".encode(Encoding::UTF_16LE)
+          # Real data never starts with 3. lz-string 2 throws here, so we raise too
+          # (1.5.0 went on with an undefined value and gave junk, null or a TypeError).
+          else raise ArgumentError, "No character found"
+          end
       dictionary[3] = c
       w = c
       result << c
 
-      while(true)
-        return "" if (data[:index] > length)
-
-        bits = 0
-        maxpower = 2**num_bits
-        power = 1
-        while (power != maxpower)
-          resb = data[:val] & data[:position]
-          data[:position] >>= 1
-          if (data[:position] == 0)
-            data[:position] = reset_value
-            data[:val] = get_next_value[data[:index]]
-            data[:index] += 1
-          end
-          bits |= (resb > 0 ? 1 : 0) * power
-          power <<= 1
-        end
-
-        case(c = bits)
-        when 0
-          bits = 0
-          maxpower = 2**8
-          power = 1
-          while (power != maxpower)
-            resb = data[:val] & data[:position]
-            data[:position] >>= 1
-            if (data[:position] == 0)
-              data[:position] = reset_value
-              data[:val] = get_next_value[data[:index]]
-              data[:index] += 1
-            end
-            bits |= (resb > 0 ? 1 : 0) * power
-            power <<= 1
-          end
+      loop do
+        return "".encode(Encoding::UTF_16LE) if data_index > length
 
-          dictionary[dict_size] = bits.chr(encoding)
+        case (c = read_bits.call(num_bits))
+        when 0, 1
+          dictionary[dict_size] = [read_bits.call(c == 0 ? 8 : 16)].pack("v")
+          c = dict_size
           dict_size += 1
-          c = dict_size - 1
-          enlarge_in -= 1
-        when 1
-          bits = 0
-          maxpower = 2**16
-          power = 1
-          while (power != maxpower)
-            resb = data[:val] & data[:position]
-            data[:position] >>= 1
-            if (data[:position] == 0)
-              data[:position] = reset_value
-              data[:val] = get_next_value[data[:index]]
-              data[:index] += 1
-            end
-            bits |= (resb > 0 ? 1 : 0) * power
-            power <<= 1
-          end
-          dictionary[dict_size] = bits.chr(encoding)
-          dict_size += 1
-          c = dict_size - 1
           enlarge_in -= 1
         when 2
-          return result.join("")
+          return result.join.force_encoding(Encoding::UTF_16LE)
         end
 
-        if (enlarge_in == 0)
+        if enlarge_in == 0
           enlarge_in = 2**num_bits
           num_bits += 1
         end
 
-        if (dictionary[c])
+        if dictionary[c]
           entry = dictionary[c]
+        elsif c == dict_size
+          entry = w + w.byteslice(0, 2)
         else
-          if (c === dict_size)
-              entry = w + w[0]
-          else
-            return nil
-          end
+          return nil
         end
-
         result << entry
 
         # Add w+entry[0] to the dictionary.
-        dictionary[dict_size] = w + entry[0]
+        dictionary[dict_size] = w + entry.byteslice(0, 2)
         dict_size += 1
         enlarge_in -= 1
 
         w = entry
 
-        if (enlarge_in == 0)
+        if enlarge_in == 0
           enlarge_in = 2**num_bits
           num_bits += 1
         end

data/lib/lz_string/base64.rb

@@ -3,6 +3,8 @@ module LZString
   class Base64
     # Base64 alphabet.
     KEY_STR_BASE64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/="
+    # Code unit of each alphabet character => its value.
+    VALUES = KEY_STR_BASE64.each_char.with_index.map { |char, i| [char.ord, i] }.to_h
 
     # @param input [String]
     def self.compress(input)
@@ -20,32 +22,20 @@ module LZString
     end
 
     # @param compressed [String]
+    # @return [String, nil] UTF-8
     def self.decompress(compressed)
-      return "" if compressed.nil?
-      return nil if compressed == ""
-
-      LZString::Base.decompress(
-        compressed.length,
-        32,
-        lambda { |index| get_base_value(KEY_STR_BASE64, compressed[index]) }
-      )
+      LZString::Base.to_utf8(decompress_to_utf16le(compressed))
     end
 
-    # @param alphabet  [String]
-    # @param character [String]
-    def self.get_base_value(alphabet, character)
-      base_reverse_dic = {}
-
-      if (!base_reverse_dic[alphabet])
-        base_reverse_dic[alphabet] = {}
-        for i in 0...alphabet.length do
-          base_reverse_dic[alphabet][alphabet[i]] = i
-        end
-      end
+    # @param compressed [String]
+    # @return [String, nil] UTF-16LE, see LZString.decompress_to_utf16le
+    def self.decompress_to_utf16le(compressed)
+      return "" if compressed.nil?
+      return nil if compressed.empty?
 
-      base_reverse_dic[alphabet][character]
+      # Like in JS, a character outside the alphabet counts as 0.
+      values = LZString::Base.code_units(compressed).map { |unit| VALUES.fetch(unit, 0) }
+      LZString::Base.decompress(values.length, 32, lambda { |index| values[index] || 0 })
     end
-
-    private_class_method :get_base_value
   end
 end

data/lib/lz_string/cli.rb

@@ -0,0 +1,107 @@
+require "optparse"
+require "lz_string"
+
+module LZString
+  # The command line tool. It has the same options as the one of lz-string 2 (JS), so
+  # lz-string's own test suite (test/test.sh in its repository) can check this gem too.
+  #
+  # Like the JS tool it is binary safe: a file is read and written as UTF-16 code units,
+  # 2 bytes each, big endian.
+  module CLI
+    FORMATS = {
+      "base64" => LZString::Base64,
+      "custom" => LZString::Custom,
+      "encodeduri" => LZString::UriSafe,
+      "raw" => LZString,
+      # The JS tool writes the same bytes for raw and uint8array.
+      "uint8array" => LZString,
+      "utf16" => LZString::UTF16,
+    }.freeze
+
+    # @param argv [Array<String>]
+    # @return [Integer] the exit code
+    def self.run(argv, stdin: $stdin, stdout: $stdout, stderr: $stderr)
+      options = { encoder: "raw" }
+      parser = OptionParser.new do |opts|
+        opts.banner = "Usage: lz_string [options] [input-file]\n\nUse lz-string to compress or decompress a file"
+        opts.separator ""
+        opts.separator "Arguments:"
+        opts.separator "    input-file                       file to process, if no file then read from stdin"
+        opts.separator ""
+        opts.separator "Options:"
+        opts.on("-V", "--version", "output the version number") { options[:version] = true }
+        opts.on("-d", "--decompress", "if unset then this will compress") { options[:decompress] = true }
+        opts.on("-e", "--encoder TYPE", FORMATS.keys,
+                "character encoding to use (choices: #{FORMATS.keys.join(", ")}, default: raw)") { |type| options[:encoder] = type }
+        opts.on("-c", "--custom DICTIONARY", "dictionary for custom encoder") { |dictionary| options[:dictionary] = dictionary }
+        opts.on("-v", "--verify", "verify before returning") { options[:verify] = true }
+        opts.on("-o", "--output OUTPUT_FILE", "output file, otherwise write to stdout") { |file| options[:output] = file }
+        opts.on("-q", "--quiet", "don't print any error messages") { options[:quiet] = true }
+        opts.on("-h", "--help", "display help for command") { options[:help] = true }
+      end
+
+      begin
+        file, = parser.parse(argv)
+      rescue OptionParser::ParseError => e
+        stderr.puts("error: #{e.message}", parser.help)
+        return 1
+      end
+      return stdout.puts(parser.help) || 0 if options[:help]
+      return stdout.puts(LZString::VERSION) || 0 if options[:version]
+
+      error = lambda do |message|
+        stderr.puts(message) unless options[:quiet]
+        1
+      end
+      return error.call("Unable to find #{file}") if file && !File.exist?(file)
+      return error.call("Unable to access #{file}") if file && !File.readable?(file)
+
+      format = FORMATS.fetch(options[:encoder])
+      dictionary = []
+      if format == LZString::Custom
+        return error.call("The custom encoder needs a dictionary (-c)") unless options[:dictionary]
+
+        dictionary = [options[:dictionary]]
+      end
+
+      input = to_string(file ? File.binread(file) : stdin.binmode.read)
+      begin
+        result = if options[:decompress]
+                   format.decompress_to_utf16le(input, *dictionary)
+                 else
+                   format.compress(input, *dictionary)
+                 end
+        return error.call("Unable to process #{file || "stdin"}") if result.nil?
+
+        if options[:verify]
+          back = options[:decompress] ? format.compress(result, *dictionary) : format.decompress_to_utf16le(result, *dictionary)
+          return error.call("Unable to validate #{file || "stdin"}") unless back && Base.code_units(back) == Base.code_units(input)
+        end
+      rescue ArgumentError, EncodingError
+        return error.call("Unable to process #{file || "stdin"}")
+      end
+
+      if options[:output]
+        File.binwrite(options[:output], to_bytes(result))
+      else
+        stdout.binmode.write(to_bytes(result))
+      end
+      0
+    end
+
+    # Bytes to a string of UTF-16 code units, 2 bytes each, big endian. A last odd byte is the
+    # high byte of a code unit.
+    def self.to_string(bytes)
+      units = bytes.unpack("C*").each_slice(2).map { |high, low| (high << 8) | low.to_i }
+      units.pack("v*").force_encoding(Encoding::UTF_16LE)
+    end
+
+    # The other way. Like the JS tool, a last 0 byte is left out, so a file with an odd count
+    # of bytes comes back the same.
+    def self.to_bytes(string)
+      units = Base.code_units(string)
+      bytes = units.pack("n*")
+      !units.empty? && units.last % 256 == 0 ? bytes.byteslice(0, bytes.bytesize - 1) : bytes
+    end
+  end
+end

data/lib/lz_string/custom.rb

@@ -0,0 +1,60 @@
+module LZString
+  # Custom compressing algorithm: writes each code unit of the raw result as a number,
+  # with a fixed count of digits from your own alphabet (from lz-string 2).
+  class Custom
+    # @param input      [String]
+    # @param dictionary [String] the alphabet: at least 2 characters
+    # @return [String] UTF-8
+    def self.compress(input, dictionary)
+      return "" if input.nil?
+
+      alphabet = LZString::Base.code_units(dictionary)
+      # JS never ends with a 1-character alphabet.
+      raise ArgumentError, "The dictionary needs at least 2 characters" if alphabet.size < 2
+
+      digits = digits_per_code_unit(alphabet.size)
+      units = LZString.compress(input).unpack("v*").flat_map do |unit|
+        (digits - 1).downto(0).map { |j| alphabet[unit / alphabet.size**j % alphabet.size] }
+      end
+      units.pack("v*").force_encoding(Encoding::UTF_16LE).encode(Encoding::UTF_8)
+    end
+
+    # @param compressed [String]
+    # @param dictionary [String]
+    # @return [String, nil] UTF-8
+    def self.decompress(compressed, dictionary)
+      LZString::Base.to_utf8(decompress_to_utf16le(compressed, dictionary))
+    end
+
+    # @param compressed [String]
+    # @param dictionary [String]
+    # @return [String, nil] UTF-16LE, see LZString.decompress_to_utf16le
+    def self.decompress_to_utf16le(compressed, dictionary)
+      return "" if compressed.nil?
+      return nil if compressed.empty?
+
+      alphabet = LZString::Base.code_units(dictionary)
+      return nil if alphabet.size < 2
+
+      digits = digits_per_code_unit(alphabet.size)
+      units = LZString::Base.code_units(compressed)
+      return nil if units.size % digits != 0
+
+      # The first place of each character, like indexOf in JS.
+      places = {}
+      alphabet.each_with_index { |unit, place| places[unit] ||= place }
+      values = units.each_slice(digits).map do |number|
+        # Like JS: a character outside the alphabet counts as -1, and String.fromCharCode keeps 16 bits.
+        number.reduce(0) { |value, unit| value * alphabet.size + places.fetch(unit, -1) } & 0xFFFF
+      end
+      LZString.decompress_to_utf16le(values.pack("v*").force_encoding(Encoding::UTF_16LE))
+    end
+
+    # Digits needed for 65536 values. Same float math as JS (checked for every size up to 70000).
+    def self.digits_per_code_unit(base)
+      (Math.log(65536) / Math.log(base)).ceil
+    end
+
+    private_class_method :digits_per_code_unit
+  end
+end

data/lib/lz_string/uint8_array.rb

@@ -0,0 +1,27 @@
+module LZString
+  # Uint8Array compressing algorithm: the raw result as bytes, 2 per code unit, big endian.
+  class Uint8Array
+    # @param input [String]
+    # @return [String] binary (ASCII-8BIT)
+    def self.compress(input)
+      LZString.compress(input).unpack("v*").pack("n*")
+    end
+
+    # @param compressed [String, Array<Integer>] the bytes, as a binary string or a list of numbers
+    # @return [String, nil] UTF-8
+    def self.decompress(compressed)
+      LZString::Base.to_utf8(decompress_to_utf16le(compressed))
+    end
+
+    # @param compressed [String, Array<Integer>]
+    # @return [String, nil] UTF-16LE, see LZString.decompress_to_utf16le
+    def self.decompress_to_utf16le(compressed)
+      return "" if compressed.nil?
+
+      bytes = compressed.is_a?(String) ? compressed.bytes : compressed
+      # Like lz-string 2, a last odd byte is the high byte of the last code unit.
+      units = bytes.each_slice(2).map { |high, low| (high << 8) | low.to_i }
+      LZString.decompress_to_utf16le(units.pack("v*").force_encoding(Encoding::UTF_16LE))
+    end
+  end
+end

data/lib/lz_string/urisafe.rb

@@ -2,6 +2,8 @@ module LZString
   class UriSafe
     # Base64 alphabet.
     KEY_STR_URISAFE = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-$"
+    # Code unit of each alphabet character => its value.
+    VALUES = KEY_STR_URISAFE.each_char.with_index.map { |char, i| [char.ord, i] }.to_h
 
     # @param input [String]
     def self.compress(input)
@@ -11,32 +13,21 @@ module LZString
     end
 
     # @param compressed [String]
+    # @return [String, nil] UTF-8
     def self.decompress(compressed)
-      return "" if compressed.nil?
-      return nil if compressed == ""
-      compressed.gsub!(" ","+")
-      LZString::Base.decompress(
-        compressed.length,
-        32,
-        lambda { |index| get_base_value(KEY_STR_URISAFE, compressed[index]) }
-      )
+      LZString::Base.to_utf8(decompress_to_utf16le(compressed))
     end
 
-    # @param alphabet  [String]
-    # @param character [String]
-    def self.get_base_value(alphabet, character)
-      base_reverse_dic = {}
-
-      if (!base_reverse_dic[alphabet])
-        base_reverse_dic[alphabet] = {}
-        for i in 0...alphabet.length do
-          base_reverse_dic[alphabet][alphabet[i]] = i
-        end
-      end
+    # @param compressed [String]
+    # @return [String, nil] UTF-16LE, see LZString.decompress_to_utf16le
+    def self.decompress_to_utf16le(compressed)
+      return "" if compressed.nil?
+      return nil if compressed.empty?
 
-      base_reverse_dic[alphabet][character]
+      # Like in JS, a space is read as "+" (a URL can turn "+" into a space),
+      # and any other character outside the alphabet counts as 0.
+      values = LZString::Base.code_units(compressed).map { |unit| VALUES.fetch(unit == " ".ord ? "+".ord : unit, 0) }
+      LZString::Base.decompress(values.length, 32, lambda { |index| values[index] || 0 })
     end
-
-    private_class_method :get_base_value
   end
 end

data/lib/lz_string/utf16.rb

@@ -9,16 +9,19 @@ module LZString
     end
 
     # @param compressed [String]
+    # @return [String, nil] UTF-8
     def self.decompress(compressed)
+      LZString::Base.to_utf8(decompress_to_utf16le(compressed))
+    end
+
+    # @param compressed [String]
+    # @return [String, nil] UTF-16LE, see LZString.decompress_to_utf16le
+    def self.decompress_to_utf16le(compressed)
       return "" if (compressed == nil)
-      return nil if (compressed == "")
+      return nil if compressed.empty?
 
-      LZString::Base.decompress(
-        compressed.length,
-        16384,
-        lambda { |index| compressed[index].ord - 32 },
-        "UTF-8"
-      )
+      values = LZString::Base.code_units(compressed)
+      LZString::Base.decompress(values.length, 16384, lambda { |index| values[index] ? values[index] - 32 : 0 })
     end
   end
 end

data/lib/lz_string/version.rb

@@ -1,4 +1,4 @@
 module LZString
   # :nodoc:
-  VERSION = "0.3.0"
+  VERSION = "1.0.0"
 end

data/lib/lz_string.rb

@@ -1,23 +1,82 @@
 require "lz_string/base"
 require "lz_string/base64"
-require "lz_string/utf16"
+require "lz_string/custom"
+require "lz_string/uint8_array"
 require "lz_string/urisafe"
+require "lz_string/utf16"
 require "lz_string/version"
 
 # LZ-based compression algorithm.
 module LZString
+  # Returns the same UTF-16 code units as LZString.compress in JS. Like in JS, they are often not
+  # valid UTF-16 (a lone surrogate), and UTF-8 can't hold them, so the result is a UTF-16LE string.
+  #
   # @param input [String]
+  # @return [String] UTF-16LE
   def self.compress(input)
     return "" if input.nil?
 
-    LZString::Base.compress(input, 16, lambda { |a| a.chr("UTF-8") })
+    LZString::Base.compress(input, 16, lambda { |a| [a].pack("v") }).force_encoding(Encoding::UTF_16LE)
   end
 
-  # @param compressed [String]
+  # @param compressed [String] the UTF-16LE string from .compress (other encodings work too)
+  # @return [String, nil] UTF-8
   def self.decompress(compressed)
-    return "" if compressed.nil? || compressed.empty?
-    return nil if compressed == ""
+    LZString::Base.to_utf8(decompress_to_utf16le(compressed))
+  end
+
+  # Like .decompress, but gives the result exactly as JS has it: a UTF-16LE string, which can
+  # also hold a lone surrogate (half of an emoji). Every format has this method.
+  #
+  # @param compressed [String]
+  # @return [String, nil] UTF-16LE
+  def self.decompress_to_utf16le(compressed)
+    return "" if compressed.nil?
+    return nil if compressed.empty?
+
+    values = LZString::Base.code_units(compressed)
+    LZString::Base.decompress(values.length, 32768, lambda { |index| values[index] || 0 })
+  end
+
+  # The other functions of JS lz-string, by their JS names.
+
+  def self.compress_to_base64(input)
+    LZString::Base64.compress(input)
+  end
+
+  def self.decompress_from_base64(compressed)
+    LZString::Base64.decompress(compressed)
+  end
+
+  def self.compress_to_encoded_uri_component(input)
+    LZString::UriSafe.compress(input)
+  end
+
+  def self.decompress_from_encoded_uri_component(compressed)
+    LZString::UriSafe.decompress(compressed)
+  end
+
+  def self.compress_to_utf16(input)
+    LZString::UTF16.compress(input)
+  end
+
+  def self.decompress_from_utf16(compressed)
+    LZString::UTF16.decompress(compressed)
+  end
+
+  def self.compress_to_uint8_array(input)
+    LZString::Uint8Array.compress(input)
+  end
+
+  def self.decompress_from_uint8_array(compressed)
+    LZString::Uint8Array.decompress(compressed)
+  end
+
+  def self.compress_to_custom(input, dictionary)
+    LZString::Custom.compress(input, dictionary)
+  end
 
-    LZString::Base.decompress(compressed.length, 32768, lambda { |index| compressed[index].ord })
+  def self.decompress_from_custom(compressed, dictionary)
+    LZString::Custom.decompress(compressed, dictionary)
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: lz_string
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Altivi
 autorequire:
-bindir: bin
+bindir: exe
 cert_chain: []
-date: 2025-04-14 00:00:00.000000000 Z
+date: 2026-10-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -122,24 +122,36 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: Ruby implementation of LZ-String compression algorithm
+description: 'A full Ruby port of lz-string (JavaScript), with the same results for
+  every input, emoji too: raw, Base64, URI-safe, UTF-16, Uint8Array and custom alphabets,
+  and the same command line tool. It passes the test suite of lz-string itself.'
 email:
 - altivi.prog@gmail.com
-executables: []
+executables:
+- lz_string
 extensions: []
 extra_rdoc_files: []
 files:
+- CHANGELOG.md
+- LICENSE
 - README.md
+- exe/lz_string
 - lib/lz_string.rb
 - lib/lz_string/base.rb
 - lib/lz_string/base64.rb
+- lib/lz_string/cli.rb
+- lib/lz_string/custom.rb
+- lib/lz_string/uint8_array.rb
 - lib/lz_string/urisafe.rb
 - lib/lz_string/utf16.rb
 - lib/lz_string/version.rb
-homepage: https://github.com/Altivi/lz-string
+homepage: https://github.com/altivi/lz_string
 licenses:
 - MIT
-metadata: {}
+metadata:
+  source_code_uri: https://github.com/altivi/lz_string
+  changelog_uri: https://github.com/altivi/lz_string/blob/master/CHANGELOG.md
+  bug_tracker_uri: https://github.com/altivi/lz_string/issues
 post_install_message:
 rdoc_options: []
 require_paths:
@@ -148,7 +160,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '2.7'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="