multi_compress 0.3.5 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9f30991dd0d788507bb43885eacc3ae6c131722b0c77428778e7e3176d6ef220
-  data.tar.gz: 53ccbd3e9bf75b8b5eb74a8a6d523baab825ec3492ac730dbe2f739732b724e2
+  metadata.gz: bc428423e6ad060dd4c421822a586e3a54681019d6e858c88076a5b44868448a
+  data.tar.gz: 0d1a9737005c28e2c5580708a94fb2473d5b6f4d7d6fa09281ece76499958f9e
 SHA512:
-  metadata.gz: ad1f3ad3d2ba84e5eec72c1c0b5fed2dade4b3f1dd96efc11b6454ec9f9af78e4829e070c7310283d6500ed71a4a9895334a9d91bd6a0c86f4e947a98c5b0706
-  data.tar.gz: 11369cc33cae0ecb46cf0e9c84e0bc35c72bd2f7cb1cec46425c5cbbc27dfcc9c2037d5e5ab75a4e23c721229577b7ede3aea7bfe7327644024d1be7af229c57
+  metadata.gz: bf176d5d014613ae6a0f8bfc3d76869c0ef9a8a72ac19ac6dac39b1ddd11d10f941e350bc5b2f1cd7b7210771ef2fc54e04035b73347d4bc6e67186520a26a0e
+  data.tar.gz: ef34ce6bb40e89d16e112e13e95ede1e0bbfe37179556d88e7c4717735d1612c11b0ee216ffc70982417863898a76d2737d1a78c7cffb3bce6a7e916c4746e68

data/CHANGELOG.md

@@ -1,5 +1,81 @@
 # Changelog
 
+## [0.5.0]
+
+- MCDB1 now rejects trailing bytes, skippable frames, and concatenated zstd frames.
+
+### Fixed
+- Pin the MySQL 5.7 Docker acceptance target to the official linux/amd64 image digest.
+
+
+### Added
+- **Casual database deployment bundles** — `multi_compress db package postgres`
+  and `multi_compress db package mysql` build a version-locked tarball directly
+  from the installed gem. A DBA extracts it on the database host and uses
+  `make verify`, `make doctor`, `sudo make install`, and `make enable` instead
+  of cloning the repository or manually locating/copying a `.so`.
+- **Readable-view generator** — `multi_compress db view postgres|mysql` emits
+  safely quoted, migration-ready views for DBeaver. MySQL output always converts
+  UDF output through `utf8mb4`.
+- **Database envelope v1 (`MCDB1`)** — a frozen, self-describing storage format
+  (`docs/database-envelope-v1.md`): 19-byte header (magic/version/codec/flags/
+  original_size/crc32) + a zstd frame. zstd-only, UTF-8 text without NUL bytes, 16 MiB cap, no
+  dictionaries/base64. Corruption raises in Ruby and PostgreSQL; MySQL returns
+  `NULL` and exposes `multi_compress_db_is_valid(blob) = 0` for diagnostics.
+- **`MultiCompress::Database`** — narrow Ruby writer/reader for `MCDB1`
+  (`compress`/`decompress`/`valid?`), independent of ActiveRecord. Validates
+  UTF-8 and size on write; requires exactly one zstd frame and verifies size +
+  CRC-32 on read.
+- **MySQL 5.7 UDF** (`mysql_udf/`) — a separate native target (not built by the
+  gem) implementing the same `MCDB1` contract in C over the vendored zstd, with
+  `multi_compress_db_version()`, `multi_compress_db_is_valid(blob)`, and
+  `multi_compress_db_decompress(blob)`, plus SQL install/uninstall/views,
+  a Makefile, a real-MySQL Docker acceptance script, and a Ruby↔C parity
+  harness.
+- **PostgreSQL extension** (`postgres_extension/`) — a PGXS target over the
+  same shared C decoder, exposing the matching `version`, `is_valid(bytea)`, and
+  `decompress(bytea)` SQL functions through `CREATE EXTENSION multi_compress`.
+  It includes SQL/view templates and a Docker e2e gate.
+- Golden fixtures (`test/fixtures/database_v1/`) shared by the Ruby tests and the
+  C parity harness, proving both implementations agree byte-for-byte.
+
+### Notes
+- The 16 MiB cap and CRC-32 check are enforced identically in Ruby, the MySQL
+  UDF, and the PostgreSQL extension. The CRC-32 matches `MultiCompress.crc32`
+  (zlib/IEEE). MCDB1 payload text must be valid UTF-8 without NUL bytes so it
+  maps safely to PostgreSQL `text`.
+- The ActiveRecord adapter remains experimental pending real Rails integration
+  tests; the `MCDB1` path does not depend on it.
+
+## [0.4.0]
+
+### Added
+- **`multi_compress` command-line tool.** Gzip-style file (de)compression with
+  `-a/--algo`, `-l/--level`, `-d/--decompress`, `-o`, `-c/--stdout`, `-k/--keep`,
+  `-f/--force`, `--max-output`. Streams through `Reader`/`Writer`, writes output
+  atomically (temp + fsync + rename, directory fsync before removing the source),
+  refuses `input == output` and `-o` with multiple inputs.
+- **`MultiCompress::Codec`** — strict, self-describing compression envelope for
+  values stored in a database. Always compresses (one storage format), 7-byte
+  magic/version/algo header, strict decode (a corrupt envelope raises
+  `DataError`, never silent passthrough), opt-in `legacy:` handling
+  (`:reject` default, `:plain`, `{ compressed: :algo }`), `encode: :base64` for
+  text columns (explicit `mc1:` prefix), locally-frozen `max_output_size`,
+  configurable `encoding:` (validated on write), and optional `serializer:`
+  (JSON/Marshal) for non-String values.
+- **ActiveRecord integration** via `MultiCompress::ActiveRecordSupport::Type`
+  (for `attribute`) and `::Coder` (for `serialize`). Opt-in `mutable:` enables
+  in-place dirty tracking.
+
+### Notes
+- The CLI writes this gem's internal LZ4 block format with the **`.mclz4`**
+  extension (not `.lz4`), since it is not interchangeable with the standard
+  `lz4` CLI frame.
+- `MultiCompress::Codec` treats one column as one fixed encoding. Use
+  `encoding: Encoding::BINARY` (a `:binary`/`bytea` column) for arbitrary bytes.
+- The AR `Type` default does not detect in-place mutation; reassign the
+  attribute, or pass `mutable: true`.
+
 ## [0.3.5]
 
 ### Changed

data/GET_STARTED.md

@@ -862,4 +862,192 @@ puts results
 # => {:lz4=>{:time=>0.0012, :ratio=>0.234, :size=>1024}, ...}
 ```
 
+## Command-Line Tool
+
+Installing the gem provides a `multi_compress` executable (gzip-style):
+
+```bash
+multi_compress file.json                 # -> file.json.zst (zstd default)
+multi_compress -a brotli -l best a.css   # -> a.css.br
+multi_compress -d file.json.zst          # -> file.json (algo from extension)
+cat big.log | multi_compress -a zstd -c > big.log.zst   # pipe mode
+```
+
+Flags: `-a/--algo` (`zstd|lz4|brotli`), `-l/--level` (number or `fastest|default|best`),
+`-d/--decompress`, `-o/--output`, `-c/--stdout`, `-k/--keep`, `-f/--force`,
+`-q/--quiet`, `--max-output SIZE` (e.g. `64MB`), `--version`.
+
+Safety: output is written to a temp file in the destination directory, fsynced,
+atomically renamed, and the directory is fsynced before the source is removed —
+an interrupted run never leaves a partial file or touches the source. The tool
+refuses `input == output` and refuses `-o` with multiple inputs.
+
+**LZ4 note:** this gem's LZ4 uses an internal block format that is *not*
+interchangeable with the standard `lz4` CLI. To avoid a false compatibility
+promise, LZ4 output uses the **`.mclz4`** extension; `.lz4` is never produced.
+
+## Database Column Compression
+
+There are two intentionally separate database paths:
+
+1. **`MultiCompress::Codec`** is a general application-side envelope for one
+   column. It is useful when only Ruby needs to read the data.
+2. **`MultiCompress::Database` (`MCDB1`)** is the narrow, frozen format for
+   MySQL 5.7 or PostgreSQL + DBeaver. It has matching server-side C readers,
+   so SQL can decode the BLOB / `bytea`.
+
+### MySQL 5.7 or PostgreSQL / DBeaver: `MultiCompress::Database` (`MCDB1`)
+
+Use this path when employees must inspect decoded values in DBeaver, DataGrip,
+or the MySQL console:
+
+```ruby
+require "multi_compress/database"
+
+blob = MultiCompress::Database.compress("Привет") # UTF-8 bytes for LONGBLOB / bytea
+text = MultiCompress::Database.decompress(blob)    # => "Привет"
+```
+
+MCDB1 is deliberately narrow: zstd only, valid non-NUL UTF-8 text, exactly one
+zstd frame, no dictionaries/Base64, and a 16 MiB decompressed limit.
+
+#### 1. Application developer: create the DBA bundle
+
+The native reader is intentionally **not** built during `gem install`: it must
+be compiled on the database host, against that host's PostgreSQL/MySQL ABI.
+The developer does not hand a DBA a Git revision or a loose `.so`; they create
+a version-locked source bundle from the installed gem:
+
+```bash
+bundle exec multi_compress db package postgres --output tmp/multi-compress-postgres.tar.gz
+# or
+bundle exec multi_compress db package mysql --output tmp/multi-compress-mysql.tar.gz
+```
+
+The archive contains the shared MCDB1 decoder, the right server-side source,
+vendored zstd, checksum manifest, and a small `Makefile`/installer. It is the
+only artifact the DBA needs.
+
+#### 2. DBA: install and enable on the database server
+
+PostgreSQL, on the PostgreSQL host:
+
+```bash
+ tar -xzf multi-compress-postgres.tar.gz
+ cd multi_compress-postgres-0.5.0
+ make verify
+ make doctor
+ sudo make install
+ sudo -u postgres make enable DB=app_production \
+  MIGRATION_ROLE=app_migrations READ_ROLE=dbeaver_readonly
+```
+
+`install` builds `multi_compress_pg.so` on that host, discovers paths through
+`pg_config`, and installs the library/control/extension SQL. Install the matching
+`postgresql-server-dev-<major>` package and a C toolchain first; `make doctor`
+checks those prerequisites before changing the server. `enable` runs
+`CREATE EXTENSION` in exactly one database. Repeat `make enable DB=...` for
+reporting databases or any separate database that must query MCDB1 columns.
+
+MySQL 5.7, on the MySQL host:
+
+```bash
+ tar -xzf multi-compress-mysql.tar.gz
+ cd multi_compress-mysql-0.5.0
+ make verify
+ make doctor MYSQL_DEFAULTS_FILE=/etc/mysql/admin.cnf
+ sudo make install MYSQL_DEFAULTS_FILE=/etc/mysql/admin.cnf
+ make enable MYSQL_DEFAULTS_FILE=/etc/mysql/admin.cnf
+```
+
+The MySQL installer verifies the local server is MySQL 5.7, asks that same
+server for `@@plugin_dir`, builds the UDF locally, and installs the library
+there. `enable` registers the functions once for that MySQL server.
+
+#### 3. Application developer: commit the readable view
+
+Do not ask DBAs or DBeaver users to type decoder calls. Generate a safe,
+quoted view definition and commit it as a Rails migration or versioned SQL:
+
+```bash
+bundle exec multi_compress db view postgres \
+  --table app.events \
+  --column payload_compressed \
+  --view admin.events_readable \
+  --columns id,created_at,status \
+  --as payload \
+  --output db/views/events_readable.sql
+```
+
+For MySQL use `db view mysql` with the same arguments. The MySQL definition
+includes `CONVERT(... USING utf8mb4)`, so Cyrillic, Kazakh text and emoji are
+returned correctly to DBeaver. If PostgreSQL was enabled in a non-default schema,
+pass `--extension-schema that_schema` to the PostgreSQL view generator.
+
+DBeaver users now make ordinary queries against the view:
+
+```sql
+SELECT id, created_at, status, payload
+FROM admin.events_readable
+WHERE id = 123;
+```
+
+The view decompresses selected rows. Filter by indexed, uncompressed fields
+before reading it; do not perform unbounded `LIKE '%text%'` searches on the
+decoded payload.
+
+The shared, long-term format contract is
+[`docs/database-envelope-v1.md`](docs/database-envelope-v1.md).
+
+### General Ruby-side `Codec`
+
+`MultiCompress::Codec` transparently compresses on write and "unwraps" on read,
+with a strict, self-describing envelope suitable for persistent storage.
+
+```ruby
+require "multi_compress/codec"
+
+codec = MultiCompress::Codec.new(algo: :zstd, level: 6)
+blob  = codec.dump(big_json)   # always-compressed envelope for a :binary column
+json  = codec.load(blob)       # exact original string back (UTF-8 by default)
+```
+
+Guarantees: always one compressed format (no mixed raw/compressed rows); a
+recognized-but-corrupt envelope raises `MultiCompress::DataError` (never silent
+data loss); invalid encoding is rejected at **write** time; the decompression
+cap is frozen locally so global config can't loosen it.
+
+Options: `encode: :base64` (for text/varchar columns; prefer `:binary`/`bytea`),
+`serializer:` (JSON or Marshal for non-String values), `encoding:`
+(default UTF-8; use `Encoding::BINARY` for arbitrary bytes), `max_output_size:`,
+`dictionary:`, and `legacy:` for reading pre-existing values:
+
+```ruby
+MultiCompress::Codec.new(legacy: :reject)                 # default: no envelope -> raise
+MultiCompress::Codec.new(legacy: :plain)                  # treat as uncompressed string
+MultiCompress::Codec.new(legacy: { compressed: :zstd })  # decompress old raw blobs
+```
+
+### ActiveRecord (experimental)
+
+Requires ActiveModel. The adapter has unit coverage but must still be integration-tested
+against each Rails/database combination before being treated as a stable deployment contract. Register the type (`attribute`) or use a coder (`serialize`):
+
+```ruby
+require "multi_compress/active_record"
+
+class Event < ApplicationRecord
+  # attribute API (prefer a :binary / bytea column)
+  attribute :payload, MultiCompress::ActiveRecordSupport::Type.new(algo: :zstd, level: 6)
+
+  # or the serialize macro with JSON objects
+  # serialize :payload, coder: MultiCompress::ActiveRecordSupport::Coder.new(algo: :zstd, serializer: JSON)
+end
+```
+
+Dirty tracking: by default in-place mutation isn't detected — reassign the
+attribute (`event.payload = event.payload.merge(...)`) or pass `mutable: true`
+to the `Type` to enable `changed_in_place?` (which decompresses the old value
+on each check).
+
 This guide covers comprehensive usage of the MultiCompress gem. For advanced use cases or questions, see the source code or create an issue on GitHub.

data/README.md

@@ -172,6 +172,62 @@ Or use the build script:
 ./build.sh
 ```
 
+## Command-line tool & DB column compression
+
+Installing the gem provides a gzip-style `multi_compress` executable:
+
+```bash
+multi_compress file.json          # -> file.json.zst
+multi_compress -d file.json.zst   # -> file.json
+cat big.log | multi_compress -a zstd -c > big.log.zst
+```
+
+`MultiCompress::Codec` is a general-purpose application-side envelope for a
+single database column (optional ActiveRecord `Type`/`Coder` adapters and
+optional Base64 for text columns). It is **not** the format for SQL-side
+inspection.
+
+For MySQL 5.7 or PostgreSQL + DBeaver, use the separate, frozen `MCDB1` API instead:
+
+```ruby
+require "multi_compress/database"
+
+blob = MultiCompress::Database.compress("JSON or UTF-8 text") # store in LONGBLOB / bytea
+text = MultiCompress::Database.decompress(blob)
+```
+
+For database-side reading, the installed gem creates a **self-contained DBA
+bundle** from the exact gem version — no Git clone, manual `.so` copy, or
+hand-written extension SQL:
+
+```bash
+bundle exec multi_compress db package postgres --output tmp/multi-compress-postgres.tar.gz
+# or
+bundle exec multi_compress db package mysql --output tmp/multi-compress-mysql.tar.gz
+```
+
+The DBA extracts that archive on the database host and runs `make verify`,
+`make doctor`, and `sudo make install`. PostgreSQL enablement also requires the
+migration and read roles; MySQL uses a local Unix socket and has explicit
+`upgrade`, `disable`, and `uninstall` confirmations. The bundle compiles the
+native reader against that host's real PostgreSQL/MySQL environment and installs
+it into the server-reported directory. Generate the DBeaver-facing read view
+from the application repository as well:
+
+```bash
+bundle exec multi_compress db view postgres \
+  --table app.events --column payload_compressed \
+  --view admin.events_readable --columns id,created_at,status \
+  --output db/views/events_readable.sql
+```
+
+Use `--extension-schema NAME` with the PostgreSQL view generator only when the
+DBA enabled the extension outside the default `multi_compress` schema. See
+[GET_STARTED.md](GET_STARTED.md#database-column-compression) for the full
+app-server → DBA → DBeaver flow.
+
+> The CLI writes this gem's internal LZ4 format as `.mclz4` (not `.lz4`), since it is not interchangeable with the standard `lz4` CLI.
+
 ## Requirements
 
 - Ruby >= 2.7.1

data/db_core/include/mcdb_format.h

@@ -0,0 +1,55 @@
+#ifndef MCDB_FORMAT_H
+#define MCDB_FORMAT_H
+
+#include <stddef.h>
+#include <stdint.h>
+
+#define MCDB_MAGIC0 0x4D /* 'M' */
+#define MCDB_MAGIC1 0x43 /* 'C' */
+#define MCDB_MAGIC2 0x44 /* 'D' */
+#define MCDB_MAGIC3 0x42 /* 'B' */
+
+#define MCDB_VERSION      1
+#define MCDB_CODEC_ZSTD   1
+#define MCDB_FLAGS_V1     0
+#define MCDB_HEADER_SIZE  19
+#define MCDB_CRC_OFFSET   15
+#define MCDB_MAX_OUTPUT   (16u * 1024u * 1024u) /* 16 MiB decompressed */
+#define MCDB_MAX_ENVELOPE 16842771u
+
+#define MCDB_ERRLEN 256
+#define MCDB_READER_VERSION "0.5.0"
+
+typedef enum {
+    MCDB_OK = 0,
+    MCDB_ERR_TRUNCATED,
+    MCDB_ERR_ENVELOPE_SIZE,
+    MCDB_ERR_MAGIC,
+    MCDB_ERR_VERSION,
+    MCDB_ERR_CODEC,
+    MCDB_ERR_FLAGS,
+    MCDB_ERR_SIZE_LIMIT,
+    MCDB_ERR_DECOMPRESS,
+    MCDB_ERR_TRAILING_DATA,
+    MCDB_ERR_FRAME_CONTENT_SIZE,
+    MCDB_ERR_SIZE_MISMATCH,
+    MCDB_ERR_CRC,
+    MCDB_ERR_UTF8,
+    MCDB_ERR_ALLOC
+} mcdb_status;
+
+mcdb_status mcdb_validate_header(const unsigned char *in, size_t in_len,
+                                 uint64_t *out_original_size);
+
+/* The caller must validate the header and size out from it before calling. */
+mcdb_status mcdb_decode_into(const unsigned char *in, size_t in_len,
+                             unsigned char *out, size_t out_capacity,
+                             size_t *out_len, char *errbuf);
+
+mcdb_status mcdb_decode(const unsigned char *in, size_t in_len,
+                        unsigned char **out, size_t *out_len,
+                        char *errbuf);
+
+const char *mcdb_status_str(mcdb_status s);
+
+#endif