multi_compress 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bc428423e6ad060dd4c421822a586e3a54681019d6e858c88076a5b44868448a
-  data.tar.gz: 0d1a9737005c28e2c5580708a94fb2473d5b6f4d7d6fa09281ece76499958f9e
+  metadata.gz: ebc73c83a9de27fcde1080879572c025a30a3f2bb391313d5c21e56fd558cde3
+  data.tar.gz: 20a2db05bc0ecafbb399a01275391e5d1c08feafe8fc51f683859463b9c14ff8
 SHA512:
-  metadata.gz: bf176d5d014613ae6a0f8bfc3d76869c0ef9a8a72ac19ac6dac39b1ddd11d10f941e350bc5b2f1cd7b7210771ef2fc54e04035b73347d4bc6e67186520a26a0e
-  data.tar.gz: ef34ce6bb40e89d16e112e13e95ede1e0bbfe37179556d88e7c4717735d1612c11b0ee216ffc70982417863898a76d2737d1a78c7cffb3bce6a7e916c4746e68
+  metadata.gz: aeac010c1419ad5b10919ba1d3e18b2053950555117dffc1075be2ce7b1e6ae9b2a9ff14d2684556a9aa0904ad030f688f4d954cb710692b263e8bb4ca238365
+  data.tar.gz: 9493318cf1321bbe525a8f65992675ef7b0760bf853192cf04146b8280f32011c925a5576876787849737d26eea7cc807c739ff0d5182c652b939c0150c3bf5b

data/CHANGELOG.md

@@ -1,5 +1,40 @@
 # Changelog
 
+## [0.6.0]
+
+### Added
+- **MCDB2** — opt-in dictionary-backed SQL-readable envelopes for a homogeneous
+  `bytea`/`LONGBLOB` column. MCDB1 remains immutable and dictionary-free.
+- **`MultiCompress::Database::Dictionary`** — strict zstd dictionary wrapper with
+  a signed-bigint-compatible application registry id, non-zero zstd DictID and
+  SHA-256 fingerprint. `MultiCompress::Database.compress(..., dictionary:)`
+  emits the 27-byte MCDB2 envelope.
+- **Dictionary registry generator** — `multi_compress db registry postgres|mysql`
+  emits append-only registry/head DDL, metadata validation and freeze triggers.
+- **Dictionary readable views** — `multi_compress db view ... --dictionary-table
+  ... --dictionary-id-column ...` emits an INNER JOIN view that supplies the
+  registry dictionary to native decoder functions. MySQL output requests
+  `ALGORITHM=MERGE SQL SECURITY DEFINER`.
+- **Native MCDB2 readers** for PostgreSQL and MySQL 5.7: dictionary reference,
+  dictionary SHA-256/DictID helpers, validity check and dictionary decode functions.
+  They use bounded DDict caches keyed by dictionary id + SHA-256 and verify bytes
+  before cache reuse.
+- **`multi_compress_db_original_size(blob)`** on PostgreSQL and MySQL for optional
+  size-aware administrative filtering.
+- PostgreSQL 17 and MySQL 5.7 E2E coverage for dictionary registry validation,
+  generated INNER JOIN views, wrong-dictionary rejection and plan shape checks.
+
+### Changed
+- PostgreSQL deployment `make enable` upgrades an existing extension before
+  refreshing grants, so reader deployment precedes MCDB2 writes.
+- MySQL UDF upgrade now preserves registrations for all MCDB1 and MCDB2 functions.
+
+### Notes
+- A dictionary is application data, not part of a DBA reader archive. It must stay
+  immutable and be backed up/replicated with every payload that references it.
+- After the first MCDB2 write, retain a 0.6 reader on every reader host until all
+  MCDB2 rows have been retired; a 0.5 reader cannot decode version 2 envelopes.
+
 ## [0.5.0]
 
 - MCDB1 now rejects trailing bytes, skippable frames, and concatenated zstd frames.

data/GET_STARTED.md

@@ -1051,3 +1051,22 @@ to the `Type` to enable `changed_in_place?` (which decompresses the old value
 on each check).
 
 This guide covers comprehensive usage of the MultiCompress gem. For advanced use cases or questions, see the source code or create an issue on GitHub.
+
+
+## MCDB2 dictionary-backed SQL columns
+
+Use `MultiCompress::Database` when PostgreSQL/MySQL/DBeaver must read a compressed
+column. MCDB1 is dictionary-free. MCDB2 is for many small, similarly shaped JSON or
+text rows and requires one immutable dictionary registry version per row:
+
+```ruby
+dictionary = MultiCompress::Database::Dictionary.train(samples, id: 42, size: 32 * 1024)
+blob = MultiCompress::Database.compress(payload_json, dictionary: dictionary)
+```
+
+Install/upgrade the native database reader before writing MCDB2 data, generate the
+append-only registry DDL with `multi_compress db registry`, then generate an INNER
+JOIN readable view with `multi_compress db view --dictionary-table ...`. The complete
+protocol and rollout contract are in [`docs/database-envelope-v2.md`](docs/database-envelope-v2.md).
+
+For MCDB2 registry DDL, pass `--payload-table`, `--payload-column`, and `--payload-dictionary-id-column` to `multi_compress db registry`; this creates the payload FK and enforces header dictionary-reference consistency.

data/README.md

@@ -187,7 +187,7 @@ single database column (optional ActiveRecord `Type`/`Coder` adapters and
 optional Base64 for text columns). It is **not** the format for SQL-side
 inspection.
 
-For MySQL 5.7 or PostgreSQL + DBeaver, use the separate, frozen `MCDB1` API instead:
+For MySQL 5.7 or PostgreSQL + DBeaver, use the SQL-readable database API. `MCDB1` is dictionary-free; `MCDB2` is opt-in for a homogeneous column with an immutable zstd dictionary registry:
 
 ```ruby
 require "multi_compress/database"
@@ -196,6 +196,17 @@ blob = MultiCompress::Database.compress("JSON or UTF-8 text") # store in LONGBLO
 text = MultiCompress::Database.decompress(blob)
 ```
 
+For repeated small JSON/text records with the same shape, use MCDB2 and keep the
+dictionary as append-only application data:
+
+```ruby
+dictionary = MultiCompress::Database::Dictionary.train(samples, id: 42)
+blob = MultiCompress::Database.compress(payload_json, dictionary: dictionary)
+```
+
+See [`docs/database-envelope-v2.md`](docs/database-envelope-v2.md) for the
+registry, generated view, rollout and DBeaver flow.
+
 For database-side reading, the installed gem creates a **self-contained DBA
 bundle** from the exact gem version — no Git clone, manual `.so` copy, or
 hand-written extension SQL:
@@ -236,3 +247,5 @@ app-server → DBA → DBeaver flow.
 ## License
 
 MIT — see [LICENSE.txt](LICENSE.txt).
+
+For MCDB2 registry DDL, pass `--payload-table`, `--payload-column`, and `--payload-dictionary-id-column` to `multi_compress db registry`; this creates the payload FK and enforces header dictionary-reference consistency.

data/db_core/include/mcdb_format.h

@@ -4,21 +4,42 @@
 #include <stddef.h>
 #include <stdint.h>
 
+#include <zstd.h>
+
 #define MCDB_MAGIC0 0x4D /* 'M' */
 #define MCDB_MAGIC1 0x43 /* 'C' */
 #define MCDB_MAGIC2 0x44 /* 'D' */
 #define MCDB_MAGIC3 0x42 /* 'B' */
 
-#define MCDB_VERSION      1
-#define MCDB_CODEC_ZSTD   1
-#define MCDB_FLAGS_V1     0
-#define MCDB_HEADER_SIZE  19
-#define MCDB_CRC_OFFSET   15
-#define MCDB_MAX_OUTPUT   (16u * 1024u * 1024u) /* 16 MiB decompressed */
-#define MCDB_MAX_ENVELOPE 16842771u
+#define MCDB_VERSION_V1 1
+#define MCDB_VERSION_V2 2
+#define MCDB_CODEC_ZSTD 1
+#define MCDB_FLAGS_NONE 0
+
+#define MCDB_V1_HEADER_SIZE 19u
+#define MCDB_V2_HEADER_SIZE 27u
+#define MCDB_HEADER_SIZE MCDB_V1_HEADER_SIZE /* MCDB1 compatibility alias */
+#define MCDB_ORIGINAL_SIZE_OFFSET 7u
+#define MCDB_CRC_OFFSET 15u
+#define MCDB_DICTIONARY_REF_OFFSET 19u
+
+#define MCDB_MAX_OUTPUT (16u * 1024u * 1024u) /* 16 MiB decompressed */
+#define MCDB_MAX_DICTIONARY_BYTES (256u * 1024u)
+#define MCDB_MAX_DICTIONARY_REF UINT64_C(0x7FFFFFFFFFFFFFFF)
+#define MCDB_MAX_ENVELOPE_V1 16842771u
+#define MCDB_MAX_ENVELOPE_V2 16842779u
+#define MCDB_MAX_ENVELOPE MCDB_MAX_ENVELOPE_V2
 
+#define MCDB_SHA256_BYTES 32u
 #define MCDB_ERRLEN 256
-#define MCDB_READER_VERSION "0.5.0"
+#define MCDB_READER_VERSION "0.6.0"
+
+typedef struct {
+    uint8_t version;
+    uint64_t original_size;
+    uint64_t dictionary_ref;
+    size_t header_size;
+} mcdb_header;
 
 typedef enum {
     MCDB_OK = 0,
@@ -29,6 +50,10 @@ typedef enum {
     MCDB_ERR_CODEC,
     MCDB_ERR_FLAGS,
     MCDB_ERR_SIZE_LIMIT,
+    MCDB_ERR_DICTIONARY_REQUIRED,
+    MCDB_ERR_DICTIONARY_REF,
+    MCDB_ERR_DICTIONARY_SIZE,
+    MCDB_ERR_DICTIONARY_ID,
     MCDB_ERR_DECOMPRESS,
     MCDB_ERR_TRAILING_DATA,
     MCDB_ERR_FRAME_CONTENT_SIZE,
@@ -38,10 +63,19 @@ typedef enum {
     MCDB_ERR_ALLOC
 } mcdb_status;
 
+/* Parses MCDB1 or MCDB2. It never decompresses. */
+mcdb_status mcdb_parse_header(const unsigned char *in, size_t in_len, mcdb_header *out_header);
+
+/* MCDB1 compatibility API. Rejects MCDB2. */
 mcdb_status mcdb_validate_header(const unsigned char *in, size_t in_len,
                                  uint64_t *out_original_size);
 
-/* The caller must validate the header and size out from it before calling. */
+/* MCDB2 parser. Rejects MCDB1. */
+mcdb_status mcdb_validate_v2_header(const unsigned char *in, size_t in_len,
+                                    uint64_t *out_original_size,
+                                    uint64_t *out_dictionary_ref);
+
+/* The caller must validate the MCDB1 header before calling. */
 mcdb_status mcdb_decode_into(const unsigned char *in, size_t in_len,
                              unsigned char *out, size_t out_capacity,
                              size_t *out_len, char *errbuf);
@@ -50,6 +84,30 @@ mcdb_status mcdb_decode(const unsigned char *in, size_t in_len,
                         unsigned char **out, size_t *out_len,
                         char *errbuf);
 
+/* MCDB2 decode. supplied_dictionary_ref must be the registry id passed by SQL. */
+mcdb_status mcdb_decode_v2_into(const unsigned char *in, size_t in_len,
+                                uint64_t supplied_dictionary_ref,
+                                const unsigned char *dictionary,
+                                size_t dictionary_len,
+                                ZSTD_DCtx *dctx,
+                                ZSTD_DDict *ddict,
+                                unsigned char *out, size_t out_capacity,
+                                size_t *out_len, char *errbuf);
+
+mcdb_status mcdb_decode_v2(const unsigned char *in, size_t in_len,
+                           uint64_t supplied_dictionary_ref,
+                           const unsigned char *dictionary,
+                           size_t dictionary_len,
+                           unsigned char **out, size_t *out_len,
+                           char *errbuf);
+
+mcdb_status mcdb_validate_dictionary(const unsigned char *dictionary, size_t dictionary_len,
+                                     uint32_t *out_zstd_dictionary_id);
+
+uint32_t mcdb_zstd_frame_dictionary_id(const unsigned char *frame, size_t frame_len);
+uint64_t mcdb_dictionary_ref(const unsigned char *in, size_t in_len, mcdb_status *out_status);
+void mcdb_sha256(const unsigned char *data, size_t len, unsigned char out[MCDB_SHA256_BYTES]);
+
 const char *mcdb_status_str(mcdb_status s);
 
 #endif