multi_compress 0.3.5 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9f30991dd0d788507bb43885eacc3ae6c131722b0c77428778e7e3176d6ef220
-  data.tar.gz: 53ccbd3e9bf75b8b5eb74a8a6d523baab825ec3492ac730dbe2f739732b724e2
+  metadata.gz: ebc73c83a9de27fcde1080879572c025a30a3f2bb391313d5c21e56fd558cde3
+  data.tar.gz: 20a2db05bc0ecafbb399a01275391e5d1c08feafe8fc51f683859463b9c14ff8
 SHA512:
-  metadata.gz: ad1f3ad3d2ba84e5eec72c1c0b5fed2dade4b3f1dd96efc11b6454ec9f9af78e4829e070c7310283d6500ed71a4a9895334a9d91bd6a0c86f4e947a98c5b0706
-  data.tar.gz: 11369cc33cae0ecb46cf0e9c84e0bc35c72bd2f7cb1cec46425c5cbbc27dfcc9c2037d5e5ab75a4e23c721229577b7ede3aea7bfe7327644024d1be7af229c57
+  metadata.gz: aeac010c1419ad5b10919ba1d3e18b2053950555117dffc1075be2ce7b1e6ae9b2a9ff14d2684556a9aa0904ad030f688f4d954cb710692b263e8bb4ca238365
+  data.tar.gz: 9493318cf1321bbe525a8f65992675ef7b0760bf853192cf04146b8280f32011c925a5576876787849737d26eea7cc807c739ff0d5182c652b939c0150c3bf5b

data/CHANGELOG.md

@@ -1,5 +1,116 @@
 # Changelog
 
+## [0.6.0]
+
+### Added
+- **MCDB2** — opt-in dictionary-backed SQL-readable envelopes for a homogeneous
+  `bytea`/`LONGBLOB` column. MCDB1 remains immutable and dictionary-free.
+- **`MultiCompress::Database::Dictionary`** — strict zstd dictionary wrapper with
+  a signed-bigint-compatible application registry id, non-zero zstd DictID and
+  SHA-256 fingerprint. `MultiCompress::Database.compress(..., dictionary:)`
+  emits the 27-byte MCDB2 envelope.
+- **Dictionary registry generator** — `multi_compress db registry postgres|mysql`
+  emits append-only registry/head DDL, metadata validation and freeze triggers.
+- **Dictionary readable views** — `multi_compress db view ... --dictionary-table
+  ... --dictionary-id-column ...` emits an INNER JOIN view that supplies the
+  registry dictionary to native decoder functions. MySQL output requests
+  `ALGORITHM=MERGE SQL SECURITY DEFINER`.
+- **Native MCDB2 readers** for PostgreSQL and MySQL 5.7: dictionary reference,
+  dictionary SHA-256/DictID helpers, validity check and dictionary decode functions.
+  They use bounded DDict caches keyed by dictionary id + SHA-256 and verify bytes
+  before cache reuse.
+- **`multi_compress_db_original_size(blob)`** on PostgreSQL and MySQL for optional
+  size-aware administrative filtering.
+- PostgreSQL 17 and MySQL 5.7 E2E coverage for dictionary registry validation,
+  generated INNER JOIN views, wrong-dictionary rejection and plan shape checks.
+
+### Changed
+- PostgreSQL deployment `make enable` upgrades an existing extension before
+  refreshing grants, so reader deployment precedes MCDB2 writes.
+- MySQL UDF upgrade now preserves registrations for all MCDB1 and MCDB2 functions.
+
+### Notes
+- A dictionary is application data, not part of a DBA reader archive. It must stay
+  immutable and be backed up/replicated with every payload that references it.
+- After the first MCDB2 write, retain a 0.6 reader on every reader host until all
+  MCDB2 rows have been retired; a 0.5 reader cannot decode version 2 envelopes.
+
+## [0.5.0]
+
+- MCDB1 now rejects trailing bytes, skippable frames, and concatenated zstd frames.
+
+### Fixed
+- Pin the MySQL 5.7 Docker acceptance target to the official linux/amd64 image digest.
+
+
+### Added
+- **Casual database deployment bundles** — `multi_compress db package postgres`
+  and `multi_compress db package mysql` build a version-locked tarball directly
+  from the installed gem. A DBA extracts it on the database host and uses
+  `make verify`, `make doctor`, `sudo make install`, and `make enable` instead
+  of cloning the repository or manually locating/copying a `.so`.
+- **Readable-view generator** — `multi_compress db view postgres|mysql` emits
+  safely quoted, migration-ready views for DBeaver. MySQL output always converts
+  UDF output through `utf8mb4`.
+- **Database envelope v1 (`MCDB1`)** — a frozen, self-describing storage format
+  (`docs/database-envelope-v1.md`): 19-byte header (magic/version/codec/flags/
+  original_size/crc32) + a zstd frame. zstd-only, UTF-8 text without NUL bytes, 16 MiB cap, no
+  dictionaries/base64. Corruption raises in Ruby and PostgreSQL; MySQL returns
+  `NULL` and exposes `multi_compress_db_is_valid(blob) = 0` for diagnostics.
+- **`MultiCompress::Database`** — narrow Ruby writer/reader for `MCDB1`
+  (`compress`/`decompress`/`valid?`), independent of ActiveRecord. Validates
+  UTF-8 and size on write; requires exactly one zstd frame and verifies size +
+  CRC-32 on read.
+- **MySQL 5.7 UDF** (`mysql_udf/`) — a separate native target (not built by the
+  gem) implementing the same `MCDB1` contract in C over the vendored zstd, with
+  `multi_compress_db_version()`, `multi_compress_db_is_valid(blob)`, and
+  `multi_compress_db_decompress(blob)`, plus SQL install/uninstall/views,
+  a Makefile, a real-MySQL Docker acceptance script, and a Ruby↔C parity
+  harness.
+- **PostgreSQL extension** (`postgres_extension/`) — a PGXS target over the
+  same shared C decoder, exposing the matching `version`, `is_valid(bytea)`, and
+  `decompress(bytea)` SQL functions through `CREATE EXTENSION multi_compress`.
+  It includes SQL/view templates and a Docker e2e gate.
+- Golden fixtures (`test/fixtures/database_v1/`) shared by the Ruby tests and the
+  C parity harness, proving both implementations agree byte-for-byte.
+
+### Notes
+- The 16 MiB cap and CRC-32 check are enforced identically in Ruby, the MySQL
+  UDF, and the PostgreSQL extension. The CRC-32 matches `MultiCompress.crc32`
+  (zlib/IEEE). MCDB1 payload text must be valid UTF-8 without NUL bytes so it
+  maps safely to PostgreSQL `text`.
+- The ActiveRecord adapter remains experimental pending real Rails integration
+  tests; the `MCDB1` path does not depend on it.
+
+## [0.4.0]
+
+### Added
+- **`multi_compress` command-line tool.** Gzip-style file (de)compression with
+  `-a/--algo`, `-l/--level`, `-d/--decompress`, `-o`, `-c/--stdout`, `-k/--keep`,
+  `-f/--force`, `--max-output`. Streams through `Reader`/`Writer`, writes output
+  atomically (temp + fsync + rename, directory fsync before removing the source),
+  refuses `input == output` and `-o` with multiple inputs.
+- **`MultiCompress::Codec`** — strict, self-describing compression envelope for
+  values stored in a database. Always compresses (one storage format), 7-byte
+  magic/version/algo header, strict decode (a corrupt envelope raises
+  `DataError`, never silent passthrough), opt-in `legacy:` handling
+  (`:reject` default, `:plain`, `{ compressed: :algo }`), `encode: :base64` for
+  text columns (explicit `mc1:` prefix), locally-frozen `max_output_size`,
+  configurable `encoding:` (validated on write), and optional `serializer:`
+  (JSON/Marshal) for non-String values.
+- **ActiveRecord integration** via `MultiCompress::ActiveRecordSupport::Type`
+  (for `attribute`) and `::Coder` (for `serialize`). Opt-in `mutable:` enables
+  in-place dirty tracking.
+
+### Notes
+- The CLI writes this gem's internal LZ4 block format with the **`.mclz4`**
+  extension (not `.lz4`), since it is not interchangeable with the standard
+  `lz4` CLI frame.
+- `MultiCompress::Codec` treats one column as one fixed encoding. Use
+  `encoding: Encoding::BINARY` (a `:binary`/`bytea` column) for arbitrary bytes.
+- The AR `Type` default does not detect in-place mutation; reassign the
+  attribute, or pass `mutable: true`.
+
 ## [0.3.5]
 
 ### Changed

data/GET_STARTED.md

@@ -862,4 +862,211 @@ puts results
 # => {:lz4=>{:time=>0.0012, :ratio=>0.234, :size=>1024}, ...}
 ```
 
+## Command-Line Tool
+
+Installing the gem provides a `multi_compress` executable (gzip-style):
+
+```bash
+multi_compress file.json                 # -> file.json.zst (zstd default)
+multi_compress -a brotli -l best a.css   # -> a.css.br
+multi_compress -d file.json.zst          # -> file.json (algo from extension)
+cat big.log | multi_compress -a zstd -c > big.log.zst   # pipe mode
+```
+
+Flags: `-a/--algo` (`zstd|lz4|brotli`), `-l/--level` (number or `fastest|default|best`),
+`-d/--decompress`, `-o/--output`, `-c/--stdout`, `-k/--keep`, `-f/--force`,
+`-q/--quiet`, `--max-output SIZE` (e.g. `64MB`), `--version`.
+
+Safety: output is written to a temp file in the destination directory, fsynced,
+atomically renamed, and the directory is fsynced before the source is removed —
+an interrupted run never leaves a partial file or touches the source. The tool
+refuses `input == output` and refuses `-o` with multiple inputs.
+
+**LZ4 note:** this gem's LZ4 uses an internal block format that is *not*
+interchangeable with the standard `lz4` CLI. To avoid a false compatibility
+promise, LZ4 output uses the **`.mclz4`** extension; `.lz4` is never produced.
+
+## Database Column Compression
+
+There are two intentionally separate database paths:
+
+1. **`MultiCompress::Codec`** is a general application-side envelope for one
+   column. It is useful when only Ruby needs to read the data.
+2. **`MultiCompress::Database` (`MCDB1`)** is the narrow, frozen format for
+   MySQL 5.7 or PostgreSQL + DBeaver. It has matching server-side C readers,
+   so SQL can decode the BLOB / `bytea`.
+
+### MySQL 5.7 or PostgreSQL / DBeaver: `MultiCompress::Database` (`MCDB1`)
+
+Use this path when employees must inspect decoded values in DBeaver, DataGrip,
+or the MySQL console:
+
+```ruby
+require "multi_compress/database"
+
+blob = MultiCompress::Database.compress("Привет") # UTF-8 bytes for LONGBLOB / bytea
+text = MultiCompress::Database.decompress(blob)    # => "Привет"
+```
+
+MCDB1 is deliberately narrow: zstd only, valid non-NUL UTF-8 text, exactly one
+zstd frame, no dictionaries/Base64, and a 16 MiB decompressed limit.
+
+#### 1. Application developer: create the DBA bundle
+
+The native reader is intentionally **not** built during `gem install`: it must
+be compiled on the database host, against that host's PostgreSQL/MySQL ABI.
+The developer does not hand a DBA a Git revision or a loose `.so`; they create
+a version-locked source bundle from the installed gem:
+
+```bash
+bundle exec multi_compress db package postgres --output tmp/multi-compress-postgres.tar.gz
+# or
+bundle exec multi_compress db package mysql --output tmp/multi-compress-mysql.tar.gz
+```
+
+The archive contains the shared MCDB1 decoder, the right server-side source,
+vendored zstd, checksum manifest, and a small `Makefile`/installer. It is the
+only artifact the DBA needs.
+
+#### 2. DBA: install and enable on the database server
+
+PostgreSQL, on the PostgreSQL host:
+
+```bash
+ tar -xzf multi-compress-postgres.tar.gz
+ cd multi_compress-postgres-0.5.0
+ make verify
+ make doctor
+ sudo make install
+ sudo -u postgres make enable DB=app_production \
+  MIGRATION_ROLE=app_migrations READ_ROLE=dbeaver_readonly
+```
+
+`install` builds `multi_compress_pg.so` on that host, discovers paths through
+`pg_config`, and installs the library/control/extension SQL. Install the matching
+`postgresql-server-dev-<major>` package and a C toolchain first; `make doctor`
+checks those prerequisites before changing the server. `enable` runs
+`CREATE EXTENSION` in exactly one database. Repeat `make enable DB=...` for
+reporting databases or any separate database that must query MCDB1 columns.
+
+MySQL 5.7, on the MySQL host:
+
+```bash
+ tar -xzf multi-compress-mysql.tar.gz
+ cd multi_compress-mysql-0.5.0
+ make verify
+ make doctor MYSQL_DEFAULTS_FILE=/etc/mysql/admin.cnf
+ sudo make install MYSQL_DEFAULTS_FILE=/etc/mysql/admin.cnf
+ make enable MYSQL_DEFAULTS_FILE=/etc/mysql/admin.cnf
+```
+
+The MySQL installer verifies the local server is MySQL 5.7, asks that same
+server for `@@plugin_dir`, builds the UDF locally, and installs the library
+there. `enable` registers the functions once for that MySQL server.
+
+#### 3. Application developer: commit the readable view
+
+Do not ask DBAs or DBeaver users to type decoder calls. Generate a safe,
+quoted view definition and commit it as a Rails migration or versioned SQL:
+
+```bash
+bundle exec multi_compress db view postgres \
+  --table app.events \
+  --column payload_compressed \
+  --view admin.events_readable \
+  --columns id,created_at,status \
+  --as payload \
+  --output db/views/events_readable.sql
+```
+
+For MySQL use `db view mysql` with the same arguments. The MySQL definition
+includes `CONVERT(... USING utf8mb4)`, so Cyrillic, Kazakh text and emoji are
+returned correctly to DBeaver. If PostgreSQL was enabled in a non-default schema,
+pass `--extension-schema that_schema` to the PostgreSQL view generator.
+
+DBeaver users now make ordinary queries against the view:
+
+```sql
+SELECT id, created_at, status, payload
+FROM admin.events_readable
+WHERE id = 123;
+```
+
+The view decompresses selected rows. Filter by indexed, uncompressed fields
+before reading it; do not perform unbounded `LIKE '%text%'` searches on the
+decoded payload.
+
+The shared, long-term format contract is
+[`docs/database-envelope-v1.md`](docs/database-envelope-v1.md).
+
+### General Ruby-side `Codec`
+
+`MultiCompress::Codec` transparently compresses on write and "unwraps" on read,
+with a strict, self-describing envelope suitable for persistent storage.
+
+```ruby
+require "multi_compress/codec"
+
+codec = MultiCompress::Codec.new(algo: :zstd, level: 6)
+blob  = codec.dump(big_json)   # always-compressed envelope for a :binary column
+json  = codec.load(blob)       # exact original string back (UTF-8 by default)
+```
+
+Guarantees: always one compressed format (no mixed raw/compressed rows); a
+recognized-but-corrupt envelope raises `MultiCompress::DataError` (never silent
+data loss); invalid encoding is rejected at **write** time; the decompression
+cap is frozen locally so global config can't loosen it.
+
+Options: `encode: :base64` (for text/varchar columns; prefer `:binary`/`bytea`),
+`serializer:` (JSON or Marshal for non-String values), `encoding:`
+(default UTF-8; use `Encoding::BINARY` for arbitrary bytes), `max_output_size:`,
+`dictionary:`, and `legacy:` for reading pre-existing values:
+
+```ruby
+MultiCompress::Codec.new(legacy: :reject)                 # default: no envelope -> raise
+MultiCompress::Codec.new(legacy: :plain)                  # treat as uncompressed string
+MultiCompress::Codec.new(legacy: { compressed: :zstd })  # decompress old raw blobs
+```
+
+### ActiveRecord (experimental)
+
+Requires ActiveModel. The adapter has unit coverage but must still be integration-tested
+against each Rails/database combination before being treated as a stable deployment contract. Register the type (`attribute`) or use a coder (`serialize`):
+
+```ruby
+require "multi_compress/active_record"
+
+class Event < ApplicationRecord
+  # attribute API (prefer a :binary / bytea column)
+  attribute :payload, MultiCompress::ActiveRecordSupport::Type.new(algo: :zstd, level: 6)
+
+  # or the serialize macro with JSON objects
+  # serialize :payload, coder: MultiCompress::ActiveRecordSupport::Coder.new(algo: :zstd, serializer: JSON)
+end
+```
+
+Dirty tracking: by default in-place mutation isn't detected — reassign the
+attribute (`event.payload = event.payload.merge(...)`) or pass `mutable: true`
+to the `Type` to enable `changed_in_place?` (which decompresses the old value
+on each check).
+
 This guide covers comprehensive usage of the MultiCompress gem. For advanced use cases or questions, see the source code or create an issue on GitHub.
+
+
+## MCDB2 dictionary-backed SQL columns
+
+Use `MultiCompress::Database` when PostgreSQL/MySQL/DBeaver must read a compressed
+column. MCDB1 is dictionary-free. MCDB2 is for many small, similarly shaped JSON or
+text rows and requires one immutable dictionary registry version per row:
+
+```ruby
+dictionary = MultiCompress::Database::Dictionary.train(samples, id: 42, size: 32 * 1024)
+blob = MultiCompress::Database.compress(payload_json, dictionary: dictionary)
+```
+
+Install/upgrade the native database reader before writing MCDB2 data, generate the
+append-only registry DDL with `multi_compress db registry`, then generate an INNER
+JOIN readable view with `multi_compress db view --dictionary-table ...`. The complete
+protocol and rollout contract are in [`docs/database-envelope-v2.md`](docs/database-envelope-v2.md).
+
+For MCDB2 registry DDL, pass `--payload-table`, `--payload-column`, and `--payload-dictionary-id-column` to `multi_compress db registry`; this creates the payload FK and enforces header dictionary-reference consistency.

data/README.md

@@ -172,6 +172,73 @@ Or use the build script:
 ./build.sh
 ```
 
+## Command-line tool & DB column compression
+
+Installing the gem provides a gzip-style `multi_compress` executable:
+
+```bash
+multi_compress file.json          # -> file.json.zst
+multi_compress -d file.json.zst   # -> file.json
+cat big.log | multi_compress -a zstd -c > big.log.zst
+```
+
+`MultiCompress::Codec` is a general-purpose application-side envelope for a
+single database column (optional ActiveRecord `Type`/`Coder` adapters and
+optional Base64 for text columns). It is **not** the format for SQL-side
+inspection.
+
+For MySQL 5.7 or PostgreSQL + DBeaver, use the SQL-readable database API. `MCDB1` is dictionary-free; `MCDB2` is opt-in for a homogeneous column with an immutable zstd dictionary registry:
+
+```ruby
+require "multi_compress/database"
+
+blob = MultiCompress::Database.compress("JSON or UTF-8 text") # store in LONGBLOB / bytea
+text = MultiCompress::Database.decompress(blob)
+```
+
+For repeated small JSON/text records with the same shape, use MCDB2 and keep the
+dictionary as append-only application data:
+
+```ruby
+dictionary = MultiCompress::Database::Dictionary.train(samples, id: 42)
+blob = MultiCompress::Database.compress(payload_json, dictionary: dictionary)
+```
+
+See [`docs/database-envelope-v2.md`](docs/database-envelope-v2.md) for the
+registry, generated view, rollout and DBeaver flow.
+
+For database-side reading, the installed gem creates a **self-contained DBA
+bundle** from the exact gem version — no Git clone, manual `.so` copy, or
+hand-written extension SQL:
+
+```bash
+bundle exec multi_compress db package postgres --output tmp/multi-compress-postgres.tar.gz
+# or
+bundle exec multi_compress db package mysql --output tmp/multi-compress-mysql.tar.gz
+```
+
+The DBA extracts that archive on the database host and runs `make verify`,
+`make doctor`, and `sudo make install`. PostgreSQL enablement also requires the
+migration and read roles; MySQL uses a local Unix socket and has explicit
+`upgrade`, `disable`, and `uninstall` confirmations. The bundle compiles the
+native reader against that host's real PostgreSQL/MySQL environment and installs
+it into the server-reported directory. Generate the DBeaver-facing read view
+from the application repository as well:
+
+```bash
+bundle exec multi_compress db view postgres \
+  --table app.events --column payload_compressed \
+  --view admin.events_readable --columns id,created_at,status \
+  --output db/views/events_readable.sql
+```
+
+Use `--extension-schema NAME` with the PostgreSQL view generator only when the
+DBA enabled the extension outside the default `multi_compress` schema. See
+[GET_STARTED.md](GET_STARTED.md#database-column-compression) for the full
+app-server → DBA → DBeaver flow.
+
+> The CLI writes this gem's internal LZ4 format as `.mclz4` (not `.lz4`), since it is not interchangeable with the standard `lz4` CLI.
+
 ## Requirements
 
 - Ruby >= 2.7.1
@@ -180,3 +247,5 @@ Or use the build script:
 ## License
 
 MIT — see [LICENSE.txt](LICENSE.txt).
+
+For MCDB2 registry DDL, pass `--payload-table`, `--payload-column`, and `--payload-dictionary-id-column` to `multi_compress db registry`; this creates the payload FK and enforces header dictionary-reference consistency.

data/db_core/include/mcdb_format.h

@@ -0,0 +1,113 @@
+#ifndef MCDB_FORMAT_H
+#define MCDB_FORMAT_H
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include <zstd.h>
+
+#define MCDB_MAGIC0 0x4D /* 'M' */
+#define MCDB_MAGIC1 0x43 /* 'C' */
+#define MCDB_MAGIC2 0x44 /* 'D' */
+#define MCDB_MAGIC3 0x42 /* 'B' */
+
+#define MCDB_VERSION_V1 1
+#define MCDB_VERSION_V2 2
+#define MCDB_CODEC_ZSTD 1
+#define MCDB_FLAGS_NONE 0
+
+#define MCDB_V1_HEADER_SIZE 19u
+#define MCDB_V2_HEADER_SIZE 27u
+#define MCDB_HEADER_SIZE MCDB_V1_HEADER_SIZE /* MCDB1 compatibility alias */
+#define MCDB_ORIGINAL_SIZE_OFFSET 7u
+#define MCDB_CRC_OFFSET 15u
+#define MCDB_DICTIONARY_REF_OFFSET 19u
+
+#define MCDB_MAX_OUTPUT (16u * 1024u * 1024u) /* 16 MiB decompressed */
+#define MCDB_MAX_DICTIONARY_BYTES (256u * 1024u)
+#define MCDB_MAX_DICTIONARY_REF UINT64_C(0x7FFFFFFFFFFFFFFF)
+#define MCDB_MAX_ENVELOPE_V1 16842771u
+#define MCDB_MAX_ENVELOPE_V2 16842779u
+#define MCDB_MAX_ENVELOPE MCDB_MAX_ENVELOPE_V2
+
+#define MCDB_SHA256_BYTES 32u
+#define MCDB_ERRLEN 256
+#define MCDB_READER_VERSION "0.6.0"
+
+typedef struct {
+    uint8_t version;
+    uint64_t original_size;
+    uint64_t dictionary_ref;
+    size_t header_size;
+} mcdb_header;
+
+typedef enum {
+    MCDB_OK = 0,
+    MCDB_ERR_TRUNCATED,
+    MCDB_ERR_ENVELOPE_SIZE,
+    MCDB_ERR_MAGIC,
+    MCDB_ERR_VERSION,
+    MCDB_ERR_CODEC,
+    MCDB_ERR_FLAGS,
+    MCDB_ERR_SIZE_LIMIT,
+    MCDB_ERR_DICTIONARY_REQUIRED,
+    MCDB_ERR_DICTIONARY_REF,
+    MCDB_ERR_DICTIONARY_SIZE,
+    MCDB_ERR_DICTIONARY_ID,
+    MCDB_ERR_DECOMPRESS,
+    MCDB_ERR_TRAILING_DATA,
+    MCDB_ERR_FRAME_CONTENT_SIZE,
+    MCDB_ERR_SIZE_MISMATCH,
+    MCDB_ERR_CRC,
+    MCDB_ERR_UTF8,
+    MCDB_ERR_ALLOC
+} mcdb_status;
+
+/* Parses MCDB1 or MCDB2. It never decompresses. */
+mcdb_status mcdb_parse_header(const unsigned char *in, size_t in_len, mcdb_header *out_header);
+
+/* MCDB1 compatibility API. Rejects MCDB2. */
+mcdb_status mcdb_validate_header(const unsigned char *in, size_t in_len,
+                                 uint64_t *out_original_size);
+
+/* MCDB2 parser. Rejects MCDB1. */
+mcdb_status mcdb_validate_v2_header(const unsigned char *in, size_t in_len,
+                                    uint64_t *out_original_size,
+                                    uint64_t *out_dictionary_ref);
+
+/* The caller must validate the MCDB1 header before calling. */
+mcdb_status mcdb_decode_into(const unsigned char *in, size_t in_len,
+                             unsigned char *out, size_t out_capacity,
+                             size_t *out_len, char *errbuf);
+
+mcdb_status mcdb_decode(const unsigned char *in, size_t in_len,
+                        unsigned char **out, size_t *out_len,
+                        char *errbuf);
+
+/* MCDB2 decode. supplied_dictionary_ref must be the registry id passed by SQL. */
+mcdb_status mcdb_decode_v2_into(const unsigned char *in, size_t in_len,
+                                uint64_t supplied_dictionary_ref,
+                                const unsigned char *dictionary,
+                                size_t dictionary_len,
+                                ZSTD_DCtx *dctx,
+                                ZSTD_DDict *ddict,
+                                unsigned char *out, size_t out_capacity,
+                                size_t *out_len, char *errbuf);
+
+mcdb_status mcdb_decode_v2(const unsigned char *in, size_t in_len,
+                           uint64_t supplied_dictionary_ref,
+                           const unsigned char *dictionary,
+                           size_t dictionary_len,
+                           unsigned char **out, size_t *out_len,
+                           char *errbuf);
+
+mcdb_status mcdb_validate_dictionary(const unsigned char *dictionary, size_t dictionary_len,
+                                     uint32_t *out_zstd_dictionary_id);
+
+uint32_t mcdb_zstd_frame_dictionary_id(const unsigned char *frame, size_t frame_len);
+uint64_t mcdb_dictionary_ref(const unsigned char *in, size_t in_len, mcdb_status *out_status);
+void mcdb_sha256(const unsigned char *data, size_t len, unsigned char out[MCDB_SHA256_BYTES]);
+
+const char *mcdb_status_str(mcdb_status s);
+
+#endif