pq_crypto 0.3.0 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 15356ea593b25eefd75360992b4b716df348601a5198b8085e271e866e431371
-  data.tar.gz: b9667b1d123009b44009b8819f02c1f2855bd5117bd6142faa36cef3bcd4bf22
+  metadata.gz: 8cd88ab6ebe3042111e60711895a9563a1510057a321ac9dab510496634656b8
+  data.tar.gz: 684f469f8be7912780e00d368989418499aae00bb530d7fc32f8b6c6d5576593
 SHA512:
-  metadata.gz: 722560e27ea481d4f9acaee6798b3a34c06796d5d365a2164cb6ca612917e1b6db0e06e36c74b7624b1f33a78ab3e91930f3e67b5e67988363c4498cba24e585
-  data.tar.gz: 843fdee9d26cca30bfe5e6013a759713a062f9d63b08544ea5727b86b3b94e9b58dab0a201d52d0fe2e94c6aedb6ec7c770743bab65ea6d1fbf8545a71d81abc
+  metadata.gz: 54c1f3b0b8a2f7141d3ec4c8649c003793a3469f212fed94b0dc7ef0e2b0ff3ddba510383a0b62813d9ee98700bf266547be1900693c9ad465fb36deec91ab7b
+  data.tar.gz: 41c0bdea2d91bbd2a2e8884ee2b2a328c4c06d410fba8d92c9c1a9470b9d5597d0b8f5233b0ba87225535a80c9055033518ef1dd82e90101a9f5ffcd606b81a6

data/CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## [0.3.2] — 2026-04-25
+
+### Added — streaming ML-DSA for large inputs
+
+- Added `PQCrypto::Signature::SecretKey#sign_io(io, chunk_size: 1 << 20, context: "".b)`.
+- Added `PQCrypto::Signature::PublicKey#verify_io(io, signature, chunk_size: 1 << 20, context: "".b)` and `verify_io!`.
+- Implemented streaming pure ML-DSA through an internal FIPS 204 ExternalMu path. Existing one-shot `sign` / `verify` semantics are unchanged; no public `sign_mu` / `verify_mu` API is exposed.
+
+### Notes
+
+- Streaming is primarily for large IO inputs and lower peak memory pressure. It is not a HashML-DSA/prehash speed mode; CPU cost is still dominated by SHAKE/Keccak.
+- Default empty-context streaming signatures interoperate with the existing one-shot `verify(message, signature)` API. Non-empty `context:` must be supplied again during `verify_io`.
+
+## [0.3.1] — 2026-04-24
+
+### Fixed — X-Wing draft-10 compatibility
+
+- Changed `:ml_kem_768_x25519_xwing` secret keys to the draft-10 32-byte
+  X-Wing decapsulation seed and derive ML-KEM/X25519 private material with
+  SHAKE256 during key generation and decapsulation.
+- Corrected the X-Wing combiner transcript to
+  `ss = SHA3-256( ss_M || ss_X || ct_X || pk_X || XWingLabel )`.
+- Updated the hybrid serialization OID to the X-Wing draft OID
+  `1.3.6.1.4.1.62253.25722`.
+- Redacted key `inspect` output, removed public secret-key fingerprints,
+  improved native extension load diagnostics, switched the extension build
+  flag to C11, and aligned docs with the implementation.
+
 ## [0.3.0] — 2026-04-24
 
 **Breaking release.** Hybrid KEM keys, ciphertexts, and `pqc_container_*`
@@ -9,17 +37,14 @@ and ML-DSA-65 material is unaffected.
 ### Changed — hybrid KEM (breaking)
 
 - Replaced the 0.2.0 ad-hoc `HKDF-SHA256`-with-double-transcript combiner
-  with the **X-Wing** construction from
-  [draft-connolly-cfrg-xwing-kem](https://datatracker.ietf.org/doc/draft-connolly-cfrg-xwing-kem/):
-  `ss = SHA3-256( XWingLabel || ss_M || ss_X || ct_X || pk_X )`, where
-  `XWingLabel` is the 6-byte ASCII string `\.//^\`.
+  with a SHA3-256 X-Wing-inspired combiner. This was later corrected in
+  `0.3.1` to match draft-10 transcript order and 32-byte secret keys.
 - Renamed the hybrid algorithm symbol
   `:ml_kem_768_x25519_hkdf_sha256` → `:ml_kem_768_x25519_xwing`.
 - Retired the 0.2.0 project-local hybrid OID
-  (`2.25.260242945110721168101139140490528778800`). The new OID
-  (`2.25.318532651283923671095712569430174917109`) identifies the X-Wing
-  combiner. `pqc_container_*` blobs carry the new OID; decoding a 0.2.0
-  hybrid container now fails fast with `SerializationError`.
+  (`2.25.260242945110721168101139140490528778800`). 0.3.0 used
+  `2.25.318532651283923671095712569430174917109`; this was later replaced
+  in `0.3.1` by the X-Wing draft OID.
 
 ### Changed — native code hygiene
 
@@ -39,9 +64,8 @@ and ML-DSA-65 material is unaffected.
   `hybrid_public_key_t`, `hybrid_secret_key_t`, and
   `hybrid_ciphertext_t` so any future change that introduces padding
   fails at compile time rather than silently shifting byte offsets.
-- Migrated PEM codec from `EVP_EncodeBlock` / `EVP_DecodeBlock` to the
-  streaming `EVP_EncodeUpdate` / `EVP_DecodeUpdate` API, which rejects
-  invalid base64 characters rather than treating them as zeros.
+- Migrated PEM codec to OpenSSL `BIO_f_base64` with stricter PEM
+  header/footer framing and trailing-garbage checks.
 - Deleted the entire internal HKDF and SHA-256 helper paths that 0.2.0
   used for its combiner; the X-Wing combiner is a single SHA3-256
   invocation through `EVP_DigestUpdate`.
@@ -61,9 +85,9 @@ and ML-DSA-65 material is unaffected.
   `CRYPTO_memcmp` through a new `PQCrypto.ct_equals` native helper, so
   key equality checks no longer leak timing information about a
   prefix-match.
-- `SecretKey#hash` (and `PublicKey#hash` for symmetry) now hash a
-  SHA-256 fingerprint of the bytes instead of the raw bytes, and a
-  public `#fingerprint` method is exposed.
+- `PublicKey#hash` and `SecretKey#hash` now hash a SHA-256 fingerprint
+  of the bytes instead of the raw bytes. The public secret-key fingerprint
+  method is removed in `0.3.1` to reduce accidental logging risk.
 - Native entrypoints and their `native_*` aliases are installed once via
   the new `PQCrypto::NativeBindings` module instead of the ad-hoc
   `unless method_defined?` guards on the singleton.
@@ -72,7 +96,8 @@ and ML-DSA-65 material is unaffected.
 
 ### Changed — packaging
 
-- `required_ruby_version` from `">= 3.4.0.a"` to `">= 3.4"`.
+- Intended to change `required_ruby_version` from `">= 3.4.0.a"` to
+  `">= 3.4"`; the gemspec is aligned in `0.3.1`.
 - Version bumped to `0.3.0`.
 - `VerificationError` class is still defined (and still raised by
   `verify!`) for backward compatibility, but the native `verify`

data/GET_STARTED.md

@@ -35,6 +35,29 @@ sig.public_key.verify("message", signature)
 sig.public_key.verify!("message", signature)
 ```
 
+For large files, use streaming ML-DSA:
+
+```ruby
+signature = File.open("document.bin", "rb") do |io|
+  sig.secret_key.sign_io(io, chunk_size: 1 << 20)
+end
+
+ok = File.open("document.bin", "rb") do |io|
+  sig.public_key.verify_io(io, signature, chunk_size: 1 << 20)
+end
+```
+
+With an optional context:
+
+```ruby
+ctx = "document-v1".b
+signature = File.open("document.bin", "rb") { |io| sig.secret_key.sign_io(io, context: ctx) }
+ok = File.open("document.bin", "rb") { |io| sig.public_key.verify_io(io, signature, context: ctx) }
+```
+
+`sign_io` / `verify_io` are pure ML-DSA streaming helpers, not prehash
+shortcuts. `verify_io!` raises on mismatch.
+
 ## 6. Hybrid KEM (X-Wing)
 
 ```ruby
@@ -43,6 +66,9 @@ result = hybrid.public_key.encapsulate
 shared_secret = hybrid.secret_key.decapsulate(result.ciphertext)
 ```
 
+The raw X-Wing secret key exported by this API is the draft-10 32-byte
+decapsulation seed, not the expanded ML-KEM/X25519 private material.
+
 The hybrid mode follows `draft-connolly-cfrg-xwing-kem`. See
 `SECURITY.md` for audit status.
 

data/README.md

@@ -13,13 +13,14 @@ It exposes three public building blocks:
 The gem is backed by vendored `PQClean` sources for `ML-KEM-768` /
 `ML-DSA-65` and by OpenSSL for `X25519` and `SHA3-256`. Every piece of
 conventional-crypto functionality goes through standard library calls
-(`EVP_*`, `RAND_bytes`, `CRYPTO_memcmp`, streaming `EVP_Encode*` /
-`EVP_Decode*`) — nothing roll-your-own where a library primitive exists.
+(`EVP_*`, `RAND_bytes`, `CRYPTO_memcmp`, `BIO_f_base64`) — nothing
+roll-your-own where a library primitive exists.
 
 ## Status
 
 - primitive-first API only
 - no protocol/session helpers in the public surface
+- streaming ML-DSA signing/verification is available for large IO inputs
 - serialization uses pq_crypto-specific `pqc_container_*` wrappers
 - not audited
 - not yet positioned as production-ready
@@ -42,7 +43,21 @@ bundle exec rake compile
 
 - Ruby 3.4.x
 - a C toolchain with C11 support (for `_Static_assert` / `_Thread_local`)
-- OpenSSL **3.0 or later** with SHA3-256 available (default provider)
+- OpenSSL **3.0 or later** with SHA3-256 and SHAKE256 available (default provider)
+
+### Build-time Keccak backend
+
+The default build uses PQClean's scalar `common/fips202.c` backend:
+
+```bash
+PQCRYPTO_KECCAK_BACKEND=clean bundle exec rake compile
+```
+
+`PQCRYPTO_KECCAK_BACKEND=xkcp` is reserved for a separately vendored,
+reviewed, `fips202.h`-compatible XKCP adapter. If requested without that
+adapter, the build aborts instead of silently falling back to `clean`.
+This avoids mixing OpenSSL EVP SHAKE state with PQClean SHAKE state and
+keeps output-byte compatibility explicit.
 
 ## Async / Fiber scheduler support
 
@@ -100,6 +115,8 @@ shared_secret = keypair.secret_key.decapsulate(result.ciphertext)
 
 ### ML-DSA-65
 
+One-shot signing keeps the existing API:
+
 ```ruby
 keypair = PQCrypto::Signature.generate(:ml_dsa_65)
 signature = keypair.secret_key.sign("hello")
@@ -108,6 +125,36 @@ keypair.public_key.verify("hello", signature)    # => true / false
 keypair.public_key.verify!("hello", signature)   # raises on mismatch
 ```
 
+For large inputs, use streaming IO so the message does not need to be
+materialized as one Ruby string:
+
+```ruby
+signature = File.open("document.bin", "rb") do |io|
+  keypair.secret_key.sign_io(io, chunk_size: 1 << 20)
+end
+
+ok = File.open("document.bin", "rb") do |io|
+  keypair.public_key.verify_io(io, signature, chunk_size: 1 << 20)
+end
+```
+
+`sign_io` / `verify_io` use pure ML-DSA with an internal FIPS 204
+ExternalMu flow. They are not HashML-DSA/prehash shortcuts and do not
+expose public `sign_mu` / `verify_mu` APIs. With the default empty
+context, streaming signatures verify with `verify(message, signature)`
+and one-shot signatures verify with `verify_io(io, signature)`.
+
+Optional context is supported and must match on verify:
+
+```ruby
+ctx = "invoice-v1".b
+signature = File.open("document.bin", "rb") { |io| keypair.secret_key.sign_io(io, context: ctx) }
+ok = File.open("document.bin", "rb") { |io| keypair.public_key.verify_io(io, signature, context: ctx) }
+```
+
+`chunk_size` must be positive. `context` is limited to 255 bytes by
+FIPS 204. `verify_io!` raises `PQCrypto::VerificationError` on mismatch.
+
 Note: `verify` returns a plain boolean for normal outcomes. `verify!`
 raises `PQCrypto::VerificationError` when the signature does not
 match.
@@ -120,14 +167,16 @@ result = keypair.public_key.encapsulate
 shared_secret = keypair.secret_key.decapsulate(result.ciphertext)
 ```
 
-The combiner is exactly:
+The implementation follows draft-10 key expansion: the X-Wing secret
+decapsulation key is a 32-byte seed expanded with SHAKE256 into ML-KEM
+and X25519 private material. The combiner is exactly:
 
 ```
-ss = SHA3-256( "\.//^\" || ss_M || ss_X || ct_X || pk_X )
+ss = SHA3-256( ss_M || ss_X || ct_X || pk_X || "\.//^\" )
 ```
 
-as specified by `draft-connolly-cfrg-xwing-kem`. See `SECURITY.md` for
-audit status and interoperability caveats.
+as specified by `draft-connolly-cfrg-xwing-kem-10`. See `SECURITY.md`
+for audit status and interoperability caveats.
 
 ## Serialization
 
@@ -172,6 +221,12 @@ key.wipe!                  # scrub the key's internal copy
 `CRYPTO_memcmp` through a `PQCrypto.ct_equals` helper so comparisons
 do not leak timing information about a prefix match.
 
+Secret key `inspect` output is intentionally redacted and secret key
+objects do not expose a public `fingerprint` method. `wipe!` remains
+best-effort only: it clears the current Ruby string buffer owned by the
+key object, not every possible copy made by Ruby, OpenSSL, serialization,
+logging, or the garbage collector.
+
 ## Introspection
 
 ```ruby

data/SECURITY.md

@@ -2,7 +2,7 @@
 
 ## Scope of the public API
 
-`pq_crypto` 0.3.0 exposes a primitive-first public surface:
+`pq_crypto` exposes a primitive-first public surface:
 
 - `PQCrypto::KEM` (`ML-KEM-768`)
 - `PQCrypto::Signature` (`ML-DSA-65`)
@@ -29,23 +29,25 @@ gem.
 ### HybridKEM
 
 `PQCrypto::HybridKEM` implements the **X-Wing** construction from
-[`draft-connolly-cfrg-xwing-kem`](https://datatracker.ietf.org/doc/draft-connolly-cfrg-xwing-kem/):
+[`draft-connolly-cfrg-xwing-kem-10`](https://datatracker.ietf.org/doc/draft-connolly-cfrg-xwing-kem/).
 
-    ss = SHA3-256( XWingLabel || ss_M || ss_X || ct_X || pk_X )
+The X-Wing secret decapsulation key is a 32-byte seed. It is expanded
+with SHAKE256 into the ML-KEM-768 and X25519 private material used
+internally for decapsulation. The public key and ciphertext are the
+fixed-length concatenations specified by the draft.
 
-where `XWingLabel = "\.//^\"` (6 ASCII bytes). The hybrid public key,
-secret key, and ciphertext are the byte concatenations of the ML-KEM
-and X25519 halves exactly as specified by X-Wing.
+    ss = SHA3-256( ss_M || ss_X || ct_X || pk_X || XWingLabel )
+
+where `XWingLabel = "\.//^\"` (6 ASCII bytes).
 
 X-Wing as specified has a proof of classical IND-CCA security under
 the strong Diffie-Hellman assumption for X25519 (in the ROM), and
 post-quantum IND-CCA security in the standard model assuming ML-KEM-768
 is IND-CCA secure and SHA3-256 behaves as a PRF.
 
-This gem is **not** yet part of a cross-language X-Wing interop test
-suite; wire-format claims for this release are limited to "follows the
-X-Wing draft as of version 10." External interoperability should be
-verified against the reference implementation before relying on it.
+This gem is intended to match the X-Wing draft as of version 10. External
+interoperability should still be verified against the reference
+implementation before relying on it.
 
 ### Deterministic test hooks
 
@@ -68,16 +70,16 @@ They are:
 - not real PKCS#8
 - not advertised as interoperable with OpenSSL, Go, Java, or PKI tooling
 
-The OIDs embedded in these containers are project-local UUID-derived
-OIDs under `2.25.*`. They are part of pq_crypto's own serialized
-container schema, not external interoperability identifiers.
+The `pqc_container_*` envelope itself is project-specific. ML-KEM and
+ML-DSA currently use project-local UUID-derived OIDs under `2.25.*`.
+Hybrid X-Wing uses the draft X-Wing OID `1.3.6.1.4.1.62253.25722`.
 
 The hybrid OID used by 0.2.0
-(`2.25.260242945110721168101139140490528778800`) is retired in 0.3.0
-because the combiner semantics changed (0.2.0 used an ad-hoc
-HKDF-SHA256 combiner; 0.3.0 uses X-Wing / SHA3-256). The new hybrid
-OID is `2.25.318532651283923671095712569430174917109`. A 0.2.0 hybrid
-container is rejected at decode time in 0.3.0.
+(`2.25.260242945110721168101139140490528778800`) is retired. The
+intermediate 0.3.0 project-local hybrid OID
+(`2.25.318532651283923671095712569430174917109`) is also retired in
+favor of the draft X-Wing OID. Older hybrid containers are rejected at
+decode time.
 
 ## Memory wiping
 
@@ -98,9 +100,19 @@ OpenSSL is used for:
 - `SHA3-256` (X-Wing combiner, via `EVP_sha3_256`)
 - `RAND_bytes` (production entropy source for `randombytes()`)
 - `CRYPTO_memcmp` (constant-time comparison used by `PQCrypto.ct_equals`)
-- Base64 encode/decode for PEM via the streaming `EVP_Encode*` /
-  `EVP_Decode*` API, which rejects invalid base64 rather than treating
-  it as zeros.
+- Base64 encode/decode for PEM via OpenSSL `BIO_f_base64`, with strict
+  header/footer framing and trailing-garbage checks.
+
+## Secret key display and wiping
+
+Secret key objects redact `inspect` output and intentionally do not expose
+a public `fingerprint` method. This avoids accidental logging of raw secret
+bytes or stable secret-derived identifiers.
+
+`wipe!` is best-effort only. It wipes the current Ruby string buffer held
+by the key object; it cannot guarantee erasure of copies made by Ruby,
+OpenSSL, native wrapper buffers, serialization, logging, crash dumps, or
+the garbage collector.
 
 ## Threading
 

data/ext/pqcrypto/extconf.rb

@@ -3,7 +3,7 @@
 
 require "mkmf"
 
-$CFLAGS << " -std=c99 -Wall -Wextra -O2"
+$CFLAGS << " -std=c11 -Wall -Wextra -O2"
 $CFLAGS << " -fstack-protector-strong -D_FORTIFY_SOURCE=2"
 VENDOR_ONLY_CFLAGS = "-Wno-unused-parameter -Wno-unused-function -Wno-strict-prototypes -Wno-pedantic -Wno-c23-extensions -Wno-undef"
 
@@ -11,10 +11,14 @@ $LDFLAGS << " -Wl,-no_warn_duplicate_libraries" if RbConfig::CONFIG["host_os"] =
 
 USE_SYSTEM = arg_config("--use-system-libraries") || ENV["PQCRYPTO_USE_SYSTEM_LIBRARIES"]
 
+KECCAK_BACKEND = (ENV["PQCRYPTO_KECCAK_BACKEND"] || "clean").strip.downcase
+SUPPORTED_KECCAK_BACKENDS = %w[clean xkcp].freeze
+
 SANITIZE = ENV["PQCRYPTO_SANITIZE"]
 
 if SANITIZE && !SANITIZE.strip.empty?
   sanitize = SANITIZE.strip
+  $CFLAGS.gsub!(/\s-D_FORTIFY_SOURCE=\d+/, "")
   $CFLAGS << " -O1 -g -fno-omit-frame-pointer -fsanitize=#{sanitize}"
   $LDFLAGS << " -fsanitize=#{sanitize}"
 end
@@ -72,9 +76,53 @@ def configure_openssl!
   SRC
   abort "OpenSSL SHA3-256 is required (X-Wing combiner)" unless try_compile(sha3_check)
 
+  shake_check = <<~SRC
+    #include <openssl/evp.h>
+    int main(void) {
+        const EVP_MD *md = EVP_shake256();
+        return md == NULL ? 1 : 0;
+    }
+  SRC
+  abort "OpenSSL SHAKE256 is required (X-Wing key expansion)" unless try_compile(shake_check)
+
   $CFLAGS << " -DHAVE_OPENSSL_EVP_H -DHAVE_OPENSSL_RAND_H"
 end
 
+def configure_keccak_backend(vendor_dir, common_dir)
+  abort "Unsupported PQCRYPTO_KECCAK_BACKEND=#{KECCAK_BACKEND.inspect}. Supported: #{SUPPORTED_KECCAK_BACKENDS.join(", ")}" unless SUPPORTED_KECCAK_BACKENDS.include?(KECCAK_BACKEND)
+
+  case KECCAK_BACKEND
+  when "clean"
+    {
+      name: "clean",
+      include_dirs: [],
+      source_group: ["pqclean_common", [File.join(common_dir, "fips202.c")]]
+    }
+  when "xkcp"
+    # The optimized backend must provide the same fips202.h-compatible API as
+    # PQClean's common/fips202.c. Do not substitute OpenSSL EVP SHAKE here: the
+    # PQClean SHAKE state layout is part of the ML-KEM/ML-DSA call graph.
+    xkcp_dir = File.join(vendor_dir, "xkcp")
+    adapter_source = File.join(xkcp_dir, "pqclean_fips202_xkcp.c")
+
+    abort <<~MSG unless File.exist?(adapter_source)
+      PQCRYPTO_KECCAK_BACKEND=xkcp was requested, but no reviewed XKCP adapter was found.
+
+      Expected:
+        #{adapter_source}
+
+      Refusing to fall back silently to the clean backend. Vendor a fips202.h-compatible
+      XKCP adapter first, then run the full SHAKE-dependent KAT/regression test matrix.
+    MSG
+
+    {
+      name: "xkcp",
+      include_dirs: [xkcp_dir],
+      source_group: ["xkcp_keccak", [adapter_source]]
+    }
+  end
+end
+
 def configure_pqclean(vendor_dir)
   return nil unless vendor_dir
 
@@ -85,17 +133,20 @@ def configure_pqclean(vendor_dir)
   mldsa_dir = File.join(pqclean_dir, "crypto_sign", "ml-dsa-65", "clean")
   common_dir = File.join(pqclean_dir, "common")
 
-  include_dirs = [mlkem_dir, mldsa_dir, common_dir]
+  keccak_config = configure_keccak_backend(vendor_dir, common_dir)
+
+  include_dirs = [mlkem_dir, mldsa_dir, common_dir, *keccak_config[:include_dirs]]
   return nil unless include_dirs.all? { |dir| Dir.exist?(dir) }
 
   mlkem_sources = Dir.glob(File.join(mlkem_dir, "*.c")).sort
   mldsa_sources = Dir.glob(File.join(mldsa_dir, "*.c")).sort
-  common_sources = %w[fips202.c sha2.c sp800-185.c].map { |name| File.join(common_dir, name) }
+  common_sources = %w[sha2.c sp800-185.c].map { |name| File.join(common_dir, name) }
 
   source_groups = [
     ["pqclean_mlkem", mlkem_sources],
     ["pqclean_mldsa", mldsa_sources],
-    ["pqclean_common", common_sources]
+    ["pqclean_common", common_sources],
+    keccak_config[:source_group]
   ]
 
   return nil unless source_groups.all? { |_, sources| sources.all? { |path| File.exist?(path) } }
@@ -105,6 +156,7 @@ def configure_pqclean(vendor_dir)
 
   {
     include_dirs: include_dirs,
+    keccak_backend: keccak_config[:name],
     source_groups: source_groups
   }
 end
@@ -155,6 +207,7 @@ pqclean_config = configure_pqclean(vendor_dir)
 puts "OpenSSL: system"
 abort "PQClean vendored sources are required. Run: bundle exec rake vendor" unless pqclean_config
 puts "PQClean: vendored (randombytes overridden by pq_randombytes.c)"
+puts "Keccak backend: #{pqclean_config[:keccak_backend]}"
 puts "Output: pqcrypto/pqcrypto_secure"
 puts "===================================="