pq_crypto 0.3.2 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8cd88ab6ebe3042111e60711895a9563a1510057a321ac9dab510496634656b8
-  data.tar.gz: 684f469f8be7912780e00d368989418499aae00bb530d7fc32f8b6c6d5576593
+  metadata.gz: 4f34f4ae9f34414fbbc1f6ffc63de9c24306a6a432138e36442c1e43661fa59e
+  data.tar.gz: 6b2a754c55b9a1be4706cf5d835ee08844b54d75d96d43611f22c8c4f68cb10f
 SHA512:
-  metadata.gz: 54c1f3b0b8a2f7141d3ec4c8649c003793a3469f212fed94b0dc7ef0e2b0ff3ddba510383a0b62813d9ee98700bf266547be1900693c9ad465fb36deec91ab7b
-  data.tar.gz: 41c0bdea2d91bbd2a2e8884ee2b2a328c4c06d410fba8d92c9c1a9470b9d5597d0b8f5233b0ba87225535a80c9055033518ef1dd82e90101a9f5ffcd606b81a6
+  metadata.gz: 0b72822b4b645f891e8f87693004f736070b9bba37432a48ffc840ec1a114865ba691db7c1ffc12aab7643369c3883892d9fbcf3c4f7cf895b5938d2ee650b1c
+  data.tar.gz: 13c0263600408685f5d484528032305c48e3e3d551799abad990167c9a9d14d0287aabb70fd88dc00f72d6236f15e70c3d9f6e9572ad02276e11674aafd87774

data/.github/workflows/ci.yml

@@ -35,3 +35,59 @@ jobs:
 
       - name: Run tests
         run: bundle exec rake test
+
+  interop-openssl-3-5-required:
+    name: interop-openssl-3.5-required
+    runs-on: ubuntu-24.04
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Build OpenSSL 3.5
+        run: |
+          set -euxo pipefail
+          sudo apt-get update
+          sudo apt-get install -y build-essential perl pkg-config curl ca-certificates
+          curl -fsSLO https://www.openssl.org/source/openssl-3.5.0.tar.gz
+          tar -xzf openssl-3.5.0.tar.gz
+          cd openssl-3.5.0
+          ./Configure linux-x86_64 --prefix=/opt/openssl-3.5 --openssldir=/opt/openssl-3.5 shared
+          make -s -j"$(nproc)"
+          sudo make -s install_sw
+          echo /opt/openssl-3.5/lib64 | sudo tee /etc/ld.so.conf.d/openssl-3.5.conf
+          sudo ldconfig
+          /opt/openssl-3.5/bin/openssl version
+          /opt/openssl-3.5/bin/openssl version | grep -E '^OpenSSL 3\.(5|[6-9]|[1-9][0-9])\.'
+          echo /opt/openssl-3.5/bin >> "$GITHUB_PATH"
+          {
+            echo "OPENSSL_ROOT_DIR=/opt/openssl-3.5"
+            echo "PKG_CONFIG_PATH=/opt/openssl-3.5/lib64/pkgconfig"
+            echo "LD_LIBRARY_PATH=/opt/openssl-3.5/lib64"
+          } >> "$GITHUB_ENV"
+
+      - name: Set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: "3.4"
+          bundler-cache: true
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: "1.26.0"
+
+      - name: Compile extension
+        run: bundle exec rake compile
+
+      - name: Run tests
+        run: |
+          set -o pipefail
+          bundle exec rake test 2>&1 | tee rake_test.log
+
+      - name: Refuse skipped OpenSSL 3.5 interop tests
+        run: |
+          if grep -E "Skipped: .* (mlkem|mldsa)_spki" rake_test.log; then
+            echo "OpenSSL 3.5 interop tests must NOT skip on this matrix entry."
+            exit 1
+          fi

data/CHANGELOG.md

@@ -1,5 +1,67 @@
 # Changelog
 
+## [0.5.0] - 2026-05-04
+
+### Changed — native backend migration
+
+- Replaced the PQClean runtime/build path with PQ Code Package `mlkem-native` and `mldsa-native` as the only ML-KEM / ML-DSA backend.
+- Removed the PQClean fallback entirely so backend failures are attributable to the new native path instead of mixed old/new implementations.
+- Updated the native extension build to require `ext/pqcrypto/vendor/mlkem-native/mlkem/mlkem_native.c` and `ext/pqcrypto/vendor/mldsa-native/mldsa/mldsa_native.c`.
+- Changed vendoring and gem packaging to keep only a minimal PQ Code Package source snapshot, avoiding upstream examples and symlink-heavy trees in packaged gems.
+- Switched native compilation to `-O3`; optional upstream native assembly remains opt-in through `PQCRYPTO_NATIVE_ASM=1`.
+- `PQCrypto.backend` now reports `:native_pq_code_package`.
+
+### Fixed
+
+- Corrected deterministic ML-DSA test signing on `mldsa-native` by passing the FIPS 204 pure-mode domain-separation prefix (`00 00` for an empty context) into `signature_internal`. This restores deterministic round-trip verification and ML-DSA siggen KAT compatibility.
+
+### Documentation
+
+- Updated README, GET_STARTED, SECURITY, and migration notes for the native-only backend and the no-PQClean policy.
+- Documented that `pqc_container_*` remains a compatibility serialization format while the cryptographic backend has moved to PQ Code Package native sources.
+
+### Migration notes
+
+- Source checkouts must refresh vendor sources with `bundle exec rake vendor` before compiling if `ext/pqcrypto/vendor/.vendored` is missing or stale.
+- This release intentionally does not support falling back to PQClean. If native sources are absent or incompatible, the extension build fails early.
+
+## [0.4.2] - 2026-04-29
+
+### Fixed
+- Fixed native extension build from packaged gem by keeping generated
+  `pqcrypto_version.h` available after `make clean`.
+
+## [0.4.1] — 2026-04-29
+
+### Fixed
+
+- Hardened native ML-DSA signing error handling by checking RNG failures.
+- Improved streaming ML-DSA validation and context handling without regressing throughput.
+
+## [0.4.0] — 2026-04-28
+
+### Added — standard serialization and expanded parameter sets
+
+- Added a central algorithm registry with legacy `pqc_container_*` OIDs kept separate from RFC 9935 / RFC 9881 standard OIDs.
+- Added RFC 9935 SPKI public-key and PKCS#8 private-key support for ML-KEM.
+- Added RFC 9881 SPKI public-key and expanded-key PKCS#8 support for ML-DSA.
+- Added public API support for ML-KEM-512, ML-KEM-1024, ML-DSA-44, and ML-DSA-87 alongside the existing ML-KEM-768 and ML-DSA-65 support.
+- Added OpenSSL 3.5+ interoperability tests for standard SPKI / PKCS#8 encodings where supported by the linked OpenSSL.
+- Added NIST ACVP KAT test infrastructure for ML-KEM and ML-DSA parameter sets.
+- Added opt-in ML-DSA seed/both PKCS#8 import support. The default remains off; callers must set `PQCrypto::PKCS8.allow_ml_dsa_seed_format = true`.
+
+### Changed
+
+- Bumped the gem version to `0.4.0`.
+- `pqc_container_*` remains frozen and project-local. It is still limited to the original three algorithms: `:ml_kem_768`, `:ml_dsa_65`, and `:ml_kem_768_x25519_xwing`.
+- `DETAILS[:oid]` continues to expose the legacy `pqc_container_*` OID for backward compatibility. Standard OIDs are available via `PQCrypto::AlgorithmRegistry.standard_oid`.
+- The native `PQCrypto.version` path now derives from `lib/pq_crypto/version.rb` through a generated C header, avoiding a second manually maintained version string.
+
+### Security notes
+
+- ML-DSA seed-format imports are opt-in because PQClean does not expose a public ML-DSA derandomized keypair entrypoint. The implementation reuses the thread-local seed-replay path documented in `SECURITY.md`.
+- The project remains unaudited and experimental.
+
 ## [0.3.2] — 2026-04-25
 
 ### Added — streaming ML-DSA for large inputs

data/GET_STARTED.md

@@ -1,98 +1,424 @@
 # Getting started with pq_crypto
 
-## 1. Build the extension
+This guide shows the common `pq_crypto` workflows. The README is intentionally
+short; examples and practical details live here.
+
+## 1. Install
+
+Add the gem to your application:
+
+```ruby
+# Gemfile
+gem "pq_crypto"
+```
+
+Install it:
 
 ```bash
 bundle install
+```
+
+If you are working from a repository checkout, compile the native extension:
+
+```bash
 bundle exec rake compile
 ```
 
-## 2. Generate an ML-KEM-768 keypair
+Use the gem from Ruby:
+
+```ruby
+require "pq_crypto"
+```
+
+## 2. Check the available algorithms
+
+```ruby
+PQCrypto.supported_kems
+# => [:ml_kem_512, :ml_kem_768, :ml_kem_1024]
+
+PQCrypto.supported_signatures
+# => [:ml_dsa_44, :ml_dsa_65, :ml_dsa_87]
+
+PQCrypto.supported_hybrid_kems
+# => [:ml_kem_768_x25519_xwing]
+```
+
+Useful metadata:
+
+```ruby
+PQCrypto.version
+PQCrypto.backend
+
+PQCrypto::KEM.details(:ml_kem_768)
+PQCrypto::Signature.details(:ml_dsa_65)
+PQCrypto::AlgorithmRegistry.standard_oid(:ml_kem_768)
+```
+
+`DETAILS[:oid]` remains the legacy `pqc_container_*` OID for backward
+compatibility. Use `AlgorithmRegistry.standard_oid` when you need the standard
+RFC OID.
+
+## 3. ML-KEM: generate, encapsulate, decapsulate
+
+Generate a keypair:
 
 ```ruby
 keypair = PQCrypto::KEM.generate(:ml_kem_768)
+public_key = keypair.public_key
+secret_key = keypair.secret_key
 ```
 
-## 3. Encapsulate and decapsulate
+Encapsulate with the public key:
 
 ```ruby
-result = keypair.public_key.encapsulate
-shared_secret = keypair.secret_key.decapsulate(result.ciphertext)
+result = public_key.encapsulate
+ciphertext = result.ciphertext
+sender_shared_secret = result.shared_secret
+```
+
+Decapsulate with the secret key:
+
+```ruby
+receiver_shared_secret = secret_key.decapsulate(ciphertext)
+
+sender_shared_secret == receiver_shared_secret
+# => true
+```
+
+The same API shape applies to other supported ML-KEM parameter sets:
+
+```ruby
+PQCrypto::KEM.generate(:ml_kem_512)
+PQCrypto::KEM.generate(:ml_kem_1024)
+```
+
+## 4. ML-DSA: sign and verify
+
+Generate a signature keypair:
+
+```ruby
+keypair = PQCrypto::Signature.generate(:ml_dsa_65)
+public_key = keypair.public_key
+secret_key = keypair.secret_key
 ```
 
-## 4. Generate an ML-DSA-65 keypair
+Sign and verify a message:
 
 ```ruby
-sig = PQCrypto::Signature.generate(:ml_dsa_65)
+message = "hello".b
+signature = secret_key.sign(message)
+
+public_key.verify(message, signature)
+# => true
+
+public_key.verify!(message, signature)
+# returns true, or raises on mismatch
 ```
 
-## 5. Sign and verify
+The same API shape applies to other supported ML-DSA parameter sets:
 
 ```ruby
-signature = sig.secret_key.sign("message")
+PQCrypto::Signature.generate(:ml_dsa_44)
+PQCrypto::Signature.generate(:ml_dsa_87)
+```
+
+## 5. ML-DSA for large files
+
+For large inputs, use the streaming helpers so the whole message does not need
+to be materialized as one Ruby string.
+
+```ruby
+keypair = PQCrypto::Signature.generate(:ml_dsa_65)
+
+signature = File.open("document.bin", "rb") do |io|
+  keypair.secret_key.sign_io(io, chunk_size: 1 << 20)
+end
 
-sig.public_key.verify("message", signature)
-sig.public_key.verify!("message", signature)
+ok = File.open("document.bin", "rb") do |io|
+  keypair.public_key.verify_io(io, signature, chunk_size: 1 << 20)
+end
 ```
 
-For large files, use streaming ML-DSA:
+With an optional FIPS 204 context:
 
 ```ruby
+context = "invoice-v1".b
+
 signature = File.open("document.bin", "rb") do |io|
-  sig.secret_key.sign_io(io, chunk_size: 1 << 20)
+  keypair.secret_key.sign_io(io, context: context)
 end
 
 ok = File.open("document.bin", "rb") do |io|
-  sig.public_key.verify_io(io, signature, chunk_size: 1 << 20)
+  keypair.public_key.verify_io(io, signature, context: context)
 end
 ```
 
-With an optional context:
+Notes:
+
+- `context` must match during verification.
+- `context` is limited to 255 bytes by FIPS 204.
+- `chunk_size` must be positive.
+- `sign_io` / `verify_io` are pure ML-DSA streaming helpers. They are not
+  HashML-DSA/prehash shortcuts and do not expose public `sign_mu` /
+  `verify_mu` APIs.
+
+## 6. Hybrid KEM: ML-KEM-768 + X25519 X-Wing
+
+```ruby
+keypair = PQCrypto::HybridKEM.generate(:ml_kem_768_x25519_xwing)
+
+result = keypair.public_key.encapsulate
+ciphertext = result.ciphertext
+sender_shared_secret = result.shared_secret
+
+receiver_shared_secret = keypair.secret_key.decapsulate(ciphertext)
+
+sender_shared_secret == receiver_shared_secret
+# => true
+```
+
+The raw X-Wing secret key exported by this API is the draft-style 32-byte
+secret seed, not the expanded ML-KEM/X25519 private material.
+
+See `SECURITY.md` for audit status and hybrid interoperability caveats.
+
+## 7. SPKI public-key serialization
+
+Use SPKI when you need standard public-key serialization for ML-KEM or ML-DSA.
+
+ML-KEM example:
+
+```ruby
+keypair = PQCrypto::KEM.generate(:ml_kem_768)
+
+pem = keypair.public_key.to_spki_pem
+imported = PQCrypto::KEM.public_key_from_spki_pem(pem)
+
+imported == keypair.public_key
+# => true
+```
+
+DER form:
+
+```ruby
+der = keypair.public_key.to_spki_der
+imported = PQCrypto::KEM.public_key_from_spki_der(der)
+```
+
+ML-DSA example:
+
+```ruby
+keypair = PQCrypto::Signature.generate(:ml_dsa_65)
+
+pem = keypair.public_key.to_spki_pem
+imported = PQCrypto::Signature.public_key_from_spki_pem(pem)
+```
+
+You can require an expected algorithm during import:
 
 ```ruby
-ctx = "document-v1".b
-signature = File.open("document.bin", "rb") { |io| sig.secret_key.sign_io(io, context: ctx) }
-ok = File.open("document.bin", "rb") { |io| sig.public_key.verify_io(io, signature, context: ctx) }
+PQCrypto::KEM.public_key_from_spki_pem(pem, algorithm: :ml_kem_768)
 ```
 
-`sign_io` / `verify_io` are pure ML-DSA streaming helpers, not prehash
-shortcuts. `verify_io!` raises on mismatch.
+## 8. PKCS#8 private-key serialization
 
-## 6. Hybrid KEM (X-Wing)
+Use PKCS#8 when you need standard private-key serialization for ML-KEM or
+ML-DSA.
+
+### ML-KEM PKCS#8
+
+`SecretKey#to_pkcs8_pem` exports the expanded private key by default:
 
 ```ruby
-hybrid = PQCrypto::HybridKEM.generate(:ml_kem_768_x25519_xwing)
-result = hybrid.public_key.encapsulate
-shared_secret = hybrid.secret_key.decapsulate(result.ciphertext)
+keypair = PQCrypto::KEM.generate(:ml_kem_768)
+
+pem = keypair.secret_key.to_pkcs8_pem
+imported = PQCrypto::KEM.secret_key_from_pkcs8_pem(pem)
+
+imported == keypair.secret_key
+# => true
 ```
 
-The raw X-Wing secret key exported by this API is the draft-10 32-byte
-decapsulation seed, not the expanded ML-KEM/X25519 private material.
+DER form:
+
+```ruby
+der = keypair.secret_key.to_pkcs8_der
+imported = PQCrypto::KEM.secret_key_from_pkcs8_der(der)
+```
 
-The hybrid mode follows `draft-connolly-cfrg-xwing-kem`. See
-`SECURITY.md` for audit status.
+ML-KEM PKCS#8 supports `:seed`, `:expanded`, and `:both` formats. A generated
+`SecretKey` does not retain the original seed, so exporting `:seed` or `:both`
+from `SecretKey#to_pkcs8_*` is intentionally unavailable. If you explicitly
+have the seed material, use the low-level PKCS#8 encoder.
 
-## 7. Serialize a key
+### ML-DSA PKCS#8
+
+ML-DSA private keys support expanded-key PKCS#8 by default:
+
+```ruby
+keypair = PQCrypto::Signature.generate(:ml_dsa_65)
+
+pem = keypair.secret_key.to_pkcs8_pem
+imported = PQCrypto::Signature.secret_key_from_pkcs8_pem(pem)
+```
+
+ML-DSA seed/both import is intentionally opt-in:
 
 ```ruby
+PQCrypto::PKCS8.allow_ml_dsa_seed_format = true
+imported = PQCrypto::Signature.secret_key_from_pkcs8_pem(pem)
+```
+
+Seed/both export from an existing ML-DSA `SecretKey` is intentionally not
+available because the object does not retain the original seed material. When
+you explicitly have seed material, call `PQCrypto::PKCS8.encode_der` /
+`encode_pem` directly.
+
+## 9. pq_crypto-local container serialization
+
+The `pqc_container_*` APIs are retained for backward compatibility with older
+`pq_crypto` releases.
+
+```ruby
+keypair = PQCrypto::KEM.generate(:ml_kem_768)
+
 der = keypair.public_key.to_pqc_container_der
 imported = PQCrypto::KEM.public_key_from_pqc_container_der(der)
 ```
 
-`pqc_container_*` formats are pq_crypto-specific.
+PEM form:
+
+```ruby
+pem = keypair.secret_key.to_pqc_container_pem
+imported = PQCrypto::KEM.secret_key_from_pqc_container_pem(pem)
+```
+
+Important caveats:
+
+- `pqc_container_*` is project-local.
+- It is not ASN.1 SPKI or PKCS#8.
+- It is not advertised as interoperable with external PKI tooling.
+- It remains limited to the original algorithms:
+  - `:ml_kem_768`
+  - `:ml_dsa_65`
+  - `:ml_kem_768_x25519_xwing`
+
+For external interoperability, prefer SPKI for public keys and PKCS#8 for
+private keys.
+
+## 10. Raw key import/export
+
+Raw bytes can be useful when integrating with storage or protocols that already
+handle framing.
+
+```ruby
+keypair = PQCrypto::KEM.generate(:ml_kem_768)
+
+public_bytes = keypair.public_key.to_bytes
+secret_bytes = keypair.secret_key.to_bytes
+
+public_key = PQCrypto::KEM.public_key_from_bytes(:ml_kem_768, public_bytes)
+secret_key = PQCrypto::KEM.secret_key_from_bytes(:ml_kem_768, secret_bytes)
+```
+
+Signature keys have the same shape:
+
+```ruby
+keypair = PQCrypto::Signature.generate(:ml_dsa_65)
+
+public_key = PQCrypto::Signature.public_key_from_bytes(:ml_dsa_65, keypair.public_key.to_bytes)
+secret_key = PQCrypto::Signature.secret_key_from_bytes(:ml_dsa_65, keypair.secret_key.to_bytes)
+```
+
+## 11. Secure wiping
+
+`PQCrypto.secure_wipe(str)` zeroes a mutable Ruby string in place.
+
+```ruby
+raw = File.binread("secret-key.bin")
+key = PQCrypto::KEM.secret_key_from_bytes(:ml_kem_768, raw)
+
+PQCrypto.secure_wipe(raw)
+key.wipe!
+```
+
+`wipe!` is best-effort only. It clears the current Ruby string buffer owned by
+the key object. It cannot erase every copy that may have been created by Ruby,
+OpenSSL, serialization, logging, or the garbage collector.
+
+## 12. Equality and inspection
 
-## 8. Inspect supported algorithms
+Key equality uses constant-time comparison through OpenSSL `CRYPTO_memcmp` via
+`PQCrypto.ct_equals`.
 
 ```ruby
-PQCrypto.supported_kems           # => [:ml_kem_768]
-PQCrypto.supported_hybrid_kems    # => [:ml_kem_768_x25519_xwing]
-PQCrypto.supported_signatures     # => [:ml_dsa_65]
+keypair.public_key == imported_public_key
+keypair.secret_key == imported_secret_key
 ```
 
-## 9. Practical notes
+Secret key `inspect` output is intentionally redacted, and secret key objects
+do not expose a public fingerprint method.
+
+## 13. Native backend
+
+Since `0.5.0`, the build uses PQ Code Package `mlkem-native` / `mldsa-native`
+sources only. There is no PQClean fallback and no separate
+`PQCRYPTO_KECCAK_BACKEND` switch: Keccak/SHAKE comes from the selected PQ Code
+Package native source tree.
+
+```bash
+bundle exec rake vendor
+bundle exec rake compile
+```
+
+To try the upstream native assembly backend, opt in explicitly:
+
+```bash
+PQCRYPTO_NATIVE_ASM=1 bundle exec rake compile
+```
 
-- OpenSSL 3.0+ with SHA3-256 is required.
-- `PQCrypto::Testing` exposes deterministic helpers only for
-  regression tests.
-- Key equality uses constant-time comparison. `#hash` returns a
-  hash derived from a SHA-256 fingerprint, not the raw bytes.
+## 14. Async / Fiber scheduler behavior
+
+On Ruby 3.4, signing and verification use Ruby's scheduler-aware
+`rb_nogvl(..., RB_NOGVL_OFFLOAD_SAFE)` path automatically. With a scheduler
+that implements `blocking_operation_wait`, blocking native work can be moved
+off the event loop.
+
+## 15. Test-only deterministic helpers
+
+`PQCrypto::Testing` exposes deterministic helpers for regression tests:
+
+```ruby
+PQCrypto::Testing.ml_kem_keypair_from_seed(seed)       # 64-byte d||z seed
+PQCrypto::Testing.ml_kem_encapsulate_from_seed(pk, seed) # 32-byte seed
+PQCrypto::Testing.ml_dsa_keypair_from_seed(seed)       # 32-byte seed
+PQCrypto::Testing.ml_dsa_sign_from_seed(message, sk, seed)
+```
+
+These helpers are intended for tests only. They drive deterministic PQ Code
+Package native entrypoints and are not part of the normal application API.
+
+## 16. Development commands
+
+Run the test suite:
+
+```bash
+bundle exec rake test
+```
+
+Refresh the pinned PQ Code Package native vendor snapshot only when intentionally
+updating vendored sources. The vendoring script keeps a minimal, source-gem-safe
+snapshot: `mlkem-native/mlkem`, `mldsa-native/mldsa`, and upstream license/docs
+only. It intentionally omits examples and symlink-heavy upstream trees:
+
+```bash
+bundle exec ruby script/vendor_libs.rb
+```
+
+To intentionally change the upstream snapshot, override the native package refs:
+
+```bash
+MLKEM_NATIVE_REF=<tag-or-commit> \
+MLDSA_NATIVE_REF=<tag-or-commit> \
+  bundle exec ruby script/vendor_libs.rb
+```