async-matrix 3.0.0 → 4.0.0

data/lib/async/matrix/e2ee/schema/readme.md

@@ -0,0 +1,95 @@
+# Matrix end-to-end encryption specification
+
+The official specification material for end-to-end encryption, vendored so that the flow can be read beside the code which implements it.
+
+This is **input you read, not output you produce**: it is a copy of upstream, and the way to change it is to re-vendor at a newer commit.
+
+## Provenance
+
+| | |
+| --- | --- |
+| Source | https://github.com/matrix-org/matrix-spec |
+| Commit | `a484823f68d29671a968410d8ad98bb00c554af7` |
+| Dated | 2026-10-09 |
+| Vendored | 2026-10-11 |
+| Licence | Apache License 2.0, the same as this project |
+
+Copyright belongs to The Matrix.org Foundation C.I.C. and the contributors to the specification.
+
+Upstream paths are preserved, so any file here is `<repository>/<path>` at that commit.
+
+## Why these files
+
+`MATRIX_API_VERSIONS.txt` and the generated `schemas/` describe the *surface* of each endpoint, version by version. They do not describe the protocol a client must run to use that surface: which event unlocks which other event, what must be validated, and what must be remembered. These files do.
+
+The `x-addedInMatrixVersion` and `x-changedInMatrixVersion` annotations in the schemas, and the `added-in` and `changed-in` markers in the prose, say which version layer a behaviour belongs in.
+
+## Layout
+
+### Prose
+
+| File | What it answers |
+| --- | --- |
+| `content/client-server-api/modules/end_to_end_encryption.md` | the whole flow: key distribution, device verification, cross-signing, key sharing, backups, history sharing, both messaging algorithms, and the withheld report |
+| `content/client-server-api/modules/secrets.md` | secret storage and secret sharing, which carry the cross-signing and backup private keys |
+| `content/client-server-api/modules/send_to_device.md` | the transport every key-bearing event travels over |
+| `content/client-server-api/modules/device_management.md` | the device list the keys hang from |
+| `content/olm-megolm/olm.md` | the Olm double ratchet: the pairwise channel, and the pre-key and normal message formats |
+| `content/olm-megolm/megolm.md` | the Megolm hash ratchet: the group session, and its message format |
+| `content/olm-megolm/_index.md` | how the two ratchets relate |
+| `content/appendices.md` | canonical JSON, signing JSON, unpadded Base64 and the cryptographic key representation |
+
+### Events
+
+Under `data/event-schemas/schema/`, one file per event, plus the `core-event-schema/` and `components/` they reference. Every reference resolves within this directory.
+
+The timeline and state events:
+
+- `m.room.encryption.yaml` — the state event which turns a room encrypted, and carries the rotation policy.
+- `m.room.encrypted.yaml` — the ciphertext envelope, used for both algorithms and on both transports.
+- `m.room.message$m.key.verification.request.yaml` — a verification request made in a room rather than to a device.
+
+The key-bearing to-device events:
+
+- `m.room_key.yaml` — the Megolm session, shared over Olm.
+- `m.room_key.withheld.yaml` — the report that a session was deliberately not shared, and the `code` values it can carry.
+- `m.room_key_request.yaml` — asking another device for a session, and cancelling the ask.
+- `m.forwarded_room_key.yaml` — answering that ask, with the forwarding chain.
+- `m.dummy.yaml` — announcing a replacement Olm session after one is found corrupt.
+- `m.room_key_bundle.yaml` — the bundle of sessions shared with a newly invited user.
+
+The verification events, which share one framework and add their own steps:
+
+- `m.key.verification.request.yaml`, `.ready.yaml`, `.start.yaml`, `.accept.yaml`, `.key.yaml`, `.mac.yaml`, `.done.yaml`, `.cancel.yaml`
+- `m.key.verification.start$m.sas.v1.yaml` — the short authentication string method.
+- `m.key.verification.start$m.reciprocate.v1.yaml` — the QR code method.
+- `m.key.verification.m.relates_to.yaml` — the framing which carries a verification in a room.
+
+The secrets and backup events:
+
+- `m.secret.request.yaml` and `m.secret.send.yaml` — asking one of our own verified devices for a named secret, and the answer.
+- `m.key_backup.yaml` — the account data recording whether the user wants key backup.
+
+### Endpoints and definitions
+
+Under `data/api/client-server/`:
+
+- `keys.yaml` — `/keys/upload`, `/keys/query`, `/keys/claim` and `/keys/changes`.
+- `cross_signing.yaml` — `/keys/device_signing/upload` and `/keys/signatures/upload`.
+- `key_backup.yaml` — the `/room_keys` endpoints.
+- `to_device.yaml` — `/sendToDevice`.
+- `device_management.yaml` — `/devices`.
+
+Under `data/api/client-server/definitions/`, the structures those endpoints and events are built from. The ones which matter most to a reader:
+
+- `olm_plaintext.yaml` — what an Olm-encrypted event decrypts to, and so the fields which must be validated before it is believed.
+- `device_keys.yaml` and `cross_signing_key.yaml` — the signed identities.
+- `one_time_keys.yaml` — the keys which establish an Olm session.
+- `room_key_bundle.yaml` — the bundle shared on invite.
+- `encrypted_file.yaml` — how an attachment, and a key bundle, is encrypted.
+- `key_backup_auth_data.yaml`, `key_backup_session_data.yaml`, `key_backup_data.yaml`, `room_key_backup.yaml` — the backup.
+- `megolm_export_session_data.yaml` — the manual export, which carries the `shared_history` flag too.
+
+### Data
+
+- `data-definitions/sas-emoji.json` — the 64 emoji a short authentication string is drawn from, with their descriptions.

data/lib/async/matrix/e2ee/secret_storage.rb

@@ -0,0 +1,214 @@
+# frozen_string_literal: true
+
+# Released under the Apache License, Version 2.0.
+# Copyright, 2026, by General Intelligence Systems.
+
+require "base64"
+require "openssl"
+require "securerandom"
+
+require_relative "../key_representation"
+
+module Async
+  module Matrix
+    module E2EE
+      # Secrets kept on the server, under a key the server never sees.
+      #
+      # This is where the cross signing keys and the backup key live, so that a new device can reach them.
+      module SecretStorage
+        # The only algorithm the specification defines.
+        ALGORITHM = "m.secret_storage.v1.aes-hmac-sha2"
+
+        # The only way of deriving one from a passphrase.
+        PBKDF2 = "m.pbkdf2"
+
+        # What a key is, and what a passphrase derives to.
+        BYTES = 32
+
+        # The salt the derivation uses, which is fixed at nothing.
+        ZERO_SALT = ("\x00" * 32).b
+
+        # What the check value encrypts, which is nothing in particular.
+        ZERO_MESSAGE = ("\x00" * 32).b
+
+        # The key secret storage is encrypted under.
+        #
+        # It never leaves the device: what goes to the server is what it encrypted, and a description which only says how to arrive at it again.
+        class Key
+          # @returns [Key] a key of random bytes, which the user keeps.
+          def self.generate
+            self.new(SecureRandom.bytes(BYTES))
+          end
+
+          # @parameter bytes [String] the raw key.
+          # @returns [Key] that key.
+          def self.from_bytes(bytes)
+            self.new(bytes.b)
+          end
+
+          # The key a passphrase arrives at, which is the same every time for the same description.
+          #
+          # @parameter passphrase [String] what the user typed.
+          # @parameter salt [String] the salt from the key's description.
+          # @parameter iterations [Integer] how many rounds the description asks for.
+          # @parameter bits [Integer] how many bits to derive, which means 256 where the description says nothing.
+          # @returns [Key] the key it derives to.
+          def self.from_passphrase(passphrase, salt:, iterations:, bits: 256)
+            derived = OpenSSL::KDF.pbkdf2_hmac(
+              passphrase,
+              salt: salt,
+              iterations: iterations,
+              length: bits / 8,
+              hash: "SHA512",
+            )
+
+            self.new(derived)
+          end
+
+          # @parameter bytes [String] the raw key.
+          def initialize(bytes)
+            @bytes = bytes
+          end
+
+          # @attribute [String] the raw key, which is what is shown to a user and never sent.
+          attr :bytes
+
+          # Encrypt a secret under this key, named so that one key cannot read another's.
+          #
+          # @parameter name [String] what the secret is called, which is the account data type.
+          # @parameter plaintext [String] the secret.
+          # @returns [Hash] the counter, the ciphertext and the code over it.
+          def encrypt(name, plaintext)
+            aes, mac = self.derive(name)
+            iv = self.class.counter
+
+            ciphertext = self.cipher(:encrypt, aes, iv, plaintext)
+
+            {
+              iv: self.class.encode(iv),
+              ciphertext: self.class.encode(ciphertext),
+              mac: self.class.encode(OpenSSL::HMAC.digest("SHA256", mac, ciphertext)),
+            }
+          end
+
+          # Read a secret, if the code over it is the one this key would have written.
+          #
+          # @parameter name [String] what the secret is called.
+          # @parameter data [Hash] the counter, the ciphertext and the code.
+          # @returns [String | nil] the secret, or nil when it was not this key which wrote it.
+          def decrypt(name, data)
+            aes, mac = self.derive(name)
+            ciphertext = self.class.decode(data[:ciphertext])
+
+            expected = OpenSSL::HMAC.digest("SHA256", mac, ciphertext)
+
+            case OpenSSL.secure_compare(expected, self.class.decode(data[:mac]))
+            in false
+              nil
+            in true
+              self.cipher(:decrypt, aes, self.class.decode(data[:iv]), ciphertext)
+            end
+          end
+
+          # Whether this key is the one a description was written by.
+          #
+          # The description carries the code over a known message, so a wrong passphrase is caught before anything is decrypted with it.
+          #
+          # @parameter iv [String] the counter from the description.
+          # @parameter mac [String] the code from the description.
+          # @returns [Boolean] whether they match.
+          def correct?(iv:, mac:)
+            aes, key = self.derive("")
+
+            ciphertext = self.cipher(:encrypt, aes, self.class.decode(iv), ZERO_MESSAGE)
+            expected = OpenSSL::HMAC.digest("SHA256", key, ciphertext)
+
+            OpenSSL.secure_compare(expected, self.class.decode(mac))
+          end
+
+          # What goes into account data to describe this key, so another device can arrive at it.
+          #
+          # It carries no key material: a passphrase description says how to derive one, and the check says whether the answer was right.
+          #
+          # @parameter name [String] what to call it to the user.
+          # @parameter passphrase [Hash | nil] how it was derived, where it was.
+          # @returns [Hash] the description.
+          def description(name: nil, passphrase: nil)
+            aes, key = self.derive("")
+            iv = self.class.counter
+
+            ciphertext = self.cipher(:encrypt, aes, iv, ZERO_MESSAGE)
+
+            {
+              algorithm: ALGORITHM,
+              name: name,
+              iv: self.class.encode(iv),
+              mac: self.class.encode(OpenSSL::HMAC.digest("SHA256", key, ciphertext)),
+              passphrase: passphrase,
+            }.compact
+          end
+
+          # @returns [String] the key as it is shown to a person, which is the only place it is.
+          def to_s
+            KeyRepresentation.encode(@bytes)
+          end
+
+          def inspect
+            "\#<#{self.class}>"
+          end
+
+          # Sixteen random bytes, with the top bit of the counter cleared.
+          #
+          # The specification asks for that to work around implementations which disagree about where the counter wraps.
+          #
+          # @returns [String] the counter block.
+          def self.counter
+            SecureRandom.bytes(16).tap do |iv|
+              iv.setbyte(8, iv.getbyte(8) & 0x7f)
+            end
+          end
+
+          # @parameter bytes [String] what to encode.
+          # @returns [String] unpadded base64, which is what the specification asks for.
+          def self.encode(bytes)
+            Base64.strict_encode64(bytes).delete("=")
+          end
+
+          # Padded base64 is accepted too, because some clients write it.
+          #
+          # @parameter string [String] what to decode.
+          # @returns [String] the bytes.
+          def self.decode(string)
+            Base64.decode64(string.to_s)
+          end
+
+          private
+
+          # The two keys a name derives to: one to encrypt with, one to sign with.
+          #
+          # @parameter name [String] the name of the secret, or the empty string for the check.
+          # @returns [Array(String, String)] the AES key and the MAC key.
+          def derive(name)
+            derived = OpenSSL::KDF.hkdf(@bytes, salt: ZERO_SALT, info: name, length: 64, hash: "SHA256")
+
+            [derived.byteslice(0, 32), derived.byteslice(32, 32)]
+          end
+
+          # @parameter direction [Symbol] whether to encrypt or decrypt.
+          # @parameter key [String] the AES key.
+          # @parameter iv [String] the counter block.
+          # @parameter data [String] what to put through it.
+          # @returns [String] the result.
+          def cipher(direction, key, iv, data)
+            cipher = OpenSSL::Cipher.new("AES-256-CTR")
+            cipher.public_send(direction)
+            cipher.key = key
+            cipher.iv = iv
+
+            cipher.update(data) + cipher.final
+          end
+        end
+      end
+    end
+  end
+end

data/lib/async/matrix/e2ee/secrets.rb

@@ -0,0 +1,220 @@
+# frozen_string_literal: true
+
+# Released under the Apache License, Version 2.0.
+# Copyright, 2026, by General Intelligence Systems.
+
+require "console"
+
+require_relative "secret_storage"
+
+module Async
+  module Matrix
+    module E2EE
+      # The private keys a device holds, and what it would write them to the server as.
+      #
+      # It makes no request: a caller hands it the account data, and it answers with what to send.
+      class Secrets
+        def initialize
+          # The secrets this device has opened, which never leave it in the clear.
+          @held = {}
+
+          # The description of the account's storage, and the key which opens it.
+          @description = nil
+          @key_id = nil
+          @key = nil
+
+          # What this device asked its siblings for, so that an answer can be matched to a question.
+          @asked = {}
+        end
+
+        # @attribute [String | nil] which key the account's storage is encrypted under.
+        attr :key_id
+
+        # Note the account's storage, so that a passphrase has something to open.
+        #
+        # @parameter key_id [String] the ID of the default key.
+        # @parameter description [Hash] what `m.secret_storage.key.<id>` said.
+        def describe(key_id, description)
+          @key_id = key_id
+          @description = description
+        end
+
+        # Note a secret as the account data holds it, sealed under the storage key.
+        #
+        # @parameter name [String] what the secret is called.
+        # @parameter encrypted [Hash] the `encrypted` property, by key ID.
+        def sealed(name, encrypted)
+          (@sealed ||= {})[name] = encrypted
+        end
+
+        # Open the storage with a passphrase, and read whatever was sealed under it.
+        #
+        # @parameter passphrase [String] what the user typed.
+        # @returns [Boolean] whether it opened.
+        def unlock(passphrase)
+          case @description
+          in {passphrase: {algorithm: SecretStorage::PBKDF2, salt: String => salt, iterations: Integer => iterations, **rest}}
+            key = SecretStorage::Key.from_passphrase(passphrase, salt: salt, iterations: iterations, bits: rest.fetch(:bits, 256))
+
+            self.opened(key)
+          in nil
+            Console.warn(self, "There is no storage to open: nothing has described one.")
+
+            false
+          else
+            Console.warn(self, "This storage was not derived from a passphrase.", key_id: @key_id)
+
+            false
+          end
+        end
+
+        # Open the storage with the key itself, which is what a user typed or a sibling sent.
+        #
+        # @parameter key [SecretStorage::Key] the key.
+        # @returns [Boolean] whether it is the right one.
+        def opened(key)
+          case @description
+          in {iv: String => iv, mac: String => mac} unless key.correct?(iv: iv, mac: mac)
+            Console.warn(self, "That key does not open this storage.", key_id: @key_id)
+
+            false
+          else
+            @key = key
+
+            self.read_sealed
+
+            true
+          end
+        end
+
+        # @returns [Boolean] whether this device can read and write the account's storage.
+        def open?
+          !@key.nil?
+        end
+
+        # Take a secret this device has come by, however it came by it.
+        #
+        # @parameter name [String] what it is called.
+        # @parameter secret [String | nil] the secret.
+        def accept(name, secret)
+          case secret
+          in String => held
+            @held[name] = held
+
+            Console.debug(self, "Holding a secret.", name: name)
+          in nil
+            Console.warn(self, "Nothing to hold for a secret.", name: name)
+          end
+        end
+
+        # @parameter name [String] what it is called.
+        # @returns [String | nil] the secret, if this device holds it.
+        def held(name)
+          @held[name]
+        end
+
+        # What to write to account data to keep these secrets on the server.
+        #
+        # @parameter secrets [Hash] the secrets, by name.
+        # @returns [Hash] the account data to set, by type.
+        def store(secrets)
+          secrets.each_with_object({}) do |(name, secret), written|
+            @held[name.to_s] = secret
+
+            # Without a storage key there is nowhere to put it, so it is held for this session and no further.
+            if self.open?
+              written[name.to_s] = {encrypted: {@key_id => @key.encrypt(name.to_s, secret)}}
+            else
+              Console.warn(self, "Holding a secret which cannot be written: this account has no storage open.", name: name)
+            end
+          end
+        end
+
+        # Read a secret which was sealed under this storage, now that it is open.
+        #
+        # @parameter name [String] what it is called.
+        # @returns [String | nil] the secret, or nil where it is not there or does not read.
+        def load(name)
+          case {open: self.open?, sealed: (@sealed || {})[name]}
+          in {open: true, sealed: Hash => encrypted} if encrypted.key?(@key_id.to_sym)
+            @key.decrypt(name, encrypted[@key_id.to_sym]).tap do |secret|
+              self.accept(name, secret)
+            end
+          else
+            @held[name]
+          end
+        end
+
+        # Note that this device has asked its siblings for a secret.
+        #
+        # @parameter request_id [String] the ID of the request.
+        # @parameter name [String] what was asked for.
+        def asking(request_id, name)
+          @asked[request_id] = name
+        end
+
+        # @parameter request_id [String] the ID of the request.
+        # @returns [String | nil] what was asked for under that ID.
+        def requested(request_id)
+          @asked[request_id]
+        end
+
+        # Whether this is an answer to something we asked, over a channel which proves who sent it.
+        #
+        # A secret arriving unbidden is somebody trying their luck, so it is never taken.
+        #
+        # @parameter request_id [String] the ID the answer carries.
+        # @parameter over [Async::Matrix::Events::Envelope | nil] the event it arrived in.
+        # @returns [Boolean] whether to believe it.
+        def asked?(request_id, over: nil)
+          case {asked: @asked.key?(request_id), over: over}
+          # Nothing asked for this, so whoever sent it was not answering us.
+          in {asked: false}
+            false
+
+          # Over Olm, which is what says which device it came from.
+          in {over: Events::Decrypted}
+            true
+
+          # Nothing was said about how it arrived, so the caller has already judged that.
+          in {over: nil}
+            true
+          else
+            false
+          end
+        end
+
+        # Forget a request, because it was answered or withdrawn.
+        #
+        # @parameter request_id [String] the ID of the request.
+        def cancel(request_id)
+          @asked.delete(request_id)
+        end
+
+        # A key derived from a passphrase, for a storage this device was told about elsewhere.
+        #
+        # @parameter passphrase [String] what the user typed.
+        # @parameter salt [String] the salt from the description.
+        # @parameter iterations [Integer] the rounds from the description.
+        # @parameter bits [Integer] how many bits to derive.
+        # @returns [SecretStorage::Key] the key.
+        def from_passphrase(passphrase, salt:, iterations:, bits: 256)
+          SecretStorage::Key.from_passphrase(passphrase, salt: salt, iterations: iterations, bits: bits)
+        end
+
+        def inspect
+          "\#<#{self.class} #{@held.keys.sort.join(", ")}>"
+        end
+
+        private
+
+        # Read everything which was sealed under this storage, now that there is a key for it.
+        def read_sealed
+          (@sealed || {}).each_key do |name|
+            self.load(name)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/async/matrix/e2ee/store.rb

@@ -0,0 +1,130 @@
+# frozen_string_literal: true
+
+# Released under the Apache License, Version 2.0.
+# Copyright, 2026, by General Intelligence Systems.
+
+module Async
+  module Matrix
+    module E2EE
+      # What a device remembers between syncs, held in memory.
+      #
+      # A client which must survive a restart gives the machine its own store instead.
+      class Store
+        def initialize
+          @sessions = {}
+          @withheld = {}
+          @pending = {}
+        end
+
+        # @attribute [Hash] the sessions which were shared with us, by room and session ID.
+        attr :sessions
+
+        # @attribute [Hash] why a session was refused, by room and session ID.
+        attr :withheld
+
+        # Remember a session another device shared.
+        #
+        # Whatever else is known about it is kept too, since a backup or an export carries more than the key.
+        #
+        # @parameter room_id [String] the room the session is for.
+        # @parameter session_id [String] the ID of the session.
+        # @parameter sender [String | nil] the user who shared it.
+        # @parameter rest [Hash] what else is known: whether it was imported, what index it begins at.
+        def remember(room_id, session_id:, sender:, **rest)
+          @sessions[[room_id, session_id]] = {sender: sender, at: Time.now, **rest}
+        end
+
+        # Which device shared a session, where that was known when it arrived.
+        #
+        # @parameter session_id [String] the session.
+        # @returns [String | nil] the device ID.
+        def sender_device(session_id)
+          found = @sessions.find{|(_, held), _| held == session_id}
+
+          found&.last&.fetch(:device_id, nil)
+        end
+
+        # The earliest index a session can still be read from, which is what it exports at.
+        #
+        # @parameter room_id [String] the room.
+        # @parameter session_id [String] the session.
+        # @returns [Integer] the index.
+        def first_known_index(room_id, session_id)
+          @sessions.dig([room_id, session_id], :first_known_index) || 0
+        end
+
+        # Whether a session was imported rather than shared with us directly.
+        #
+        # @parameter session_id [String] the session.
+        # @returns [Boolean] whether it carries no signature of its own.
+        def imported?(session_id)
+          @sessions.any?{|(_, held), session| held == session_id and session[:imported]}
+        end
+
+        # When a room's own session was begun, which is what its age is measured from.
+        #
+        # @parameter room_id [String] the room.
+        # @returns [Time | nil] when it began, if a session of ours was.
+        def started(room_id)
+          @started ||= {}
+          @started[room_id]
+        end
+
+        # Note that a session of our own has begun for this room.
+        #
+        # @parameter room_id [String] the room.
+        # @parameter at [Time] when it began.
+        def starting(room_id, at: Time.now)
+          @started ||= {}
+          @started[room_id] = at
+        end
+
+        # @parameter room_id [String] the room.
+        # @parameter session_id [String] the session.
+        # @returns [Boolean] whether we hold it.
+        def session?(room_id, session_id)
+          @sessions.key?([room_id, session_id])
+        end
+
+        # Remember that a session was deliberately not shared.
+        #
+        # @parameter room_id [String] the room.
+        # @parameter session_id [String] the session.
+        # @parameter code [String] the reason the sender gave, e.g. `m.unverified`.
+        def refuse(room_id, session_id:, code:)
+          @withheld[[room_id, session_id]] = code
+        end
+
+        # @parameter room_id [String] the room.
+        # @parameter session_id [String] the session.
+        # @returns [String | nil] the reason it was refused.
+        def refused(room_id, session_id)
+          @withheld[[room_id, session_id]]
+        end
+
+        # Keep an event which cannot be read yet, so it can be read when its key arrives.
+        #
+        # @parameter room_id [String] the room.
+        # @parameter session_id [String] the session the event needs.
+        # @parameter event [Async::Matrix::Events::Envelope] the event.
+        def defer(room_id, session_id:, event:)
+          (@pending[[room_id, session_id]] ||= []) << event
+        end
+
+        # The events which were waiting for a session, which are no longer waiting.
+        #
+        # @parameter room_id [String] the room.
+        # @parameter session_id [String] the session which arrived.
+        # @returns [Array(Async::Matrix::Events::Envelope)] the events.
+        def deferred(room_id, session_id)
+          @pending.delete([room_id, session_id]) || []
+        end
+
+        # @returns [Integer] how many events are waiting for a key.
+        def deferred_count
+          @pending.each_value.sum(&:size)
+        end
+      end
+    end
+  end
+end