belfort-ml-client 0.2.0__tar.gz

belfort_ml_client-0.2.0/.gitignore

@@ -0,0 +1,51 @@
+.vscode/*
+!.vscode/extensions.json
+node_modules/
+
+# The Cyclops client bundle build_client_wheel.sh adds to a wheel; never
+# committed, and kept out of the sdist a local build makes.
+native/belfort_native/cyclops_client/
+
+# Python, for every workspace member
+__pycache__/
+*.py[cod]
+*.egg-info/
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+.coverage
+htmlcov/
+.venv/
+
+gateway/backend/app/frontend/
+/test-results/
+/playwright-report/
+/blob-report/
+/playwright/.cache/
+.DS_Store
+/.gocache
+/perseus-compiler/output
+/cyclops-runtime/.gocache
+
+# test.py and scripts/sync_examples.py write here
+/build/
+/.toolchain/
+
+# uv build and scripts/build_client_wheel.sh write here
+/dist/
+
+# OpenTofu: keep provider locks, input examples, and reviewed environment inputs.
+# State remains in S3; local plans and input snapshots stay out of Git.
+**/.terraform/
+*.tfstate
+*.tfstate.*
+*.tfplan
+*.tfvars
+!example.tfvars
+!/infra/perseus-environment/staging.tfvars
+!/infra/perseus-environment/main.tfvars
+!/infra/gateway/staging.tfvars
+!/infra/gateway/main.tfvars
+*.tfvars.json
+crash.log
+crash.*.log

belfort_ml_client-0.2.0/PKG-INFO

@@ -0,0 +1,7 @@
+Metadata-Version: 2.4
+Name: belfort-ml-client
+Version: 0.2.0
+Summary: The Belfort ML client
+Requires-Python: >=3.12
+Requires-Dist: belfort-ml-bucket==0.2.0
+Requires-Dist: httpx==0.28.1

belfort_ml_client-0.2.0/README.md

@@ -0,0 +1,128 @@
+# Belfort ML client
+
+The end client: the **only** party that holds the secret key.
+
+Its provider mints a client token (`model.create_client(label)` in
+`belfort-ml`) and hands it over on its own channel. With that token the client
+talks to the gateway itself. Only the `/client/*` routes accept the token, and
+it names one key set of one model.
+
+```bash
+belfort-ml-client infer --token "$BELFORT_CLIENT_TOKEN" --input input.json
+```
+
+`input.json` is `{"values": [...]}`, or `{"inputs": [[...], ...]}` for an entry
+that takes several arguments, each flattened row-major. Only one of them may be
+encrypted: the gateway carries one ciphertext each way. The decrypted answer
+goes to stdout as JSON:
+
+```json
+{"values": [...], "request_id": "req_...", "latency_ms": 847, "encrypt_ms": 12, "decrypt_ms": 3}
+```
+
+`latency_ms` is the time the gateway's call to the GPU runtime took: the
+ciphertext transfers between the gateway and the runtime, the wait in the
+runtime, and the GPU computation. It does not include the transfers between
+this client and the gateway, or the gateway's own work. A replayed request
+reports the time of its first run. `encrypt_ms` and `decrypt_ms` are the time
+this client took to encrypt the input and to decrypt the answer.
+
+| Setting | Flag | Environment | Default |
+|---|---|---|---|
+| Token | `--token` | `BELFORT_CLIENT_TOKEN` | required |
+| Linked code | `--cache-dir` | `BELFORT_CLIENT_CACHE` | `~/.cache/belfort-ml-client` |
+| Wait for a deploy | `--timeout` | | 3600 s |
+| Secret seed (64 hex characters) | | `BELFORT_CLIENT_SECRET_SEED` | none: release at exit |
+
+## What one run does
+
+```mermaid
+sequenceDiagram
+    participant C as belfort-ml-client
+    participant G as Gateway
+    participant S as Storage
+    C->>G: GET /client/code
+    G-->>C: presigned GETs, one per artifact, and their digest
+    C->>S: download the artifacts
+    Note over C: link the client binary, run it: keygen
+    C->>G: POST /client/keys
+    G-->>C: presigned POSTs, one per evaluation key, bound to its size
+    C->>S: upload the evaluation keys
+    C->>G: POST /client/keys/verify
+    loop until a deploy holds the keys
+        C->>G: GET /client/status
+    end
+    C->>G: POST /client/infer (ciphertext)
+    G-->>C: encrypted answer
+    Note over C: decrypt
+    opt without BELFORT_CLIENT_SECRET_SEED
+        C->>G: DELETE /client
+    end
+```
+
+1. `GET /client/code` names the compile the token was opened for, one presigned
+   GET per artifact, and the digest the gateway verified. The client downloads
+   each into a fresh directory, refuses a name that would land outside it,
+   checks the tree against the digest and links `<model>_client`. The linked
+   code is cached per model and compile under the cache directory.
+2. The binary generates the key pair before its first command and keeps the
+   secret key in its own memory. `POST /client/keys` names the evaluation keys
+   with their sizes and returns one presigned POST each, which S3 accepts only
+   at that size; `POST /client/keys/verify` checks they all arrived.
+3. `GET /client/status` reports `awaiting_keys`, `awaiting_upload`,
+   `no_deploy`, `loading`, `ready` or `failed`. Once the keys are
+   verified, the gateway loads them onto the deploy the provider started for
+   this key set. `no_deploy` keeps the client waiting, since the provider may
+   start the deploy after handing over the token.
+4. `POST /client/infer` carries one ciphertext each way.
+
+A token opens one key set, and its keys can be named once. The secret key is
+discarded when the process exits, so each run needs a new token from the
+provider, unless every run sets the same `BELFORT_CLIENT_SECRET_SEED`.
+
+From Python, `belfort_ml_client.Client` does the same in steps:
+
+```python
+from belfort_ml_client import Client
+
+with Client(token) as client:
+    client.prepare()  # code, keys, a deploy holding them
+    answer = client.infer([values])  # as many as needed in this process
+```
+
+## The surface
+
+| File | What it is |
+|---|---|
+| `client.py` | `Client`: code, keys, status and inference against the gateway |
+| `cli.py` | `belfort-ml-client infer` |
+| `build.py` | Linking the downloaded C++ into `<model>_client`, through [`belfort_native`](../native) |
+| `belfort_ml_client.cc` | The client's role over one model's generated facade: keygen, encrypt, decrypt |
+
+## The link
+
+The client links `<model>_client` through this package's `build.py`, which names
+`belfort_ml_client.cc` and hands it to
+[`belfort_native`](../native). Linking needs clang++ and a Cyclops client
+build.
+
+The binary links `libcyclops_client.so` (`.dylib` on macOS) and runs without
+CUDA. The platform wheels for Linux x86-64 and aarch64 (manylinux_2_28) and
+macOS arm64 carry the library, its headers and a `REVISION` file under
+`belfort_native/cyclops_client`, with libtommath linked in statically; the link
+uses that prefix when `PERSEUS_CYCLOPS_DIR` is unset. `PERSEUS_CYCLOPS_DIR`
+overrides it, and is required with the sdist: the link looks for the library
+under `PERSEUS_CYCLOPS_DIR/lib`, `client/lib` or `client/build`.
+`uv run --script scripts/fetch_cyclops.py client .toolchain/cyclops-client`
+fetches such a prefix from the Cyclops release at the pinned commit.
+[`scripts/build_client_wheel.sh`](../scripts/build_client_wheel.sh) builds the
+platform wheel on the machine it runs on, and the
+[Build Cyclops client wheels](../.github/workflows/build-cyclops-client.yml)
+workflow builds and tests all three; the release workflow publishes them.
+
+The generated client creates the evaluation keys requested by HEIR.
+
+## Tests
+
+The [end-to-end suite](../tests/e2e/README.md) tests the Belfort ML client with the
+real gateway, compiler, and GPU runtime. See its setup and test commands.

belfort_ml_client-0.2.0/belfort_ml_client/__init__.py

@@ -0,0 +1,9 @@
+from __future__ import annotations
+
+from importlib.metadata import version
+
+__version__ = version("belfort-ml-client")
+
+from belfort_ml_client.client import Client, ClientError
+
+__all__ = ["Client", "ClientError", "__version__"]

belfort_ml_client-0.2.0/belfort_ml_client/belfort_ml_client.cc

@@ -0,0 +1,244 @@
+#include <algorithm>
+#include <cmath>
+#include <cstddef>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <filesystem>
+#include <fstream>
+#include <iostream>
+#include <memory>
+#include <nlohmann/json.hpp>
+#include <optional>
+#include <string>
+#include <tuple>
+#include <utility>
+#include <vector>
+
+#include "common_roles.h"
+
+namespace perseus {
+namespace {
+
+inline constexpr std::size_t kInputCount =
+    std::tuple_size_v<model::CleartextInputs>;
+
+// Read blob `Argument` of `example` into its slot of the cleartext tuple.
+template <std::size_t Argument>
+void read_input(model::CleartextInputs& inputs, const std::string& resource_dir,
+                int example) {
+  using Slot = std::tuple_element_t<Argument, model::CleartextInputs>;
+  const std::vector<double> values =
+      read_f64(input_path(resource_dir, example, Argument, kInputCount),
+               Nest<Slot>::count);
+  Nest<Slot>::fill(std::get<Argument>(inputs), values.data());
+}
+
+template <std::size_t... Argument>
+void read_inputs(model::CleartextInputs& inputs, const std::string& resource_dir,
+                 int example, std::index_sequence<Argument...>) {
+  (read_input<Argument>(inputs, resource_dir, example), ...);
+}
+
+using SecretSeed = ::cyclops::prng::Seed;
+
+// The seed is the secret key, so it arrives on stdin, never on argv or disk:
+// one line holding a JSON array of its bytes.
+SecretSeed read_seed() {
+  std::string line;
+  std::getline(std::cin, line);
+  SecretSeed seed;
+  const auto bytes = nlohmann::json::parse(line, nullptr, false);
+  const bool valid =
+      bytes.is_array() && bytes.size() == seed.bytes.size() &&
+      std::all_of(bytes.begin(), bytes.end(), [](const nlohmann::json& byte) {
+        return byte.is_number_unsigned() &&
+               byte.get<nlohmann::json::number_unsigned_t>() <= 0xff;
+      });
+  if (!valid) {
+    std::fprintf(stderr, "expected a JSON array of %zu secret seed bytes\n",
+                 seed.bytes.size());
+    std::exit(1);
+  }
+  bytes.get_to(seed.bytes);
+  return seed;
+}
+
+}  // namespace
+
+void export_keys(model::Context& context,
+                 const ::cyclops::EvkMap<model::word>& keys,
+                 const std::string& dir) {
+  using Keys = ::cyclops::EvkMap<model::word>;
+  std::filesystem::create_directories(dir);
+  std::size_t n = 0;
+  for (const auto& [index, evk] : keys) {
+    // The sparse-to-dense key has its own, smaller wire form.
+    if (index.idx == Keys::kSparseToDenseKeyIndex) {
+      write_file(blob_path(dir, n++, "s2d"),
+                 model::SendSparseToDenseKey(kWireSerializer, context.param_,
+                                             evk));
+    } else {
+      write_file(blob_path(dir, n++, "evk"),
+                 model::SendEvaluationKey(kWireSerializer, context.param_, evk,
+                                          index.idx));
+    }
+  }
+}
+
+void encrypt_example_to(model::Context& context, model::SecretKey key,
+                        const std::string& resource_dir, int example,
+                        const std::string& dir) {
+  model::CleartextInputs inputs{};
+  read_inputs(inputs, resource_dir, example,
+              std::make_index_sequence<kInputCount>{});
+  auto encrypted = model::Encrypt(context, key, inputs);
+  std::size_t n = 0;
+  auto write = [&](const auto& ciphertext) {
+    write_file(blob_path(dir, n++, "ct"),
+               model::SendCiphertext(kWireSerializer, context.param_,
+                                     ciphertext));
+  };
+  heir::cyclops::forEachLeaf(encrypted, write);
+}
+
+std::vector<double> decrypt_result_from(model::Context& context,
+                                        model::SecretKey key,
+                                        const std::string& dir) {
+  model::EncryptedOutputs outputs{};
+  std::size_t n = 0;
+  auto read = [&](auto& ciphertext) {
+    // The file name fixes the order, so only the ciphertext is kept.
+    ciphertext =
+        model::ReceiveCiphertext(context.param_,
+                                 read_file(blob_path(dir, n++, "ct")))
+            .ct;
+  };
+  heir::cyclops::forEachLeaf(outputs, read);
+  // `Decrypt` returns one output's nest, or a tuple of them; `Nest` handles both.
+  auto decrypted = model::Decrypt(context, key, outputs);
+  std::vector<double> values;
+  values.reserve(Nest<decltype(decrypted)>::count);
+  Nest<decltype(decrypted)>::collect(decrypted, values);
+  return values;
+}
+
+// Debug ciphertexts leave the server encrypted; only this process decrypts
+// them.
+nlohmann::json decrypt_checkpoints(model::Context& context,
+                                   model::SecretKey key,
+                                   const std::string& path) {
+  std::ifstream in(path, std::ios::binary);
+  cereal::PortableBinaryInputArchive archive(in);
+  auto records = nlohmann::json::array();
+  while (in.peek() != std::char_traits<char>::eof()) {
+    std::string name, metadata;
+    std::uint32_t count;
+    archive(name, metadata, count);
+    std::vector<double> values;
+    double scale = 0;
+    int level = 0;
+    for (std::uint32_t i = 0; i < count; ++i) {
+      std::string bytes;
+      archive(bytes);
+      auto received = model::ReceiveCiphertext(context.param_, bytes);
+      if (i == 0) {
+        scale = std::log2(received.ct.GetScale());
+        level = received.ct.GetNP().num_main_;
+      }
+      if (values.size() >= 256) continue;
+      heir::cyclops::Plaintext<model::word> plaintext;
+      key->Decrypt(plaintext, received.ct);
+      std::vector<double> decoded, imaginary;
+      context.encoder_.DecodeSlots(decoded, imaginary, plaintext);
+      values.insert(
+          values.end(), decoded.begin(),
+          decoded.begin() + std::min(256 - values.size(), decoded.size()));
+    }
+    records.push_back({{"name", name},
+                       {"is_input", name.find("/input/") != std::string::npos},
+                       {"scale", scale},
+                       {"level", level},
+                       {"values", values}});
+  }
+  return records;
+}
+
+int client_main(const std::shared_ptr<model::Context>& context,
+                const std::string& resource_dir,
+                const std::optional<SecretSeed>& secret_seed, Timings timings,
+                Clock::time_point started) {
+  model::KeyPair keys;
+  timings.key_setup_ms =
+      timed("keygen", [&] { keys = model::KeyGen(context, secret_seed); });
+  export_keys(*context, keys.storage->GetEvkMap(), resource_dir + "/keys");
+  timings.harness_total_ms = ms_since(started);
+  write_report({}, timings);
+
+  std::string command;
+  while (std::cin >> command) {
+    const Clock::time_point request = Clock::now();
+    Timings t;
+    std::vector<std::vector<double>> results;
+    if (command == "debug") {
+      std::string path;
+      std::cin >> path;
+      nlohmann::json report = {
+          {"debug_records",
+           decrypt_checkpoints(*context, keys.secret_key, path)}};
+      std::cout << report.dump() << std::endl;
+      continue;
+    } else if (command == "secret_seed") {
+      nlohmann::json report = {
+          {"secret_seed", keys.storage->GetSecretSeed().bytes}};
+      std::cout << report.dump() << std::endl;
+      continue;
+    } else if (command == "encrypt") {
+      int example = 0;
+      std::string dir;
+      std::cin >> example >> dir;
+      t.encrypt_ms = timed("encrypt", [&] {
+        encrypt_example_to(*context, keys.secret_key, resource_dir, example, dir);
+      });
+    } else if (command == "decrypt") {
+      std::string dir;
+      std::cin >> dir;
+      t.decrypt_ms = timed("decrypt", [&] {
+        results.push_back(decrypt_result_from(*context, keys.secret_key, dir));
+      });
+    } else {
+      unknown_command(command);
+    }
+    t.harness_total_ms = ms_since(request);
+    write_report(results, t);
+  }
+  return 0;
+}
+
+}  // namespace perseus
+
+int main(int argc, char** argv) {
+  const std::string resource_dir = argc > 1 ? argv[1] : ".";
+  // --seed-from-stdin: the first stdin line is the hex secret seed to restore.
+  const bool seeded = argc > 3 && std::strcmp(argv[3], "--seed-from-stdin") == 0;
+  if ((argc > 2 && std::strcmp(argv[2], "--client") != 0) ||
+      (argc > 3 && !seeded) || argc > 4) {
+    std::fprintf(stderr, "usage: %s <dir> [--client [--seed-from-stdin]]\n",
+                 argv[0]);
+    return 1;
+  }
+
+  namespace model = perseus::model;
+  const perseus::Clock::time_point started = perseus::Clock::now();
+  perseus::Timings timings;
+  timings.cuda_init_ms = perseus::device_init();
+
+  std::shared_ptr<model::Context> context;
+  timings.setup_ms = perseus::timed("setup", [&] { context = model::Setup(); });
+
+  std::optional<perseus::SecretSeed> secret_seed;
+  if (seeded) secret_seed = perseus::read_seed();
+  return perseus::client_main(context, resource_dir, secret_seed, timings,
+                              started);
+}

belfort_ml_client-0.2.0/belfort_ml_client/build.py

@@ -0,0 +1,13 @@
+"""Link a downloaded compile and the client's drivers into `<model>_client`."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from belfort_native import link
+
+DRIVER = Path(__file__).with_name("belfort_ml_client.cc")
+
+
+def build_client(model_dir: Path, runtime: str) -> Path:
+    return link(model_dir, runtime, DRIVER, "_client", side="client")

belfort_ml_client-0.2.0/belfort_ml_client/cli.py

@@ -0,0 +1,118 @@
+"""`belfort-ml-client infer`: one inference on the token a provider handed over.
+
+    belfort-ml-client infer --token "$BELFORT_CLIENT_TOKEN" --input input.json
+
+`input.json` holds `{"values": [...]}` for a single-argument model, or
+`{"inputs": [[...], ...]}` for several arguments, each flattened row-major. The
+decrypted answer goes to stdout as JSON; progress goes to stderr.
+
+With BELFORT_CLIENT_SECRET_SEED set to a 32-byte secret seed (64 hex
+characters), the keys derive from it, a later run with the same seed resumes the
+key set, and the run keeps the key set at exit. Without it, the run releases the
+key set at exit, which ends the token.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+import httpx
+
+from belfort_ml_client.client import Client, ClientError
+
+SEED_ENV = "BELFORT_CLIENT_SECRET_SEED"
+log = logging.getLogger("belfort_ml_client")
+
+
+def read_arguments(payload: Any) -> list[list[float]]:
+    if isinstance(payload, dict) and "inputs" in payload:
+        arguments = payload["inputs"]
+    elif isinstance(payload, dict) and "values" in payload:
+        arguments = [payload["values"]]
+    else:
+        raise ClientError('expected {"values": [...]} or {"inputs": [[...], ...]}')
+    if not isinstance(arguments, list) or not arguments:
+        raise ClientError("no input values")
+    if not all(isinstance(argument, list) and argument for argument in arguments):
+        raise ClientError("each argument must be a non-empty list of numbers")
+    numbers = [value for argument in arguments for value in argument]
+    # bool is an int; neither it nor a string is an input value.
+    if not all(
+        isinstance(value, int | float) and not isinstance(value, bool)
+        for value in numbers
+    ):
+        raise ClientError("each argument must be a list of numbers")
+    return [[float(value) for value in argument] for argument in arguments]
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="belfort-ml-client")
+    commands = parser.add_subparsers(dest="command", required=True)
+    infer = commands.add_parser("infer", help="Run one encrypted inference.")
+    infer.add_argument("--token", help="The client token (or BELFORT_CLIENT_TOKEN).")
+    infer.add_argument(
+        "--cache-dir", type=Path, help="Where linked client code is kept."
+    )
+    infer.add_argument(
+        "--input", required=True, help="A JSON file with the input, or - for stdin."
+    )
+    infer.add_argument(
+        "--timeout",
+        type=float,
+        default=3600,
+        help="Seconds to wait for a deploy to load this client's keys.",
+    )
+    args = parser.parse_args(argv)
+
+    logging.basicConfig(
+        level=logging.INFO, format="%(asctime)s %(message)s", stream=sys.stderr
+    )
+    # httpx logs every request URL at INFO, and a presigned URL is a credential.
+    logging.getLogger("httpx").setLevel(logging.WARNING)
+    # Keep stdout for the JSON answer: the link step and its compilers write to
+    # fd 1, so point fd 1 at stderr until then.
+    answer_out = os.fdopen(os.dup(sys.stdout.fileno()), "w")
+    sys.stdout.flush()
+    os.dup2(sys.stderr.fileno(), sys.stdout.fileno())
+    try:
+        text = sys.stdin.read() if args.input == "-" else Path(args.input).read_text()
+        arguments = read_arguments(json.loads(text))
+        seed = os.environ.get(SEED_ENV)
+        secret_seed = bytes.fromhex(seed) if seed else None
+        with Client(
+            args.token, secret_seed=secret_seed, cache_dir=args.cache_dir
+        ) as client:
+            try:
+                client.prepare(timeout=args.timeout)
+                answer = client.infer(arguments)
+            finally:
+                # Nobody can resume a key set whose seed is gone.
+                if secret_seed is None and client.harness is not None:
+                    try:
+                        client.release()
+                    except (ClientError, httpx.HTTPError) as exc:
+                        log.warning("could not release this client: %s", exc)
+    # The link step raises RuntimeError; a gateway unreachable after the
+    # retries raises httpx.TransportError.
+    except (ClientError, OSError, ValueError, RuntimeError, httpx.HTTPError) as exc:
+        print(f"belfort-ml-client: {exc}", file=sys.stderr)
+        return 1
+    values = answer["values"]
+    value = values[0] if len(values) == 1 else values
+    print(
+        f"The encrypted computation was a success. The result is {value}",
+        file=sys.stderr,
+    )
+    with answer_out:
+        print(json.dumps(answer), file=answer_out)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())