foretoken 0.0.2__tar.gz → 0.0.4__tar.gz

foretoken-0.0.4/PKG-INFO

@@ -0,0 +1,249 @@
+Metadata-Version: 2.4
+Name: foretoken
+Version: 0.0.4
+Summary: Command-line tools for installing and operating Foretoken
+Author: Foretoken contributors
+License-Expression: Apache-2.0
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: datasets>=2.14
+Requires-Dist: evalscope[perf]==1.11.1
+Requires-Dist: httpx>=0.27
+Requires-Dist: huggingface_hub>=0.23
+Requires-Dist: ijson>=3.2
+Requires-Dist: lm-eval[api]==0.4.13
+Requires-Dist: matplotlib>=3.7
+Requires-Dist: numpy>=1.24
+Requires-Dist: openai>=1.40
+Requires-Dist: packaging>=24.0
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: tqdm>=4.65
+Requires-Dist: transformers>=4.40
+Requires-Dist: wandb>=0.19.10
+Provides-Extra: dev
+Requires-Dist: grafana-foundation-sdk==1769699379!11.6.0; extra == "dev"
+Dynamic: license-file
+
+<!--
+SPDX-License-Identifier: Apache-2.0
+SPDX-FileCopyrightText: Copyright contributors to the Foretoken project
+-->
+
+# Foretoken command-line tool
+
+English | [简体中文](README_zh.md)
+
+The Foretoken command-line tool installs the shared Kubernetes platform, deploys model services from Kustomize configurations, reports serving readiness, resolves frontend URLs, and runs benchmarks through one `foretoken` entry point.
+
+## Before you start
+
+You need Python 3.11 or later, an active Kubernetes context, `kubectl`, and Helm. GPU nodes must already have their vendor driver and Kubernetes device plugin.
+
+## Install the command-line tool
+
+Install the published command-line tool with pip:
+
+```bash
+pip install foretoken
+
+# From a source checkout:
+# pip install -e .
+```
+
+Or create and activate a virtual environment with uv:
+
+```bash
+uv venv
+source .venv/bin/activate
+uv pip install foretoken
+```
+
+Run `foretoken --version` to check the installed CLI version.
+
+## Create a local cluster
+
+On a Linux GPU host with Docker, NVIDIA Container Toolkit, and k3d installed:
+
+```bash
+# Name the local cluster and use GPU index 0 from nvidia-smi.
+# To use two GPUs, pass --gpus 0,1.
+foretoken cluster create k3d --name foretoken-dev --gpus 0
+```
+
+For a local kind development cluster:
+
+```bash
+foretoken cluster create kind --name foretoken-dev
+```
+
+Remove a cluster created by the CLI with:
+
+```bash
+foretoken cluster delete k3d --name foretoken-dev
+```
+
+## Install the Kubernetes platform
+
+`foretoken install` installs the Foretoken CRDs and controller in the active Kubernetes context. Platform resources use the `foretoken-platform` namespace. The command also configures monitoring and, in Gateway mode, the Gateway resources. Deploy model services separately with `foretoken deploy`.
+
+### Default installation
+
+The default uses release images and local access through a `LoadBalancer` Service:
+
+```bash
+foretoken install
+```
+
+Installation selects the NVIDIA or MetaX runtime and automatically reuses or installs LeaderWorkerSet and the shared RDMA device plugin. Explicit runtime settings in `--values` take precedence; in a mixed-GPU cluster, select a resource with `runtime.vllm.gpu.resourceName` or restrict the nodes with `runtime.vllm.gpu.nodeSelector`.
+
+Log collection and persistence are enabled by default. See [Observability](../observability/README.md) for configuration, log queries, dashboards, and alerts.
+
+### Gateway mode
+
+Gateway mode creates a dedicated `GatewayClass` and `Gateway`, installing Envoy Gateway if no compatible controller is available:
+
+```bash
+foretoken install --frontend-mode gateway
+```
+
+With another Gateway Controller, reuse a Gateway managed by that controller:
+
+```bash
+foretoken install \
+  --frontend-mode gateway \
+  --gateway-name inference-gateway \
+  --gateway-namespace gateway-system
+```
+
+Add `--gateway-section-name LISTENER` only when more than one listener matches.
+
+### Current source
+
+Build and install from the repository root. The cluster needs a default StorageClass for compiler caches; see the [source deployment guide](../docs/custom-deployment.md) for storage overrides.
+
+```bash
+foretoken install -e .
+```
+
+This builds the platform in dedicated Pods and binds the checkout to the target cluster.
+
+After editing it, use `foretoken deploy` to [redeploy source changes](../docs/custom-deployment.md#deploy-and-update-code). Use `--engine-source PATH` to also bind a [vLLM engine checkout](../docs/custom-deployment.md#edit-an-inference-engine).
+
+A standard active kind or k3d context loads the built images directly into its nodes. Other Kubernetes contexts need a registry reachable by the Build Pods and nodes. For an internal registry without authentication:
+
+```bash
+foretoken install -e . --registry registry.example.com:5000/foretoken
+```
+
+If the registry requires authentication, follow the [source deployment guide](../docs/kubernetes-deployment.md) before installation.
+
+### Model distribution
+
+To share public model downloads between nodes through Dragonfly, save this in `deploy/platform-values.yaml`:
+
+```yaml
+modelDistribution:
+  dragonfly:
+    enabled: true
+```
+
+For a published platform installation, apply the values with:
+
+```bash
+foretoken install --values deploy/platform-values.yaml
+```
+
+For a source installation, run `foretoken install -e . --values deploy/platform-values.yaml` from the repository root, retaining the original registry and engine-source options.
+
+Installation prepares Dragonfly or reuses an existing installation. Models that require authentication and custom model endpoints download directly from their provider. To select a particular Dragonfly Helm release, set `existingRelease: {name: dragonfly, namespace: dragonfly-system}` under `modelDistribution.dragonfly`.
+
+On NVIDIA clusters with RDMA, ModelExpress can load weights from running replicas. Add this alongside `dragonfly` to enable it:
+
+```yaml
+modelDistribution:
+  modelexpress:
+    enabled: true
+```
+
+Automatic weight transfer uses remote models with a persistent cache, data parallelism of one, and fixed expert placement. An explicit `load-format` remains unchanged. Each GPU worker selects a nearby available RDMA interface; replicas without a compatible source load the prepared files.
+
+Reapply the installation command after changing either setting. Set `enabled: false` to disable it. `foretoken uninstall` removes managed Dragonfly resources once no workloads use them; reused installations are retained.
+
+### Installation options
+
+Use `--values` only to override platform image, runtime, or hardware settings. Without an override, installation compares supported public sources for default platform images and OCI charts. Use `--oci-registry` to select a registry explicitly; image references supplied through values remain unchanged. Source selection runs on the CLI host, so the selected registry must also be reachable from the cluster nodes.
+
+Model services are reached through an IP address outside the cluster. k3d, k3s, and cloud clusters assign one automatically. Clusters built with kubeadm, RKE2, or kubespray have no address assignment by default, so installation there ends with `LoadBalancer support Not verified`. Give Foretoken a range of unused addresses in the nodes' subnet, confirmed with the cluster administrator, and it assigns them to services:
+
+```yaml
+loadBalancer:
+  managedAddresses:
+    - 192.168.1.240-192.168.1.250
+```
+
+## Deploy and operate model services
+
+Run from the repository checkout prepared in the [Quick Start](../README.md). Deploy one frontend and all models rendered by a Kustomize root.
+
+See the [multi-model example](../examples/multi-model-quickstart/README.md) for resources and [model storage](../docs/model-storage.md) for directory or PVC configuration. Use `examples/quickstart` for a single model.
+
+```bash
+foretoken deploy examples/multi-model-quickstart --timeout 20m
+```
+
+The command applies the configuration, shows service status, and streams Pod and container logs with source prefixes while waiting. It exits when every service reports Ready and its selected alerts are configured. Without `--timeout`, it waits up to ten minutes. Configure service alerts in the Kustomize deployment; see [service observability](../examples/observability/README.md).
+
+Inspect the same deployment without applying it:
+
+```bash
+foretoken status examples/multi-model-quickstart
+```
+
+Inspect every Foretoken service in a namespace. With `--watch`, follow service state changes and Pod/container logs until Ctrl+C:
+
+```bash
+foretoken status -n foretoken-multi-model-demo
+foretoken status -n foretoken-multi-model-demo --watch
+```
+
+Resolve the public frontend URL after deployment:
+
+```bash
+FORETOKEN_FRONTEND_URL="$(foretoken endpoint examples/multi-model-quickstart)"
+```
+
+For an HTTP Gateway, resolve its request `Host` separately:
+
+```bash
+FORETOKEN_REQUEST_HOST="$(foretoken endpoint examples/multi-model-quickstart --host)"
+```
+
+`--host` returns the host and optional port for direct access, or the configured routing hostname for an HTTP Gateway. `foretoken endpoint` waits for the LoadBalancer or Gateway address; use `foretoken deploy` to wait for the services to become ready.
+
+## Measure serving performance
+
+Use `foretoken perf` to measure response latency and request or token throughput. Pass a Kustomize directory, or `--url` with `--model` for an existing endpoint. Choose a workload in [Performance examples](../benchmarks/docs/perf/README.md).
+
+## Evaluate and compare models
+
+Use `foretoken eval` to score model answers with lm-evaluation-harness or EvalScope. It accepts the same service selection options; task and scoring parameters use the selected framework's syntax. See [Quality evaluation](../benchmarks/docs/eval/README.md). Add `--reference` to [compare a candidate's probabilities against a reference](../benchmarks/docs/eval/distribution-comparison.md).
+
+## Export figures
+
+Use `--output local,wandb,plot` with a benchmark, or `foretoken plot RESULT_DIR` to redraw a saved run or sweep without running inference. Export options and comparisons are in [parameter sweeps](../benchmarks/docs/perf/sweep.md).
+
+## Find execution bottlenecks
+
+Add `--profile` to `foretoken deploy` or `foretoken perf` to record CPU/GPU execution, then browse captures with `foretoken profile view`. Setup and commands are in [Profiling](../benchmarks/docs/profile/README.md).
+
+## Clean up
+
+Delete the deployed services before uninstalling the platform:
+
+```bash
+foretoken delete examples/multi-model-quickstart
+foretoken uninstall
+```
+
+CRDs and reused cluster components are retained. Managed LeaderWorkerSet and MetalLB controllers are also retained while workloads still depend on them.

foretoken-0.0.4/README.md

@@ -0,0 +1,205 @@
+# Foretoken
+
+English | [简体中文](README_zh.md)
+
+Foretoken is a generative inference orchestration framework built for SLO/SLA targets and heterogeneous accelerators.
+
+Built on inference engines such as vLLM and SGLang, Foretoken organizes multiple generation instances into a cluster service for request routing, autoscaling, instance management, and benchmarking.
+We aim to turn an inference cluster into a token factory that continuously converts compute into tokens while meeting latency and quality requirements.
+
+## When to Use Foretoken
+
+- Serve one or more models across multiple GPUs or nodes.
+- Route requests based on load, queue depth, or KV cache state.
+- Autoscale inference instances based on traffic and SLO targets.
+- Compare aggregated serving, Prefill/Decode disaggregation, and different parallelism strategies.
+- Use the same orchestration stack across NVIDIA and MetaX accelerators.
+
+If you only need to serve a single model on one GPU, using an inference engine such as vLLM directly is usually enough.
+
+## Features and Status
+
+| Feature | Description | Status |
+|---|---|---|
+| [Evaluation](benchmarks/README.md) | Measure service performance and model quality | In development |
+| [Profiling](benchmarks/docs/profile/README.md) | Capture PyTorch, NVIDIA Nsight Systems, or MetaX mcTracer timelines for a model service | In development |
+| Hardware support | Common interfaces for device capabilities, runtimes, communication, and metrics; see [MetaX deployment](docs/metax-deployment.md) | In development |
+| Request routing | Select instances based on load, queues, KV reuse, and service levels | Research |
+| Distributed inference | Aggregated serving, Prefill/Decode disaggregation, and WideEP parallelism | Research |
+| Control plane | Model services, replica management, autoscaling, updates, and failure recovery | In development |
+| [Observability](observability/README.md) | Collect metrics and persistent service logs, evaluate alerts, and inspect the system Dashboard | In development |
+
+## Quick Start
+
+Choose the deployment path before running the common steps:
+
+| Situation | Guide |
+|---|---|
+| Single-host local deployment | [k3d deployment](docs/k3d-deployment.md) · [kind deployment](docs/kind-deployment.md) |
+| Kubernetes deployment with K3s, RKE2, KubeSphere, cloud, or another cluster | [Kubernetes deployment](docs/kubernetes-deployment.md) |
+| MetaX GPU deployment | [MetaX deployment](docs/metax-deployment.md) |
+
+The steps below use k3d as the example.
+
+### 1. Get the examples and install the command-line tool
+
+```bash
+git clone https://github.com/shiweijiezero/foretoken.git
+cd foretoken
+pip install -e .
+
+# For the published CLI instead:
+# pip install foretoken
+```
+
+### 2. Install the Kubernetes platform
+
+Create a local k3d cluster named `foretoken-dev` and install Foretoken:
+
+```bash
+# Use GPU index 0 from nvidia-smi. To use two GPUs, pass --gpus 0,1.
+foretoken cluster create k3d --name foretoken-dev --gpus 0
+
+# Build from the current source checkout:
+foretoken install -e .
+
+# Use published images instead:
+# foretoken install
+```
+
+The host must have Docker, NVIDIA Container Toolkit, k3d, kubectl, and Helm, and your user must be able to run `docker info` without `sudo`. Install host dependencies separately when needed. For kind or an existing Kubernetes cluster, use the corresponding guide in the table above.
+
+### 3. Deploy the Quick Start
+
+```bash
+foretoken deploy examples/quickstart --timeout 20m
+```
+
+This example deploys one frontend service and one `Qwen/Qwen3-0.6B` model replica. The model requests 1 GPU, 4 CPU, and 48 GiB memory, with limits of 8 CPU and 64 GiB. The example uses the repository-root `./data` directory for model files and runtime cache. More deployments are available in [`examples/`](examples/).
+
+### 4. Send a test request
+
+```bash
+FORETOKEN_FRONTEND_URL="$(foretoken endpoint examples/quickstart)"
+
+curl --fail-with-body --no-buffer \
+  "$FORETOKEN_FRONTEND_URL/v1/chat/completions" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Hello"}],"stream":true}'
+```
+
+### Iterate on source
+
+After editing the checkout, run the same deploy command again to apply the change without recreating the cluster or manually importing the runtime image:
+
+```bash
+foretoken deploy examples/quickstart --timeout 20m
+```
+
+Python, Triton, Rust, CUDA, C/C++, and vLLM source changes use the cluster build caches and reuse the runtime environment when dependencies and startup code are unchanged. See [Deploy Foretoken from Source](docs/custom-deployment.md) for engine checkouts and runtime changes.
+
+### 5. Evaluate and profile the service
+
+The examples save results locally and to W&B. Run `wandb login` once before using W&B.
+
+#### Performance: latency and throughput
+
+```bash
+foretoken perf examples/quickstart --num-prompts 20 --output local,wandb
+```
+
+Read request success, latency, and throughput in the summary. [Performance examples](benchmarks/docs/perf/README.md) cover other workloads and load settings.
+
+#### Quality: score model answers
+
+```bash
+foretoken eval examples/quickstart \
+  --evaluator lm-eval --tasks gsm8k --limit 100 --output local,wandb
+```
+
+This scores 100 GSM8K math problems. See [Quality evaluation](benchmarks/docs/eval/README.md) for EvalScope, task parameters, and saved scores.
+
+#### Profiling: inspect execution bottlenecks
+
+Use a source-installed CLI and platform for profiling, as described in the [profiling guide](benchmarks/docs/profile/README.md). The Quick Start already configures persistent capture storage.
+
+```bash
+foretoken perf examples/quickstart \
+  --profile --profile-engine pytorch --profile-duration 15s \
+  --num-prompts 2 --max-tokens 128 --output local,wandb
+foretoken profile view
+```
+
+Open the printed URL to inspect the capture. Press Ctrl+C to close the viewer; the model service remains running.
+
+## Gateway Mode
+
+Gateway mode provides a shared entry point through Kubernetes Gateway and a hostname. It suits clusters that already use Gateway or manage external traffic centrally.
+
+Add the public hostname under `spec` in `examples/quickstart/frontend.yaml`:
+
+```yaml
+spec:
+  hostname: foretoken.example.com
+```
+
+Then run:
+
+```bash
+# Install the platform in Gateway mode
+foretoken install --frontend-mode gateway
+# For a source-installed platform:
+# foretoken install -e . --frontend-mode gateway
+
+# Deploy the Quick Start
+foretoken deploy examples/quickstart --timeout 20m
+
+# Resolve the Gateway address and request hostname
+FORETOKEN_FRONTEND_URL="$(foretoken endpoint examples/quickstart)"
+FORETOKEN_REQUEST_HOST="$(foretoken endpoint examples/quickstart --host)"
+
+# Send a test request
+curl --fail-with-body --no-buffer \
+  "$FORETOKEN_FRONTEND_URL/v1/chat/completions" \
+  -H "Host: $FORETOKEN_REQUEST_HOST" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Hello"}],"stream":true}'
+```
+
+The command installs Envoy Gateway when needed. See the [command-line tool guide](cli/README.md) to reuse an existing Gateway or select a listener.
+
+## Stop and Uninstall
+
+```bash
+# Delete the Quick Start resources, including its namespace and runtime cache PVC
+foretoken delete examples/quickstart
+
+# Uninstall the Foretoken platform
+foretoken uninstall
+```
+
+The uninstall command preserves Foretoken CRDs, log storage, and reused cluster components. It removes the platform and the monitoring or Gateway resources managed by the command-line tool.
+
+## Related Projects
+
+- [vLLM](https://github.com/vllm-project/vllm)
+- [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo)
+- [llm-d](https://github.com/llm-d/llm-d)
+- [AIBrix](https://github.com/vllm-project/aibrix)
+- [vLLM Production Stack](https://github.com/vllm-project/production-stack)
+
+## Contributing
+
+Contributions of all kinds are welcome, including code, documentation, tests, design discussions, issue reports, and improvements to deployment, hardware, benchmarking, routing, and autoscaling.
+Performance-related changes should include the test setup, raw results, and reproducible commands.
+See [Contributing to Foretoken](CONTRIBUTING.md) for development principles, collaboration expectations, and the pull request workflow.
+
+Thank you to everyone who has contributed to Foretoken.
+
+<a href="https://github.com/shiweijiezero/foretoken/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=shiweijiezero/foretoken" width="256" alt="Foretoken contributors" />
+</a>
+
+## License
+
+This project is licensed under the [Apache License 2.0](LICENSE).

foretoken-0.0.4/benchmarks/__init__.py

@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the Foretoken project
+
+"""Foretoken benchmark command family; each domain owns an independent lifecycle."""

foretoken-0.0.4/benchmarks/config/__init__.py

@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the Foretoken project
+
+"""Benchmark configuration types and command-line parsing."""