foretoken 0.0.3__tar.gz → 0.0.4__tar.gz

foretoken-0.0.4/PKG-INFO

@@ -0,0 +1,249 @@
+Metadata-Version: 2.4
+Name: foretoken
+Version: 0.0.4
+Summary: Command-line tools for installing and operating Foretoken
+Author: Foretoken contributors
+License-Expression: Apache-2.0
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: datasets>=2.14
+Requires-Dist: evalscope[perf]==1.11.1
+Requires-Dist: httpx>=0.27
+Requires-Dist: huggingface_hub>=0.23
+Requires-Dist: ijson>=3.2
+Requires-Dist: lm-eval[api]==0.4.13
+Requires-Dist: matplotlib>=3.7
+Requires-Dist: numpy>=1.24
+Requires-Dist: openai>=1.40
+Requires-Dist: packaging>=24.0
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: tqdm>=4.65
+Requires-Dist: transformers>=4.40
+Requires-Dist: wandb>=0.19.10
+Provides-Extra: dev
+Requires-Dist: grafana-foundation-sdk==1769699379!11.6.0; extra == "dev"
+Dynamic: license-file
+
+<!--
+SPDX-License-Identifier: Apache-2.0
+SPDX-FileCopyrightText: Copyright contributors to the Foretoken project
+-->
+
+# Foretoken command-line tool
+
+English | [简体中文](README_zh.md)
+
+The Foretoken command-line tool installs the shared Kubernetes platform, deploys model services from Kustomize configurations, reports serving readiness, resolves frontend URLs, and runs benchmarks through one `foretoken` entry point.
+
+## Before you start
+
+You need Python 3.11 or later, an active Kubernetes context, `kubectl`, and Helm. GPU nodes must already have their vendor driver and Kubernetes device plugin.
+
+## Install the command-line tool
+
+Install the published command-line tool with pip:
+
+```bash
+pip install foretoken
+
+# From a source checkout:
+# pip install -e .
+```
+
+Or create and activate a virtual environment with uv:
+
+```bash
+uv venv
+source .venv/bin/activate
+uv pip install foretoken
+```
+
+Run `foretoken --version` to check the installed CLI version.
+
+## Create a local cluster
+
+On a Linux GPU host with Docker, NVIDIA Container Toolkit, and k3d installed:
+
+```bash
+# Name the local cluster and use GPU index 0 from nvidia-smi.
+# To use two GPUs, pass --gpus 0,1.
+foretoken cluster create k3d --name foretoken-dev --gpus 0
+```
+
+For a local kind development cluster:
+
+```bash
+foretoken cluster create kind --name foretoken-dev
+```
+
+Remove a cluster created by the CLI with:
+
+```bash
+foretoken cluster delete k3d --name foretoken-dev
+```
+
+## Install the Kubernetes platform
+
+`foretoken install` installs the Foretoken CRDs and controller in the active Kubernetes context. Platform resources use the `foretoken-platform` namespace. The command also configures monitoring and, in Gateway mode, the Gateway resources. Deploy model services separately with `foretoken deploy`.
+
+### Default installation
+
+The default uses release images and local access through a `LoadBalancer` Service:
+
+```bash
+foretoken install
+```
+
+Installation selects the NVIDIA or MetaX runtime and automatically reuses or installs LeaderWorkerSet and the shared RDMA device plugin. Explicit runtime settings in `--values` take precedence; in a mixed-GPU cluster, select a resource with `runtime.vllm.gpu.resourceName` or restrict the nodes with `runtime.vllm.gpu.nodeSelector`.
+
+Log collection and persistence are enabled by default. See [Observability](../observability/README.md) for configuration, log queries, dashboards, and alerts.
+
+### Gateway mode
+
+Gateway mode creates a dedicated `GatewayClass` and `Gateway`, installing Envoy Gateway if no compatible controller is available:
+
+```bash
+foretoken install --frontend-mode gateway
+```
+
+With another Gateway Controller, reuse a Gateway managed by that controller:
+
+```bash
+foretoken install \
+  --frontend-mode gateway \
+  --gateway-name inference-gateway \
+  --gateway-namespace gateway-system
+```
+
+Add `--gateway-section-name LISTENER` only when more than one listener matches.
+
+### Current source
+
+Build and install from the repository root. The cluster needs a default StorageClass for compiler caches; see the [source deployment guide](../docs/custom-deployment.md) for storage overrides.
+
+```bash
+foretoken install -e .
+```
+
+This builds the platform in dedicated Pods and binds the checkout to the target cluster.
+
+After editing it, use `foretoken deploy` to [redeploy source changes](../docs/custom-deployment.md#deploy-and-update-code). Use `--engine-source PATH` to also bind a [vLLM engine checkout](../docs/custom-deployment.md#edit-an-inference-engine).
+
+A standard active kind or k3d context loads the built images directly into its nodes. Other Kubernetes contexts need a registry reachable by the Build Pods and nodes. For an internal registry without authentication:
+
+```bash
+foretoken install -e . --registry registry.example.com:5000/foretoken
+```
+
+If the registry requires authentication, follow the [source deployment guide](../docs/kubernetes-deployment.md) before installation.
+
+### Model distribution
+
+To share public model downloads between nodes through Dragonfly, save this in `deploy/platform-values.yaml`:
+
+```yaml
+modelDistribution:
+  dragonfly:
+    enabled: true
+```
+
+For a published platform installation, apply the values with:
+
+```bash
+foretoken install --values deploy/platform-values.yaml
+```
+
+For a source installation, run `foretoken install -e . --values deploy/platform-values.yaml` from the repository root, retaining the original registry and engine-source options.
+
+Installation prepares Dragonfly or reuses an existing installation. Models that require authentication and custom model endpoints download directly from their provider. To select a particular Dragonfly Helm release, set `existingRelease: {name: dragonfly, namespace: dragonfly-system}` under `modelDistribution.dragonfly`.
+
+On NVIDIA clusters with RDMA, ModelExpress can load weights from running replicas. Add this alongside `dragonfly` to enable it:
+
+```yaml
+modelDistribution:
+  modelexpress:
+    enabled: true
+```
+
+Automatic weight transfer uses remote models with a persistent cache, data parallelism of one, and fixed expert placement. An explicit `load-format` remains unchanged. Each GPU worker selects a nearby available RDMA interface; replicas without a compatible source load the prepared files.
+
+Reapply the installation command after changing either setting. Set `enabled: false` to disable it. `foretoken uninstall` removes managed Dragonfly resources once no workloads use them; reused installations are retained.
+
+### Installation options
+
+Use `--values` only to override platform image, runtime, or hardware settings. Without an override, installation compares supported public sources for default platform images and OCI charts. Use `--oci-registry` to select a registry explicitly; image references supplied through values remain unchanged. Source selection runs on the CLI host, so the selected registry must also be reachable from the cluster nodes.
+
+Model services are reached through an IP address outside the cluster. k3d, k3s, and cloud clusters assign one automatically. Clusters built with kubeadm, RKE2, or kubespray have no address assignment by default, so installation there ends with `LoadBalancer support Not verified`. Give Foretoken a range of unused addresses in the nodes' subnet, confirmed with the cluster administrator, and it assigns them to services:
+
+```yaml
+loadBalancer:
+  managedAddresses:
+    - 192.168.1.240-192.168.1.250
+```
+
+## Deploy and operate model services
+
+Run from the repository checkout prepared in the [Quick Start](../README.md). Deploy one frontend and all models rendered by a Kustomize root.
+
+See the [multi-model example](../examples/multi-model-quickstart/README.md) for resources and [model storage](../docs/model-storage.md) for directory or PVC configuration. Use `examples/quickstart` for a single model.
+
+```bash
+foretoken deploy examples/multi-model-quickstart --timeout 20m
+```
+
+The command applies the configuration, shows service status, and streams Pod and container logs with source prefixes while waiting. It exits when every service reports Ready and its selected alerts are configured. Without `--timeout`, it waits up to ten minutes. Configure service alerts in the Kustomize deployment; see [service observability](../examples/observability/README.md).
+
+Inspect the same deployment without applying it:
+
+```bash
+foretoken status examples/multi-model-quickstart
+```
+
+Inspect every Foretoken service in a namespace. With `--watch`, follow service state changes and Pod/container logs until Ctrl+C:
+
+```bash
+foretoken status -n foretoken-multi-model-demo
+foretoken status -n foretoken-multi-model-demo --watch
+```
+
+Resolve the public frontend URL after deployment:
+
+```bash
+FORETOKEN_FRONTEND_URL="$(foretoken endpoint examples/multi-model-quickstart)"
+```
+
+For an HTTP Gateway, resolve its request `Host` separately:
+
+```bash
+FORETOKEN_REQUEST_HOST="$(foretoken endpoint examples/multi-model-quickstart --host)"
+```
+
+`--host` returns the host and optional port for direct access, or the configured routing hostname for an HTTP Gateway. `foretoken endpoint` waits for the LoadBalancer or Gateway address; use `foretoken deploy` to wait for the services to become ready.
+
+## Measure serving performance
+
+Use `foretoken perf` to measure response latency and request or token throughput. Pass a Kustomize directory, or `--url` with `--model` for an existing endpoint. Choose a workload in [Performance examples](../benchmarks/docs/perf/README.md).
+
+## Evaluate and compare models
+
+Use `foretoken eval` to score model answers with lm-evaluation-harness or EvalScope. It accepts the same service selection options; task and scoring parameters use the selected framework's syntax. See [Quality evaluation](../benchmarks/docs/eval/README.md). Add `--reference` to [compare a candidate's probabilities against a reference](../benchmarks/docs/eval/distribution-comparison.md).
+
+## Export figures
+
+Use `--output local,wandb,plot` with a benchmark, or `foretoken plot RESULT_DIR` to redraw a saved run or sweep without running inference. Export options and comparisons are in [parameter sweeps](../benchmarks/docs/perf/sweep.md).
+
+## Find execution bottlenecks
+
+Add `--profile` to `foretoken deploy` or `foretoken perf` to record CPU/GPU execution, then browse captures with `foretoken profile view`. Setup and commands are in [Profiling](../benchmarks/docs/profile/README.md).
+
+## Clean up
+
+Delete the deployed services before uninstalling the platform:
+
+```bash
+foretoken delete examples/multi-model-quickstart
+foretoken uninstall
+```
+
+CRDs and reused cluster components are retained. Managed LeaderWorkerSet and MetalLB controllers are also retained while workloads still depend on them.

{foretoken-0.0.3 → foretoken-0.0.4}/README.md

@@ -21,42 +21,53 @@ If you only need to serve a single model on one GPU, using an inference engine s
 
 | Feature | Description | Status |
 |---|---|---|
-| [Benchmarking](benchmarks/README.md) | Measure model-service performance | In development |
-| Profiling | Use PyTorch Profiler and Nsight to identify compute, communication, and CPU/GPU bottlenecks | Planned |
+| [Evaluation](benchmarks/README.md) | Measure service performance and model quality | In development |
+| [Profiling](benchmarks/docs/profile/README.md) | Capture PyTorch, NVIDIA Nsight Systems, or MetaX mcTracer timelines for a model service | In development |
 | Hardware support | Common interfaces for device capabilities, runtimes, communication, and metrics; see [MetaX deployment](docs/metax-deployment.md) | In development |
 | Request routing | Select instances based on load, queues, KV reuse, and service levels | Research |
 | Distributed inference | Aggregated serving, Prefill/Decode disaggregation, and WideEP parallelism | Research |
 | Control plane | Model services, replica management, autoscaling, updates, and failure recovery | In development |
-| [Observability](observability/README.md) | Collect service and accelerator metrics, evaluate alerts, and inspect the system Dashboard | In development |
+| [Observability](observability/README.md) | Collect metrics and persistent service logs, evaluate alerts, and inspect the system Dashboard | In development |
 
 ## Quick Start
 
-Start with a GPU-enabled Kubernetes cluster and Python 3.11+, `kubectl`, and Helm installed locally.
+Choose the deployment path before running the common steps:
+
+| Situation | Guide |
+|---|---|
+| Single-host local deployment | [k3d deployment](docs/k3d-deployment.md) · [kind deployment](docs/kind-deployment.md) |
+| Kubernetes deployment with K3s, RKE2, KubeSphere, cloud, or another cluster | [Kubernetes deployment](docs/kubernetes-deployment.md) |
+| MetaX GPU deployment | [MetaX deployment](docs/metax-deployment.md) |
+
+The steps below use k3d as the example.
 
 ### 1. Get the examples and install the command-line tool
 
 ```bash
 git clone https://github.com/shiweijiezero/foretoken.git
 cd foretoken
-pip install foretoken
+pip install -e .
 
-# From a source checkout:
-# pip install -e .
+# For the published CLI instead:
+# pip install foretoken
 ```
 
 ### 2. Install the Kubernetes platform
 
+Create a local k3d cluster named `foretoken-dev` and install Foretoken:
+
 ```bash
-# Use release images from GHCR:
-foretoken install
+# Use GPU index 0 from nvidia-smi. To use two GPUs, pass --gpus 0,1.
+foretoken cluster create k3d --name foretoken-dev --gpus 0
 
-# Build and install from a source checkout:
-# foretoken install -e .
-```
+# Build from the current source checkout:
+foretoken install -e .
 
-For deployment on MetaX GPUs, follow the [MetaX deployment guide](docs/metax-deployment.md).
+# Use published images instead:
+# foretoken install
+```
 
-See the [source deployment guide](docs/custom-deployment.md) for build tools and remote clusters.
+The host must have Docker, NVIDIA Container Toolkit, k3d, kubectl, and Helm, and your user must be able to run `docker info` without `sudo`. Install host dependencies separately when needed. For kind or an existing Kubernetes cluster, use the corresponding guide in the table above.
 
 ### 3. Deploy the Quick Start
 
@@ -64,7 +75,7 @@ See the [source deployment guide](docs/custom-deployment.md) for build tools and
 foretoken deploy examples/quickstart --timeout 20m
 ```
 
-This example deploys one frontend service and one `Qwen/Qwen3-0.6B` model replica, requesting one GPU, 8 CPU, and 52 GiB memory. More deployments are available in [`examples/`](examples/).
+This example deploys one frontend service and one `Qwen/Qwen3-0.6B` model replica. The model requests 1 GPU, 4 CPU, and 48 GiB memory, with limits of 8 CPU and 64 GiB. The example uses the repository-root `./data` directory for model files and runtime cache. More deployments are available in [`examples/`](examples/).
 
 ### 4. Send a test request
 
@@ -77,18 +88,49 @@ curl --fail-with-body --no-buffer \
   -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Hello"}],"stream":true}'
 ```
 
-### 5. Measure the model service
+### Iterate on source
+
+After editing the checkout, run the same deploy command again to apply the change without recreating the cluster or manually importing the runtime image:
 
 ```bash
-pip install 'foretoken[bench]'
+foretoken deploy examples/quickstart --timeout 20m
+```
+
+Python, Triton, Rust, CUDA, C/C++, and vLLM source changes use the cluster build caches and reuse the runtime environment when dependencies and startup code are unchanged. See [Deploy Foretoken from Source](docs/custom-deployment.md) for engine checkouts and runtime changes.
+
+### 5. Evaluate and profile the service
 
-# From a source checkout:
-# pip install -e '.[bench]'
+The examples save results locally and to W&B. Run `wandb login` once before using W&B.
 
-foretoken bench examples/quickstart --output local,wandb
+#### Performance: latency and throughput
+
+```bash
+foretoken perf examples/quickstart --num-prompts 20 --output local,wandb
 ```
 
-Run `wandb login` before first using W&B. See [Model Service Benchmarks](benchmarks/README.md) for more examples.
+Read request success, latency, and throughput in the summary. [Performance examples](benchmarks/docs/perf/README.md) cover other workloads and load settings.
+
+#### Quality: score model answers
+
+```bash
+foretoken eval examples/quickstart \
+  --evaluator lm-eval --tasks gsm8k --limit 100 --output local,wandb
+```
+
+This scores 100 GSM8K math problems. See [Quality evaluation](benchmarks/docs/eval/README.md) for EvalScope, task parameters, and saved scores.
+
+#### Profiling: inspect execution bottlenecks
+
+Use a source-installed CLI and platform for profiling, as described in the [profiling guide](benchmarks/docs/profile/README.md). The Quick Start already configures persistent capture storage.
+
+```bash
+foretoken perf examples/quickstart \
+  --profile --profile-engine pytorch --profile-duration 15s \
+  --num-prompts 2 --max-tokens 128 --output local,wandb
+foretoken profile view
+```
+
+Open the printed URL to inspect the capture. Press Ctrl+C to close the viewer; the model service remains running.
 
 ## Gateway Mode
 
@@ -136,13 +178,7 @@ foretoken delete examples/quickstart
 foretoken uninstall
 ```
 
-The uninstall command preserves Foretoken CRDs and reused cluster components. It removes the platform and the monitoring or Gateway resources managed by the command-line tool.
-
-## Deployment Guides
-
-- [Source builds and private registries](docs/custom-deployment.md)
-- [Single-machine GPU clusters with k3d](docs/k3d-deployment.md)
-- [MetaX GPUs](docs/metax-deployment.md)
+The uninstall command preserves Foretoken CRDs, log storage, and reused cluster components. It removes the platform and the monitoring or Gateway resources managed by the command-line tool.
 
 ## Related Projects