@cubis/foundry 0.3.83 → 0.3.84

package/workflows/workflows/agent-environment-setup/platforms/codex/skills/huggingface-ml/SKILL.md

@@ -1,97 +0,0 @@
----
-name: huggingface-ml
-description: "Use when building ML workflows with Hugging Face, including model loading, inference APIs, fine-tuning with Trainer or PEFT, Hub versioning, and Spaces demos."
----
-# Hugging Face ML Integration
-
-## Purpose
-
-Guide the design and implementation of machine learning workflows using the Hugging Face ecosystem, covering model discovery and loading from the Hub, local and remote inference with Transformers and the Inference API, fine-tuning with Trainer and parameter-efficient methods (LoRA, QLoRA), model deployment to production endpoints, and interactive demo creation with Spaces.
-
-## When to Use
-
-- Loading and running pre-trained models from the Hugging Face Hub for inference.
-- Calling the Inference API for serverless model execution without managing infrastructure.
-- Fine-tuning a base model on custom data with Trainer, PEFT, or TRL.
-- Deploying fine-tuned models to Inference Endpoints or custom serving infrastructure.
-- Building interactive demos with Gradio on Hugging Face Spaces.
-- Evaluating model performance with the `evaluate` library and custom metrics.
-
-## Instructions
-
-1. **Search the Hub for existing models before training from scratch** because the Hub hosts 500k+ models across tasks, and fine-tuning an existing checkpoint is 10-100x cheaper than pre-training.
-
-2. **Use `AutoModel` and `AutoTokenizer` with the model ID for loading** because Auto classes detect the correct architecture from the Hub metadata, avoiding manual class selection errors and supporting future model architectures.
-
-3. **Pin model revisions with the `revision` parameter or commit hash** because models on the Hub are mutable, and an unversioned reference may load a different checkpoint than the one you tested against.
-
-4. **Configure `device_map="auto"` for models that exceed single-GPU memory** because automatic device mapping splits the model across available GPUs and CPU RAM with offloading, enabling inference on hardware that would otherwise OOM.
-
-5. **Use the Inference API with `InferenceClient` for serverless inference** because it eliminates GPU provisioning, scales to zero when idle, and supports rate-limited production workloads without infrastructure management.
-
-6. **Set `InferenceClient` timeout and retry parameters for production calls** because serverless cold starts can exceed default timeouts, and transient failures require exponential backoff to avoid cascading failures.
-
-7. **Fine-tune with `Trainer` and a `TrainingArguments` configuration** because Trainer handles distributed training, mixed precision, gradient accumulation, checkpointing, and evaluation loops without boilerplate code.
-
-8. **Use PEFT (LoRA or QLoRA) for parameter-efficient fine-tuning** because full fine-tuning of large models requires prohibitive GPU memory, and LoRA adapters achieve comparable quality while training less than 1% of parameters.
-
-9. **Prepare datasets with the `datasets` library using streaming for large corpora** because streaming loads data lazily from disk or the Hub, avoiding RAM exhaustion on datasets larger than available memory.
-
-10. **Evaluate models with the `evaluate` library and task-specific metrics** because comparing model quality requires standardized metrics (BLEU, ROUGE, F1, accuracy) computed on held-out test splits with reproducible configurations.
-
-11. **Push fine-tuned models and adapters to the Hub with `push_to_hub`** because the Hub provides versioning, model cards, download tracking, and access control, replacing ad-hoc model storage on cloud buckets.
-
-12. **Deploy production models to Inference Endpoints with autoscaling configuration** because dedicated endpoints provide guaranteed latency, custom container images, and scaling policies that the shared Inference API does not offer.
-
-13. **Build interactive demos with Gradio on Spaces** because stakeholders and users need to test model behavior without writing code, and Spaces provides free hosting with GPU allocation for demos.
-
-14. **Write model cards with training data, evaluation results, and limitation disclosures** because model cards are the standard for ML documentation, and the Hub renders them as the model's landing page.
-
-15. **Quantize models with `bitsandbytes` or GPTQ before deployment** because quantization reduces model size by 2-4x and inference latency proportionally, enabling deployment on smaller GPUs without meaningful quality loss.
-
-16. **Use `safetensors` format for model serialization** because safetensors prevents arbitrary code execution on load (unlike pickle), supports memory-mapped loading for instant startup, and is the Hub default for new models.
-
-## Output Format
-
-```
-## Model Selection
-[Hub search results, architecture choice, task compatibility]
-
-## Implementation
-[Loading, inference, or fine-tuning code with configuration]
-
-## Evaluation
-[Metrics, test split results, comparison with baseline]
-
-## Deployment
-[Endpoint configuration, autoscaling, monitoring]
-```
-
-## References
-
-| File                             | Load when                                                                                  |
-| -------------------------------- | ------------------------------------------------------------------------------------------ |
-| `references/model-selection.md` | Loading models, comparing checkpoints, and choosing the right task framing or pipeline.    |
-| `references/inference-optimization.md` | Running inference, calling hosted APIs, and tuning latency, batching, and cost.      |
-| `references/fine-tuning-guide.md` | Training with Trainer, PEFT/LoRA, dataset preparation, or evaluation.                    |
-
-## Examples
-
-- "Fine-tune Llama 3 on a custom Q&A dataset using QLoRA with the Trainer API."
-- "Set up a production Inference Endpoint with autoscaling for a text classification model."
-- "Build a Gradio demo on Spaces for a fine-tuned image captioning model."
-
-## Codex Platform Notes
-
-- Codex supports native subagents via `.codex/agents/*.toml` files with `name`, `description`, and `developer_instructions`.
-- Each subagent TOML can specify `model` and `model_reasoning_effort` to optimize cost per task difficulty:
-  - Light tasks (exploration, docs): `model = "gpt-5.3-codex-spark"`, `model_reasoning_effort = "medium"`
-  - Heavy tasks (security audit, orchestration): `model = "gpt-5.4"`, `model_reasoning_effort = "high"`
-  - Standard tasks (implementation): inherit parent model (omit `model` field).
-- Built-in agents: `default`, `worker`, `explorer`. Custom agents extend these via TOML definitions.
-- Codex operates under network restrictions — skills should not assume outbound HTTP access.
-- Use `$ARGUMENTS` to access user-provided arguments when the skill is invoked.
-- All skill guidance executes within the sandbox; file I/O is confined to the workspace.
-- Skills are installed at `.agents/skills/<skill-id>/SKILL.md`. Workflow skills can also be compiled to `.agents/skills/<workflow-id>/SKILL.md` as `generatedSkills`.
-- Codex supports three autonomy levels: `suggest`, `auto-edit`, `full-auto`.
-- MCP skill tools are available when the Cubis Foundry MCP server is connected.

package/workflows/workflows/agent-environment-setup/platforms/codex/skills/huggingface-ml/evals/assertions.md

@@ -1,48 +0,0 @@
-# Hugging Face ML Eval Assertions
-
-## Eval 1: Model Fine-Tuning Workflow
-
-This eval tests the complete fine-tuning workflow: PEFT/QLoRA configuration, Trainer setup, dataset preparation, evaluation metrics, and model publishing.
-
-### Assertions
-
-| # | Type     | What it checks                                  | Why it matters                                                                                       |
-|---|----------|-------------------------------------------------|------------------------------------------------------------------------------------------------------|
-| 1 | contains | `LoraConfig` — PEFT adapter configuration        | LoRA enables fine-tuning large models by training low-rank adapter matrices instead of all parameters, reducing GPU memory by 10-100x while maintaining quality. |
-| 2 | contains | `TrainingArguments` — Training configuration     | Trainer requires TrainingArguments for learning rate, batch size, gradient accumulation, warmup, and precision settings. Omitting it uses defaults that are rarely optimal. |
-| 3 | contains | `compute_metrics` — Evaluation function           | Without a compute_metrics function, Trainer only reports loss. F1 and accuracy on the eval split are essential for monitoring classification quality during training. |
-| 4 | contains | `BitsAndBytesConfig` — Quantization config       | QLoRA requires 4-bit quantization via BitsAndBytesConfig to fit large models in limited GPU memory. Without it, loading the base model alone exceeds available VRAM. |
-| 5 | contains | `push_to_hub` — Hub publishing                   | Publishing to the Hub provides versioned storage, model cards, download tracking, and easy deployment, replacing ad-hoc model file management. |
-
-### What a passing response looks like
-
-- `BitsAndBytesConfig` with `load_in_4bit=True`, nf4 quantization type, and float16 compute dtype.
-- `LoraConfig` specifying target modules (q_proj, v_proj), rank, alpha, dropout, and task type.
-- Dataset loaded with the `datasets` library, tokenized with the model's tokenizer, split into train/eval.
-- `TrainingArguments` with learning rate ~2e-4, cosine scheduler, bf16/fp16, gradient accumulation, and eval strategy.
-- `compute_metrics` returning F1 (macro) and accuracy from sklearn or evaluate library.
-- `trainer.push_to_hub()` or `model.push_to_hub()` call after training completes.
-
----
-
-## Eval 2: Inference API Integration
-
-This eval tests production Inference API usage: client configuration, embedding generation, error handling with backoff, and local fallback.
-
-### Assertions
-
-| # | Type     | What it checks                                  | Why it matters                                                                                       |
-|---|----------|-------------------------------------------------|------------------------------------------------------------------------------------------------------|
-| 1 | contains | `InferenceClient` — Official client usage        | InferenceClient provides typed methods, automatic retry, and consistent authentication, avoiding raw HTTP calls that miss edge cases in the API protocol. |
-| 2 | contains | `feature_extraction` — Embedding task endpoint   | Sentence embeddings are generated through the feature_extraction task, which returns dense vectors suitable for semantic search and similarity computation. |
-| 3 | contains | `timeout` — Explicit timeout configuration       | Inference API cold starts can take 10-60 seconds for large models. Without explicit timeouts, client calls hang indefinitely during cold starts or degradation. |
-| 4 | contains | `429` — Rate limit handling                      | The Inference API enforces rate limits per model and account tier. Without backoff on 429 responses, the client enters a retry storm that extends the lockout. |
-| 5 | contains | `SentenceTransformer` — Local fallback           | A local fallback ensures the search feature continues working when the Inference API is unavailable, maintaining service availability at the cost of latency. |
-
-### What a passing response looks like
-
-- `InferenceClient` initialized with API token and explicit timeout settings.
-- Embedding generation using `client.feature_extraction()` with the sentence-transformers model ID.
-- Exponential backoff retry logic that detects 429 status codes and waits before retrying.
-- Local `SentenceTransformer` model loaded on fallback with the same model ID for embedding consistency.
-- Batch embedding support for document indexing using the client's batch capabilities.

package/workflows/workflows/agent-environment-setup/platforms/codex/skills/huggingface-ml/evals/evals.json

@@ -1,66 +0,0 @@
-[
-  {
-    "name": "model-fine-tuning-workflow",
-    "description": "Validate that the skill produces a complete fine-tuning workflow using Trainer with PEFT/LoRA, proper dataset preparation, evaluation metrics, and model publishing to the Hub.",
-    "input": "Fine-tune a Llama 3 8B model on a custom customer support dataset for text classification (5 categories: billing, technical, account, shipping, general). Use QLoRA for parameter-efficient training. The dataset is in CSV format with 'text' and 'label' columns (50k rows). Configure Trainer with proper training arguments including learning rate scheduling, gradient accumulation, and mixed precision. Evaluate with F1 score and accuracy. Push the fine-tuned adapter to the Hugging Face Hub.",
-    "assertions": [
-      {
-        "type": "contains",
-        "value": "LoraConfig",
-        "description": "Must configure PEFT with LoraConfig specifying target modules, rank, alpha, and dropout for parameter-efficient fine-tuning of the large model."
-      },
-      {
-        "type": "contains",
-        "value": "TrainingArguments",
-        "description": "Must define TrainingArguments with learning rate, batch size, gradient accumulation steps, warmup, and mixed precision settings for training configuration."
-      },
-      {
-        "type": "contains",
-        "value": "compute_metrics",
-        "description": "Must implement a compute_metrics function that calculates F1 score and accuracy on the evaluation split for monitoring training quality."
-      },
-      {
-        "type": "contains",
-        "value": "BitsAndBytesConfig",
-        "description": "Must use BitsAndBytesConfig for 4-bit quantization (QLoRA) to reduce GPU memory requirements while maintaining training quality."
-      },
-      {
-        "type": "contains",
-        "value": "push_to_hub",
-        "description": "Must push the fine-tuned model or adapter to the Hugging Face Hub for versioning, sharing, and deployment."
-      }
-    ]
-  },
-  {
-    "name": "inference-api-integration",
-    "description": "Validate that the skill produces a production-ready Inference API integration with proper client configuration, error handling, rate limiting, and fallback strategies.",
-    "input": "Build a Python service that uses the Hugging Face Inference API to power a semantic search feature. Use a sentence-transformers model for generating embeddings. The service receives search queries, generates embeddings via the Inference API, and queries a vector database. Handle rate limits with exponential backoff. Implement a fallback to local model inference when the API is unavailable. Include proper timeout configuration and batch embedding support for indexing documents.",
-    "assertions": [
-      {
-        "type": "contains",
-        "value": "InferenceClient",
-        "description": "Must use the InferenceClient from the huggingface_hub library for typed, ergonomic access to the Inference API with built-in retry support."
-      },
-      {
-        "type": "contains",
-        "value": "feature_extraction",
-        "description": "Must use the feature_extraction or sentence_similarity task endpoint for generating embeddings from the sentence-transformers model."
-      },
-      {
-        "type": "contains",
-        "value": "timeout",
-        "description": "Must configure explicit timeout values for Inference API calls to prevent indefinite hangs during cold starts or API degradation."
-      },
-      {
-        "type": "contains",
-        "value": "429",
-        "description": "Must handle HTTP 429 rate limit responses with exponential backoff and retry logic to gracefully handle API quota exhaustion."
-      },
-      {
-        "type": "contains",
-        "value": "SentenceTransformer",
-        "description": "Must implement a local fallback using SentenceTransformer for generating embeddings when the Inference API is unavailable."
-      }
-    ]
-  }
-]

package/workflows/workflows/agent-environment-setup/platforms/codex/skills/huggingface-ml/examples/01-basic-usage.md

@@ -1,194 +0,0 @@
-# Example: Text Classification with Hugging Face Transformers
-
-## Scenario
-
-A team is building a customer support ticket routing system. They need to classify incoming support tickets into categories (billing, technical, account, general) using a pre-trained text classification model from the Hugging Face Hub. The system should load the model locally for low-latency inference, handle batch classification for processing backlogs, and include a confidence threshold to flag uncertain predictions for human review.
-
-## Prompt
-
-"Set up a text classification pipeline using Hugging Face Transformers to categorize customer support tickets into billing, technical, account, and general categories. Use a pre-trained model from the Hub, implement batch classification, and add a confidence threshold for uncertain predictions."
-
-## Expected Output
-
-### Model Selection and Loading
-
-```python
-# classify.py
-from transformers import pipeline, AutoModelForSequenceClassification, AutoTokenizer
-import torch
-
-MODEL_ID = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
-# For production, fine-tune on your own labeled data. This example uses a
-# pre-trained sentiment model to demonstrate the pipeline pattern.
-
-# Option 1: High-level pipeline (simplest)
-classifier = pipeline(
-    "text-classification",
-    model=MODEL_ID,
-    device=0 if torch.cuda.is_available() else -1,
-)
-
-# Option 2: Manual loading (more control)
-tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
-model = AutoModelForSequenceClassification.from_pretrained(
-    MODEL_ID,
-    revision="main",  # Pin to a specific revision in production
-)
-model.eval()
-
-if torch.cuda.is_available():
-    model = model.cuda()
-```
-
-### Classification Service
-
-```python
-# ticket_classifier.py
-from dataclasses import dataclass
-from transformers import pipeline
-import torch
-
-CATEGORIES = ["billing", "technical", "account", "general"]
-CONFIDENCE_THRESHOLD = 0.75
-
-@dataclass
-class ClassificationResult:
-    ticket_id: str
-    text: str
-    category: str
-    confidence: float
-    needs_review: bool
-
-class TicketClassifier:
-    def __init__(self, model_id: str, device: int = -1):
-        self.classifier = pipeline(
-            "text-classification",
-            model=model_id,
-            device=device,
-            top_k=None,  # Return scores for all labels
-        )
-
-    def classify_one(self, ticket_id: str, text: str) -> ClassificationResult:
-        results = self.classifier(text, truncation=True, max_length=512)
-        top_result = max(results[0], key=lambda x: x["score"])
-
-        return ClassificationResult(
-            ticket_id=ticket_id,
-            text=text[:200],
-            category=top_result["label"],
-            confidence=top_result["score"],
-            needs_review=top_result["score"] < CONFIDENCE_THRESHOLD,
-        )
-
-    def classify_batch(
-        self, tickets: list[dict[str, str]], batch_size: int = 32
-    ) -> list[ClassificationResult]:
-        texts = [t["text"] for t in tickets]
-        ids = [t["id"] for t in tickets]
-
-        # Pipeline handles batching internally
-        all_results = self.classifier(
-            texts,
-            truncation=True,
-            max_length=512,
-            batch_size=batch_size,
-        )
-
-        classifications = []
-        for ticket_id, text, results in zip(ids, texts, all_results):
-            top_result = max(results, key=lambda x: x["score"])
-            classifications.append(
-                ClassificationResult(
-                    ticket_id=ticket_id,
-                    text=text[:200],
-                    category=top_result["label"],
-                    confidence=top_result["score"],
-                    needs_review=top_result["score"] < CONFIDENCE_THRESHOLD,
-                )
-            )
-
-        return classifications
-```
-
-### FastAPI Integration
-
-```python
-# api.py
-from fastapi import FastAPI, HTTPException
-from pydantic import BaseModel
-from ticket_classifier import TicketClassifier
-
-app = FastAPI()
-classifier = TicketClassifier(
-    model_id="your-org/ticket-classifier-v2",
-    device=0,  # GPU
-)
-
-class TicketRequest(BaseModel):
-    ticket_id: str
-    text: str
-
-class BatchRequest(BaseModel):
-    tickets: list[TicketRequest]
-
-class ClassificationResponse(BaseModel):
-    ticket_id: str
-    category: str
-    confidence: float
-    needs_review: bool
-
-@app.post("/classify", response_model=ClassificationResponse)
-def classify_ticket(request: TicketRequest):
-    result = classifier.classify_one(request.ticket_id, request.text)
-    return ClassificationResponse(
-        ticket_id=result.ticket_id,
-        category=result.category,
-        confidence=result.confidence,
-        needs_review=result.needs_review,
-    )
-
-@app.post("/classify/batch", response_model=list[ClassificationResponse])
-def classify_batch(request: BatchRequest):
-    tickets = [{"id": t.ticket_id, "text": t.text} for t in request.tickets]
-    results = classifier.classify_batch(tickets)
-    return [
-        ClassificationResponse(
-            ticket_id=r.ticket_id,
-            category=r.category,
-            confidence=r.confidence,
-            needs_review=r.needs_review,
-        )
-        for r in results
-    ]
-```
-
-### Usage
-
-```python
-# Single classification
-import requests
-
-response = requests.post("http://localhost:8000/classify", json={
-    "ticket_id": "TKT-001",
-    "text": "I was charged twice for my subscription last month and need a refund."
-})
-print(response.json())
-# {"ticket_id": "TKT-001", "category": "billing", "confidence": 0.94, "needs_review": false}
-
-# Batch classification
-response = requests.post("http://localhost:8000/classify/batch", json={
-    "tickets": [
-        {"ticket_id": "TKT-002", "text": "Cannot log into my account after password reset"},
-        {"ticket_id": "TKT-003", "text": "The API returns 500 errors on the /users endpoint"},
-        {"ticket_id": "TKT-004", "text": "How do I change my email address?"},
-    ]
-})
-```
-
-## Key Decisions
-
-- **Pipeline API over manual model inference** -- the pipeline handles tokenization, batching, and post-processing automatically, reducing boilerplate and error-prone manual tensor handling.
-- **`top_k=None` to return all label scores** -- allows computing confidence as the gap between the top prediction and alternatives, and supports the confidence threshold check.
-- **Confidence threshold with human review flag** -- tickets below 75% confidence are flagged for manual review rather than silently misrouted, balancing automation with accuracy.
-- **Batch classification** -- processes ticket backlogs efficiently by leveraging GPU parallelism through the pipeline's internal batching.
-- **Model revision pinning** -- production deployments pin to a specific model revision to prevent unexpected behavior changes when the model is updated on the Hub.