better-dsh 0.2.4 → 0.2.5-a

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/bundled/mlops/mlops-evaluation-weights-and-biases.md

@@ -0,0 +1,613 @@
+---
+title: "Weights And Biases — W&B: log ML experiments, sweeps, model registry, dashboards"
+sidebar_label: "Weights And Biases"
+description: "W&B: log ML experiments, sweeps, model registry, dashboards"
+---
+
+{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
+
+# Weights And Biases
+
+W&B: log ML experiments, sweeps, model registry, dashboards.
+
+## Skill metadata
+
+| | |
+|---|---|
+| Source | Bundled (installed by default) |
+| Path | `skills/mlops/evaluation/weights-and-biases` |
+| Version | `1.0.1` |
+| Author | Orchestra Research |
+| License | MIT |
+| Dependencies | `wandb` |
+| Platforms | linux, macos, windows |
+| Tags | `MLOps`, `Weights And Biases`, `WandB`, `Experiment Tracking`, `Hyperparameter Tuning`, `Model Registry`, `Collaboration`, `Real-Time Visualization`, `PyTorch`, `TensorFlow`, `HuggingFace` |
+
+## Reference: full SKILL.md
+
+:::info
+The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
+:::
+
+# Weights & Biases: ML Experiment Tracking & MLOps
+
+## When to Use This Skill
+
+Use Weights & Biases (W&B) when you need to:
+- **Track ML experiments** with automatic metric logging
+- **Visualize training** in real-time dashboards
+- **Compare runs** across hyperparameters and configurations
+- **Optimize hyperparameters** with automated sweeps
+- **Manage model registry** with versioning and lineage
+- **Collaborate on ML projects** with team workspaces
+- **Track artifacts** (datasets, models, code) with lineage
+
+**Users**: 200,000+ ML practitioners | **GitHub Stars**: 10.5k+ | **Integrations**: 100+
+
+## Installation
+
+```bash
+# Install W&B
+pip install wandb
+
+# Login (creates API key)
+wandb login
+
+# Or set API key programmatically
+export WANDB_API_KEY=your_api_key_here
+```
+
+## Quick Start
+
+### Basic Experiment Tracking
+
+```python
+import wandb
+
+# Initialize a run
+run = wandb.init(
+    project="my-project",
+    config={
+        "learning_rate": 0.001,
+        "epochs": 10,
+        "batch_size": 32,
+        "architecture": "ResNet50"
+    }
+)
+
+# Training loop
+for epoch in range(run.config.epochs):
+    # Your training code
+    train_loss = train_epoch()
+    val_loss = validate()
+
+    # Log metrics
+    wandb.log({
+        "epoch": epoch,
+        "train/loss": train_loss,
+        "val/loss": val_loss,
+        "train/accuracy": train_acc,
+        "val/accuracy": val_acc
+    })
+
+# Finish the run
+wandb.finish()
+```
+
+### With PyTorch
+
+```python
+import torch
+import wandb
+
+# Initialize
+wandb.init(project="pytorch-demo", config={
+    "lr": 0.001,
+    "epochs": 10
+})
+
+# Access config
+config = wandb.config
+
+# Training loop
+for epoch in range(config.epochs):
+    for batch_idx, (data, target) in enumerate(train_loader):
+        # Forward pass
+        output = model(data)
+        loss = criterion(output, target)
+
+        # Backward pass
+        optimizer.zero_grad()
+        loss.backward()
+        optimizer.step()
+
+        # Log every 100 batches
+        if batch_idx % 100 == 0:
+            wandb.log({
+                "loss": loss.item(),
+                "epoch": epoch,
+                "batch": batch_idx
+            })
+
+# Save model
+torch.save(model.state_dict(), "model.pth")
+wandb.save("model.pth")  # Upload to W&B
+
+wandb.finish()
+```
+
+## Core Concepts
+
+### 1. Projects and Runs
+
+**Project**: Collection of related experiments
+**Run**: Single execution of your training script
+
+```python
+# Create/use project
+run = wandb.init(
+    project="image-classification",
+    name="resnet50-experiment-1",  # Optional run name
+    tags=["baseline", "resnet"],    # Organize with tags
+    notes="First baseline run"      # Add notes
+)
+
+# Each run has unique ID
+print(f"Run ID: {run.id}")
+print(f"Run URL: {run.url}")
+```
+
+### 2. Configuration Tracking
+
+Track hyperparameters automatically:
+
+```python
+config = {
+    # Model architecture
+    "model": "ResNet50",
+    "pretrained": True,
+
+    # Training params
+    "learning_rate": 0.001,
+    "batch_size": 32,
+    "epochs": 50,
+    "optimizer": "Adam",
+
+    # Data params
+    "dataset": "ImageNet",
+    "augmentation": "standard"
+}
+
+wandb.init(project="my-project", config=config)
+
+# Access config during training
+lr = wandb.config.learning_rate
+batch_size = wandb.config.batch_size
+```
+
+### 3. Metric Logging
+
+```python
+# Log scalars
+wandb.log({"loss": 0.5, "accuracy": 0.92})
+
+# Log multiple metrics
+wandb.log({
+    "train/loss": train_loss,
+    "train/accuracy": train_acc,
+    "val/loss": val_loss,
+    "val/accuracy": val_acc,
+    "learning_rate": current_lr,
+    "epoch": epoch
+})
+
+# Log with custom x-axis
+wandb.log({"loss": loss}, step=global_step)
+
+# Log media (images, audio, video)
+wandb.log({"examples": [wandb.Image(img) for img in images]})
+
+# Log histograms
+wandb.log({"gradients": wandb.Histogram(gradients)})
+
+# Log tables
+table = wandb.Table(columns=["id", "prediction", "ground_truth"])
+wandb.log({"predictions": table})
+```
+
+### 4. Model Checkpointing
+
+```python
+import torch
+import wandb
+
+# Save model checkpoint
+checkpoint = {
+    'epoch': epoch,
+    'model_state_dict': model.state_dict(),
+    'optimizer_state_dict': optimizer.state_dict(),
+    'loss': loss,
+}
+
+torch.save(checkpoint, 'checkpoint.pth')
+
+# Upload to W&B
+wandb.save('checkpoint.pth')
+
+# Or use Artifacts (recommended)
+artifact = wandb.Artifact('model', type='model')
+artifact.add_file('checkpoint.pth')
+wandb.log_artifact(artifact)
+```
+
+## Hyperparameter Sweeps
+
+Automatically search for optimal hyperparameters.
+
+### Define Sweep Configuration
+
+```python
+sweep_config = {
+    'method': 'bayes',  # or 'grid', 'random'
+    'metric': {
+        'name': 'val/accuracy',
+        'goal': 'maximize'
+    },
+    'parameters': {
+        'learning_rate': {
+            'distribution': 'log_uniform_values',
+            'min': 1e-5,
+            'max': 1e-1
+        },
+        'batch_size': {
+            'values': [16, 32, 64, 128]
+        },
+        'optimizer': {
+            'values': ['adam', 'sgd', 'rmsprop']
+        },
+        'dropout': {
+            'distribution': 'uniform',
+            'min': 0.1,
+            'max': 0.5
+        }
+    }
+}
+
+# Initialize sweep
+sweep_id = wandb.sweep(sweep_config, project="my-project")
+```
+
+### Define Training Function
+
+```python
+def train():
+    # Initialize run
+    run = wandb.init()
+
+    # Access sweep parameters
+    lr = wandb.config.learning_rate
+    batch_size = wandb.config.batch_size
+    optimizer_name = wandb.config.optimizer
+
+    # Build model with sweep config
+    model = build_model(wandb.config)
+    optimizer = get_optimizer(optimizer_name, lr)
+
+    # Training loop
+    for epoch in range(NUM_EPOCHS):
+        train_loss = train_epoch(model, optimizer, batch_size)
+        val_acc = validate(model)
+
+        # Log metrics
+        wandb.log({
+            "train/loss": train_loss,
+            "val/accuracy": val_acc
+        })
+
+# Run sweep
+wandb.agent(sweep_id, function=train, count=50)  # Run 50 trials
+```
+
+### Sweep Strategies
+
+```python
+# Grid search - exhaustive
+sweep_config = {
+    'method': 'grid',
+    'parameters': {
+        'lr': {'values': [0.001, 0.01, 0.1]},
+        'batch_size': {'values': [16, 32, 64]}
+    }
+}
+
+# Random search
+sweep_config = {
+    'method': 'random',
+    'parameters': {
+        'lr': {'distribution': 'uniform', 'min': 0.0001, 'max': 0.1},
+        'dropout': {'distribution': 'uniform', 'min': 0.1, 'max': 0.5}
+    }
+}
+
+# Bayesian optimization (recommended)
+sweep_config = {
+    'method': 'bayes',
+    'metric': {'name': 'val/loss', 'goal': 'minimize'},
+    'parameters': {
+        'lr': {'distribution': 'log_uniform_values', 'min': 1e-5, 'max': 1e-1}
+    }
+}
+```
+
+## Artifacts
+
+Track datasets, models, and other files with lineage.
+
+### Log Artifacts
+
+```python
+# Create artifact
+artifact = wandb.Artifact(
+    name='training-dataset',
+    type='dataset',
+    description='ImageNet training split',
+    metadata={'size': '1.2M images', 'split': 'train'}
+)
+
+# Add files
+artifact.add_file('data/train.csv')
+artifact.add_dir('data/images/')
+
+# Log artifact
+wandb.log_artifact(artifact)
+```
+
+### Use Artifacts
+
+```python
+# Download and use artifact
+run = wandb.init(project="my-project")
+
+# Download artifact
+artifact = run.use_artifact('training-dataset:latest')
+artifact_dir = artifact.download()
+
+# Use the data
+data = load_data(f"{artifact_dir}/train.csv")
+```
+
+### Model Registry
+
+```python
+# Log model as artifact
+model_artifact = wandb.Artifact(
+    name='resnet50-model',
+    type='model',
+    metadata={'architecture': 'ResNet50', 'accuracy': 0.95}
+)
+
+model_artifact.add_file('model.pth')
+wandb.log_artifact(model_artifact, aliases=['best', 'production'])
+
+# Link to model registry
+run.link_artifact(model_artifact, 'model-registry/production-models')
+```
+
+## Integration Examples
+
+### HuggingFace Transformers
+
+```python
+from transformers import Trainer, TrainingArguments
+import wandb
+
+# Initialize W&B
+wandb.init(project="hf-transformers")
+
+# Training arguments with W&B
+training_args = TrainingArguments(
+    output_dir="./results",
+    report_to="wandb",  # Enable W&B logging
+    run_name="bert-finetuning",
+    logging_steps=100,
+    save_steps=500
+)
+
+# Trainer automatically logs to W&B
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=train_dataset,
+    eval_dataset=eval_dataset
+)
+
+trainer.train()
+```
+
+### PyTorch Lightning
+
+```python
+from pytorch_lightning import Trainer
+from pytorch_lightning.loggers import WandbLogger
+import wandb
+
+# Create W&B logger
+wandb_logger = WandbLogger(
+    project="lightning-demo",
+    log_model=True  # Log model checkpoints
+)
+
+# Use with Trainer
+trainer = Trainer(
+    logger=wandb_logger,
+    max_epochs=10
+)
+
+trainer.fit(model, datamodule=dm)
+```
+
+### Keras/TensorFlow
+
+```python
+import wandb
+from wandb.integration.keras import WandbMetricsLogger, WandbModelCheckpoint
+
+# Initialize
+wandb.init(project="keras-demo")
+
+# Add callbacks (the monolithic WandbCallback was removed;
+# use the dedicated callbacks from wandb.integration.keras instead)
+model.fit(
+    x_train, y_train,
+    validation_data=(x_val, y_val),
+    epochs=10,
+    callbacks=[
+        WandbMetricsLogger(),                        # Auto-logs metrics
+        WandbModelCheckpoint("models/model-{epoch}")  # Saves checkpoints
+    ]
+)
+```
+
+## Visualization & Analysis
+
+### Custom Charts
+
+```python
+# Log custom visualizations
+import matplotlib.pyplot as plt
+
+fig, ax = plt.subplots()
+ax.plot(x, y)
+wandb.log({"custom_plot": wandb.Image(fig)})
+
+# Log confusion matrix
+wandb.log({"conf_mat": wandb.plot.confusion_matrix(
+    probs=None,
+    y_true=ground_truth,
+    preds=predictions,
+    class_names=class_names
+)})
+```
+
+### Reports
+
+Create shareable reports in W&B UI:
+- Combine runs, charts, and text
+- Markdown support
+- Embeddable visualizations
+- Team collaboration
+
+## Best Practices
+
+### 1. Organize with Tags and Groups
+
+```python
+wandb.init(
+    project="my-project",
+    tags=["baseline", "resnet50", "imagenet"],
+    group="resnet-experiments",  # Group related runs
+    job_type="train"             # Type of job
+)
+```
+
+### 2. Log Everything Relevant
+
+```python
+# Log system metrics
+wandb.log({
+    "gpu/util": gpu_utilization,
+    "gpu/memory": gpu_memory_used,
+    "cpu/util": cpu_utilization
+})
+
+# Log code version
+wandb.log({"git_commit": git_commit_hash})
+
+# Log data splits
+wandb.log({
+    "data/train_size": len(train_dataset),
+    "data/val_size": len(val_dataset)
+})
+```
+
+### 3. Use Descriptive Names
+
+```python
+# ✅ Good: Descriptive run names
+wandb.init(
+    project="nlp-classification",
+    name="bert-base-lr0.001-bs32-epoch10"
+)
+
+# ❌ Bad: Generic names
+wandb.init(project="nlp", name="run1")
+```
+
+### 4. Save Important Artifacts
+
+```python
+# Save final model
+artifact = wandb.Artifact('final-model', type='model')
+artifact.add_file('model.pth')
+wandb.log_artifact(artifact)
+
+# Save predictions for analysis
+predictions_table = wandb.Table(
+    columns=["id", "input", "prediction", "ground_truth"],
+    data=predictions_data
+)
+wandb.log({"predictions": predictions_table})
+```
+
+### 5. Use Offline Mode for Unstable Connections
+
+```python
+import os
+
+# Enable offline mode
+os.environ["WANDB_MODE"] = "offline"
+
+wandb.init(project="my-project")
+# ... your code ...
+
+# Sync later
+# wandb sync <run_directory>
+```
+
+## Team Collaboration
+
+### Share Runs
+
+```python
+# Runs are automatically shareable via URL
+run = wandb.init(project="team-project")
+print(f"Share this URL: {run.url}")
+```
+
+### Team Projects
+
+- Create team account at wandb.ai
+- Add team members
+- Set project visibility (private/public)
+- Use team-level artifacts and model registry
+
+## Pricing
+
+- **Free**: Unlimited public projects, 100GB storage
+- **Academic**: Free for students/researchers
+- **Teams**: $50/seat/month, private projects, unlimited storage
+- **Enterprise**: Custom pricing, on-prem options
+
+## Resources
+
+- **Documentation**: https://docs.wandb.ai
+- **GitHub**: https://github.com/wandb/wandb (10.5k+ stars)
+- **Examples**: https://github.com/wandb/examples
+- **Community**: https://wandb.ai/community
+- **Discord**: https://wandb.me/discord
+
+## See Also
+
+- `references/sweeps.md` - Comprehensive hyperparameter optimization guide
+- `references/artifacts.md` - Data and model versioning patterns
+- `references/integrations.md` - Framework-specific examples

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/bundled/mlops/mlops-huggingface-hub.md

@@ -0,0 +1,100 @@
+---
+title: "Huggingface Hub — HuggingFace hf CLI: search/download/upload models, datasets"
+sidebar_label: "Huggingface Hub"
+description: "HuggingFace hf CLI: search/download/upload models, datasets"
+---
+
+{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
+
+# Huggingface Hub
+
+HuggingFace hf CLI: search/download/upload models, datasets.
+
+## Skill metadata
+
+| | |
+|---|---|
+| Source | Bundled (installed by default) |
+| Path | `skills/mlops/huggingface-hub` |
+| Version | `1.0.1` |
+| Author | Hugging Face |
+| License | MIT |
+| Platforms | linux, macos, windows |
+
+## Reference: full SKILL.md
+
+:::info
+The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
+:::
+
+# Hugging Face CLI (`hf`) Reference Guide
+
+The `hf` command is the modern command-line interface for interacting with the Hugging Face Hub, providing tools to manage repositories, models, datasets, and Spaces.
+
+> **IMPORTANT:** The `hf` command replaces the now deprecated `huggingface-cli` command.
+
+## Quick Start
+*   **Installation:** `curl -LsSf https://hf.co/cli/install.sh | bash -s`
+*   **Help:** Use `hf --help` to view all available functions and real-world examples.
+*   **Authentication:** Recommended via `HF_TOKEN` environment variable or the `--token` flag.
+
+---
+
+## Core Commands
+
+### General Operations
+*   `hf download REPO_ID`: Download files from the Hub.
+*   `hf upload REPO_ID`: Upload files/folders (recommended for single-commit; also handles resumable uploads of large directories).
+*   `hf upload-large-folder REPO_ID LOCAL_PATH`: **[Deprecated]** — use `hf upload` instead.
+*   `hf sync`: Sync files between a local directory and a bucket.
+*   `hf env` / `hf version`: View environment and version details.
+
+### Authentication (`hf auth`)
+*   `login` / `logout`: Manage sessions using tokens from [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens).
+*   `list` / `switch`: Manage and toggle between multiple stored access tokens.
+*   `whoami`: Identify the currently logged-in account.
+
+### Repository Management (`hf repos`)
+*   `create` / `delete`: Create or permanently remove repositories.
+*   `duplicate`: Clone a model, dataset, or Space to a new ID.
+*   `move`: Transfer a repository between namespaces.
+*   `branch` / `tag`: Manage Git-like references.
+*   `delete-files`: Remove specific files using patterns.
+
+---
+
+## Specialized Hub Interactions
+
+### Datasets & Models
+*   **Datasets:** `hf datasets list`, `info`, and `parquet` (list parquet URLs).
+*   **SQL Queries:** `hf datasets sql SQL` — Execute raw SQL via DuckDB against dataset parquet URLs.
+*   **Models:** `hf models list` and `info`.
+*   **Papers:** `hf papers ls` — View daily papers.
+
+### Discussions & Pull Requests (`hf discussions`)
+*   Manage the lifecycle of Hub contributions: `list`, `create`, `info`, `comment`, `close`, `reopen`, and `rename`.
+*   `diff`: View changes in a PR.
+*   `merge`: Finalize pull requests.
+
+### Infrastructure & Compute
+*   **Endpoints:** Deploy and manage Inference Endpoints (`deploy`, `pause`, `resume`, `scale-to-zero`, `catalog`).
+*   **Jobs:** Run compute tasks on HF infrastructure. Includes `hf jobs uv` for running Python scripts with inline dependencies and `stats` for resource monitoring.
+*   **Spaces:** Manage interactive apps. Includes `dev-mode` and `hot-reload` for Python files without full restarts.
+
+### Storage & Automation
+*   **Buckets:** Full S3-like bucket management (`create`, `cp`, `mv`, `rm`, `sync`).
+*   **Cache:** Manage local storage with `list`, `prune` (remove detached revisions), and `verify` (checksum checks).
+*   **Webhooks:** Automate workflows by managing Hub webhooks (`create`, `watch`, `enable`/`disable`).
+*   **Collections:** Organize Hub items into collections (`add-item`, `update`, `list`).
+
+---
+
+## Advanced Usage & Tips
+
+### Global Flags
+*   `--format json`: Produces machine-readable output for automation.
+*   `-q` / `--quiet`: Limits output to IDs only.
+
+### Extensions & Skills
+*   **Extensions:** Extend CLI functionality via GitHub repositories using `hf extensions install REPO_ID`.
+*   **Skills:** Manage AI assistant skills with `hf skills add`.