mi-crow 0.1.1.post14__tar.gz → 0.1.1.post16__tar.gz

{mi_crow-0.1.1.post14 → mi_crow-0.1.1.post16}/.github/workflows/publish.yml

@@ -20,7 +20,6 @@ jobs:
         uses: actions/checkout@v4
         with:
           fetch-depth: 0
-          # When triggered by a tag, checkout that specific tag
           ref: ${{ github.ref }}
 
       - name: Set up Python

{mi_crow-0.1.1.post14/src/mi_crow.egg-info → mi_crow-0.1.1.post16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mi-crow
-Version: 0.1.1.post14
+Version: 0.1.1.post16
 Summary: Engineer Thesis: Explaining and modifying LLM responses using SAE and concepts.
 Author-email: Hubert Kowalski <your.email@example.com>, Adam Kaniasty <adam.kaniasty@gmail.com>
 Requires-Python: >=3.10

mi_crow-0.1.1.post16/docs/experiments/index.md

@@ -0,0 +1,127 @@
+# Experiments
+
+This section provides detailed walkthroughs of sample experiments demonstrating real-world usage of mi-crow.
+
+## Overview
+
+Experiments showcase complete workflows from data collection through analysis, using real models and datasets. They demonstrate best practices and provide templates for your own research.
+
+## Available Experiments
+
+### [Verify SAE Training](verify-sae-training.md)
+
+Complete workflow for training and validating SAE models on the Bielik model using TinyStories dataset.
+
+**What it covers**:
+- Saving activations from a production model
+- Training SAEs with proper hyperparameters
+- Validating training success
+- Concept discovery and naming
+- Analysis and visualization
+
+**Time required**: Several hours (depending on hardware)
+
+**Prerequisites**:
+- Access to Bielik model or similar
+- Sufficient GPU memory
+- Understanding of basic SAE concepts
+
+### [SLURM SAE Pipeline](slurm-pipeline.md)
+
+Distributed training setup for large-scale SAE training on cluster environments.
+
+**What it covers**:
+- SLURM job configuration
+- Distributed activation saving
+- Large-scale SAE training
+- Resource management
+
+**Time required**: Days (cluster-dependent)
+
+**Prerequisites**:
+- Access to SLURM cluster
+- Understanding of cluster computing
+- Large-scale dataset
+
+## Experiment Structure
+
+Each experiment typically includes:
+
+1. **Setup**: Environment and dependencies
+2. **Data Collection**: Saving activations
+3. **Training**: SAE model training
+4. **Validation**: Verifying results
+5. **Analysis**: Understanding outcomes
+6. **Documentation**: Recording findings
+
+## Running Experiments
+
+### Prerequisites
+
+```bash
+# Install dependencies
+pip install -e .
+
+# Or with uv
+uv sync
+```
+
+### Basic Workflow
+
+```bash
+# 1. Navigate to experiment directory
+cd experiments/verify_sae_training
+
+# 2. Review README
+cat README.md
+
+# 3. Run scripts in order
+python 01_save_activations.py
+python 02_train_sae.py
+
+# 4. Open analysis notebooks
+jupyter notebook 03_analyze_training.ipynb
+```
+
+### Customization
+
+Experiments are designed to be customizable:
+
+- Modify model names
+- Adjust hyperparameters
+- Change dataset sources
+- Adapt to your hardware
+
+## Experiment Outputs
+
+Experiments produce:
+
+- **Saved activations**: Organized in store
+- **Trained models**: SAE checkpoints
+- **Analysis results**: Visualizations and metrics
+- **Documentation**: Findings and observations
+
+## Best Practices
+
+1. **Start small**: Test with limited data first
+2. **Monitor resources**: Watch memory and compute usage
+3. **Document changes**: Record any modifications
+4. **Save checkpoints**: Don't lose progress
+5. **Validate results**: Verify outputs make sense
+
+## Contributing Experiments
+
+If you create a new experiment:
+
+1. Create directory in `experiments/`
+2. Include README with description
+3. Provide runnable scripts/notebooks
+4. Document setup and requirements
+5. Share findings and observations
+
+## Next Steps
+
+- **[Verify SAE Training](verify-sae-training.md)** - Start with this experiment
+- **[User Guide](../guide/index.md)** - Learn fundamentals first
+- **[Examples](../guide/examples.md)** - Try examples before experiments
+

mi_crow-0.1.1.post16/docs/experiments/slurm-pipeline.md

@@ -0,0 +1,75 @@
+# SLURM SAE Pipeline
+
+This experiment demonstrates distributed training setup for large-scale SAE training on cluster environments using SLURM.
+
+## Overview
+
+This pipeline shows how to:
+- Configure SLURM jobs for activation saving
+- Set up distributed SAE training on clusters
+- Manage resources and job dependencies
+- Handle large-scale datasets efficiently
+
+## Prerequisites
+
+- Access to SLURM cluster
+- Understanding of cluster computing
+- Large-scale dataset
+- Sufficient cluster resources
+
+## Experiment Structure
+
+```
+slurm_sae_pipeline/
+├── 01_save_activations.py      # Activation saving script
+├── 02_train_sae.py              # SAE training script
+├── submit_save_activations.sh   # SLURM submission script for activations
+├── submit_train_sae.sh         # SLURM submission script for training
+└── README.md                    # Pipeline documentation
+```
+
+## Configuration
+
+### SLURM Job Configuration
+
+The submission scripts configure:
+- **Resources**: GPU allocation, memory, time limits
+- **Dependencies**: Job ordering (train after save)
+- **Environment**: Python environment setup
+- **Output**: Logging and error handling
+
+### mi-crow Configuration
+
+The Python scripts use standard mi-crow APIs:
+- `lm.activations.save()` for distributed activation saving
+- `SaeTrainer.train()` for SAE training
+- Store configuration for cluster filesystems
+
+## Workflow
+
+1. **Submit activation saving job**: Uses `submit_save_activations.sh`
+2. **Wait for completion**: Activation saving must complete first
+3. **Submit training job**: Uses `submit_train_sae.sh` (depends on step 1)
+4. **Monitor jobs**: Track progress through SLURM
+
+## Key Features
+
+- **Distributed processing**: Handles large datasets across cluster nodes
+- **Resource management**: Proper GPU and memory allocation
+- **Job dependencies**: Ensures correct execution order
+- **Error handling**: Robust failure recovery
+
+## Customization
+
+Adapt the pipeline for your cluster:
+- Modify resource requests in submission scripts
+- Adjust batch sizes for available memory
+- Configure store paths for cluster filesystem
+- Set appropriate time limits
+
+## Related Documentation
+
+- **[Training SAE Models](../guide/workflows/training-sae.md)** - SAE training guide
+- **[Saving Activations](../guide/workflows/saving-activations.md)** - Activation saving guide
+- **[Best Practices](../guide/best-practices.md)** - Performance optimization
+

mi_crow-0.1.1.post16/docs/experiments/verify-sae-training.md

@@ -0,0 +1,196 @@
+# Verify SAE Training Experiment
+
+This experiment demonstrates a complete workflow for training and validating SAE models on the Bielik model using the TinyStories dataset.
+
+## Overview
+
+This experiment walks through:
+1. Saving activations from a production model
+2. Training SAEs with proper hyperparameters
+3. Validating training success
+4. Concept discovery and naming
+5. Analysis and visualization
+
+## Prerequisites
+
+- Python 3.8+
+- PyTorch
+- Required packages: `mi_crow`, `torch`, `transformers`, `datasets`, `overcomplete`, `matplotlib`, `seaborn`
+- Access to Bielik model or similar
+- Sufficient GPU memory (or use CPU for smaller experiments)
+
+## Experiment Structure
+
+```
+verify_sae_training/
+├── 01_save_activations.py      # Step 1: Save activations from dataset
+├── 02_train_sae.py              # Step 2: Train SAE model
+├── 03_analyze_training.ipynb   # Step 3: Analyze training metrics and verify learning
+├── 04_name_sae_concepts.ipynb  # Step 4: Export top texts for each neuron
+├── 05_show_concepts.ipynb       # Step 5: Display and explore concepts
+├── observations.md              # Findings and observations
+└── README.md                    # Experiment documentation
+```
+
+## Step-by-Step Instructions
+
+### Step 1: Save Activations
+
+**File**: `01_save_activations.py`
+
+This script:
+- Loads the Bielik model
+- Uses resid_mid layer (post_attention_layernorm) at layer 16
+- Loads TinyStories dataset
+- Saves activations from the specified layer
+- Stores run ID in `store/run_id.txt`
+
+**Configuration**:
+- **Model**: `speakleash/Bielik-1.5B-v3.0-Instruct`
+- **Dataset**: `roneneldan/TinyStories` (train split)
+- **Layer**: `llamaforcausallm_model_layers_16_post_attention_layernorm`
+- **Store location**: `experiments/verify_sae_training/store/`
+
+**To change the layer**: Edit `LAYER_SIGNATURE` in the script (e.g., use `_0_` for first layer, `_31_` for last layer).
+
+**Run**:
+```bash
+cd experiments/verify_sae_training
+python 01_save_activations.py
+```
+
+### Step 2: Train SAE
+
+**File**: `02_train_sae.py`
+
+This script:
+- Loads the saved activations
+- Creates a TopKSAE model
+- Trains the SAE on the activations
+- Saves the trained model to `store/sae_model/topk_sae.pt`
+- Saves training history to `store/training_history.json`
+
+**Configuration**:
+- `N_LATENTS_MULTIPLIER`: Overcompleteness factor (default: 4x)
+- `TOP_K`: Sparsity parameter (default: 8)
+- `EPOCHS`: Number of training epochs (default: 100)
+- `BATCH_SIZE_TRAIN`: Training batch size (default: 1024)
+
+**Run**:
+```bash
+python 02_train_sae.py
+```
+
+### Step 3: Analyze Training
+
+**File**: `03_analyze_training.ipynb`
+
+This notebook demonstrates:
+- Accessing training history from `store/training_history.json`
+- Visualizing training metrics (loss, R², L0, dead features)
+- Validating SAE training success using mi-crow APIs
+- Analyzing reconstruction quality
+
+**Key validation checks**:
+- Loss should decrease over time
+- R² should increase (better reconstruction)
+- L0 should match expected TopK sparsity
+- Dead features should be minimal
+- Weight variance should be significant
+
+### Step 4: Export Top Texts
+
+**File**: `04_name_sae_concepts.ipynb`
+
+This notebook demonstrates:
+- Loading trained SAE using mi-crow APIs
+- Attaching SAE to language model with `lm.attach_sae()`
+- Enabling text tracking with `sae.concepts.enable_text_tracking()`
+- Collecting top texts using mi-crow's concept discovery features
+- Exporting results to JSON format
+
+**Output**: JSON file mapping neuron indices to top activating text snippets.
+
+### Step 5: Show Concepts
+
+**File**: `05_show_concepts.ipynb`
+
+This notebook demonstrates:
+- Loading exported top texts
+- Using mi-crow APIs to access concept data
+- Analyzing neuron activation patterns
+- Exploring concept relationships
+
+## Expected Outputs
+
+After running all steps, you'll have:
+
+- `store/run_id.txt` - Run ID for the activation saving run
+- `store/runs/<run_id>/` - Saved activations
+- `store/sae_model/topk_sae.pt` - Trained SAE model
+- `store/training_history.json` - Training metrics
+- `store/top_texts.json` - Exported top texts for each neuron
+
+## Analysis and Validation
+
+### Training Metrics
+
+Check that:
+- Loss decreases over epochs
+- R² score improves (target: > 0.9)
+- L0 matches expected TopK
+- Dead features < 10% of total
+
+### Concept Quality
+
+Verify that:
+- Top texts show coherent patterns
+- Neurons detect meaningful concepts
+- Concepts are interpretable
+- Multiple neurons may detect similar concepts (redundancy is normal)
+
+## Troubleshooting
+
+### Layer Signature Not Found
+
+If you get an error about layer signature:
+1. Run `01_save_activations.py` first to see available layers
+2. Copy one of the layer names
+3. Set `LAYER_SIGNATURE` in `01_save_activations.py`
+
+### Out of Memory
+
+If you run out of memory:
+- Reduce `DATA_LIMIT` in `01_save_activations.py`
+- Reduce `BATCH_SIZE_SAVE` in `01_save_activations.py`
+- Reduce `BATCH_SIZE_TRAIN` in `02_train_sae.py`
+- Use CPU instead of GPU (set `DEVICE = "cpu"`)
+
+### Missing Dependencies
+
+Install missing packages:
+```bash
+pip install torch transformers datasets overcomplete matplotlib seaborn
+```
+
+## Notes
+
+- The experiment uses a relatively small dataset (1000 samples) for quick testing
+- For production use, increase `DATA_LIMIT` and training epochs
+- See `observations.md` for findings and missing functionality notes
+
+## Related Documentation
+
+- **[Training SAE Models](../guide/workflows/training-sae.md)** - Detailed training guide
+- **[Concept Discovery](../guide/workflows/concept-discovery.md)** - Concept discovery workflow
+- **[Best Practices](../guide/best-practices.md)** - General best practices
+- **[Troubleshooting](../guide/troubleshooting.md)** - Common issues
+
+## Next Steps
+
+After completing this experiment:
+- Try different layers
+- Experiment with hyperparameters
+- Scale up to larger datasets
+- Create your own experiments
+