l0-python 0.4.0__tar.gz → 0.4.2__tar.gz

{l0_python-0.4.0 → l0_python-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: l0-python
-Version: 0.4.0
+Version: 0.4.2
 Summary: L0 regularization for sparse neural networks and intelligent sampling
 Author-email: PolicyEngine <hello@policyengine.org>
 License: MIT License
@@ -51,7 +51,7 @@ Requires-Dist: scikit-learn>=1.0; extra == "dev"
 Requires-Dist: yaml-changelog>=0.3.0; extra == "dev"
 Requires-Dist: twine>=4.0.0; extra == "dev"
 Provides-Extra: docs
-Requires-Dist: mystmd>=1.3; extra == "docs"
+Requires-Dist: jupyter-book>=1.0; extra == "docs"
 Requires-Dist: jupyter>=1.0; extra == "docs"
 Requires-Dist: notebook>=7.0; extra == "docs"
 Requires-Dist: ipykernel>=6.0; extra == "docs"
@@ -65,21 +65,29 @@ Dynamic: license-file
 
 # L0 Regularization
 
-A PyTorch implementation of L0 regularization for neural network sparsification and intelligent sampling, based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312).
+[![PyPI version](https://badge.fury.io/py/l0-python.svg)](https://pypi.org/project/l0-python/)
+[![CI](https://github.com/PolicyEngine/L0/actions/workflows/push.yml/badge.svg)](https://github.com/PolicyEngine/L0/actions)
 
-## Features
+A PyTorch implementation of L0 regularization based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312), with a **critical bug fix** to the original authors' implementation.
 
-- **Hard Concrete Distribution**: Differentiable approximation of L0 norm
-- **Sparse Neural Network Layers**: L0Linear, L0Conv2d with automatic pruning
-- **Intelligent Sampling**: Sample/feature selection gates for calibration
-- **L0L2 Combined Penalty**: Recommended approach to prevent overfitting
-- **Temperature Scheduling**: Annealing for improved convergence
-- **TDD Development**: Comprehensive test coverage
+## Why This Package?
+
+The original L0 implementation from AMLab-Amsterdam contains a bug in test-time gate computation where the temperature parameter is incorrectly omitted. This prevents proper sparsity and degrades performance. Our implementation corrects this error:
+
+```python
+# Original (buggy): gates never fully close
+pi = sigmoid(log_alpha)
+
+# Corrected: temperature division required for proper sparsity
+pi = sigmoid(log_alpha / temperature)
+```
+
+This fix enables gates to achieve true 0/1 values, producing exact sparsity as intended by the L0 formulation.
 
 ## Installation
 
 ```bash
-pip install l0
+pip install l0-python
 ```
 
 For development:
@@ -89,21 +97,76 @@ cd L0
 pip install -e .[dev]
 ```
 
-## Quick Start
+## Primary Use Case: Survey Calibration
+
+This package was developed for PolicyEngine's survey calibration system, where we select a sparse subset of survey households while matching population targets.
+
+```python
+import numpy as np
+from scipy import sparse as sp
+from l0.calibration import SparseCalibrationWeights
+
+# Setup: Q targets, N households
+Q, N = 200, 10000
+M = sp.random(Q, N, density=0.3, format="csr")  # Household characteristics
+y = np.random.uniform(1e6, 1e8, size=Q)          # Population targets
+
+# Initialize model
+model = SparseCalibrationWeights(
+    n_features=N,
+    beta=0.35,
+    gamma=-0.1,
+    zeta=1.1,
+    init_keep_prob=0.5,      # Start with 50% active probability
+    init_weights=1.0,        # Or pass array of initial weights
+    log_weight_jitter_sd=0.05,
+    device="cuda",           # GPU acceleration
+)
+
+# Train with L0+L2 regularization
+model.fit(
+    M=M,
+    y=y,
+    lambda_l0=1e-6,          # Controls sparsity level
+    lambda_l2=1e-8,          # Prevents weight explosion
+    lr=0.15,
+    epochs=2000,
+    loss_type="relative",    # Scale-invariant loss
+    verbose=True,
+    verbose_freq=100,
+)
+
+# Get results
+active = model.get_active_weights()
+print(f"Selected {active['count']} of {N} households ({100*active['count']/N:.1f}%)")
+print(f"Sparsity: {model.get_sparsity():.1%}")
+
+# Predict calibrated totals
+y_pred = model.predict(M)
+```
+
+### Key Features for Calibration
+
+- **Non-negative weights**: All weights constrained to be positive via log-space parameterization
+- **Sparse solutions**: L0 penalty directly minimizes the count of active weights
+- **Relative loss**: Scale-invariant loss for targets spanning orders of magnitude
+- **Group-wise averaging**: Balance loss contributions across target groups with different cardinalities
+- **GPU support**: CUDA acceleration for large-scale problems
+
+## Neural Network Sparsification
 
-### Neural Network Sparsification
+The package also supports traditional neural network pruning:
 
 ```python
 import torch
 from l0 import L0Linear, compute_l0l2_penalty, TemperatureScheduler, update_temperatures
 
-# Create a sparse model
 class SparseModel(torch.nn.Module):
     def __init__(self):
         super().__init__()
         self.fc1 = L0Linear(784, 256, init_sparsity=0.5)
         self.fc2 = L0Linear(256, 10, init_sparsity=0.7)
-    
+
     def forward(self, x):
         x = torch.relu(self.fc1(x))
         return self.fc2(x)
@@ -112,95 +175,71 @@ model = SparseModel()
 optimizer = torch.optim.Adam(model.parameters())
 scheduler = TemperatureScheduler(initial_temp=1.0, final_temp=0.1)
 
-# Training loop
 for epoch in range(100):
-    # Update temperature
     temp = scheduler.get_temperature(epoch)
     update_temperatures(model, temp)
-    
-    # Forward pass
+
     output = model(input_data)
     ce_loss = criterion(output, target)
-    
-    # Add L0L2 penalty
     penalty = compute_l0l2_penalty(model, l0_lambda=1e-3, l2_lambda=1e-4)
     loss = ce_loss + penalty
-    
-    # Backward pass
+
     optimizer.zero_grad()
     loss.backward()
     optimizer.step()
 ```
 
-### Intelligent Sample Selection
+### Available Layers
+
+- `L0Linear`: Fully connected layer with L0 gates on weights
+- `L0Conv2d`: 2D convolution with channel-wise L0 gates
+- `L0DepthwiseConv2d`: Depthwise convolution with L0 gates
+- `SparseMLP`: Multi-layer perceptron with built-in L0 regularization
+
+## Intelligent Sampling Gates
+
+Standalone gates for sample/feature selection:
 
 ```python
-from l0 import SampleGate, HybridGate
+from l0 import SampleGate, FeatureGate, HybridGate
 
-# Pure L0 selection
+# Select samples via learned gates
 gate = SampleGate(n_samples=10000, target_samples=1000)
 selected_data, indices = gate.select_samples(data)
 
-# Hybrid selection (25% L0, 75% random)
+# Select features
+gate = FeatureGate(n_features=1000, max_features=50)
+selected_data, indices = gate.select_features(data)
+
+# Hybrid: combine L0 selection with random sampling
 hybrid = HybridGate(
     n_items=10000,
-    l0_fraction=0.25,
-    random_fraction=0.75,
-    target_items=1000
+    l0_fraction=0.25,    # 25% via learned L0 gates
+    random_fraction=0.75 # 75% random for coverage
 )
 selected, indices, types = hybrid.select(data)
 ```
 
-### Feature Selection
-
-```python
-from l0 import FeatureGate
-
-# Select top features
-gate = FeatureGate(n_features=1000, max_features=50)
-selected_data, feature_indices = gate.select_features(data)
-
-# Get feature importance
-importance = gate.get_feature_importance()
-```
-
-## Integration with PolicyEngine
-
-This package is designed to work with PolicyEngine's calibration system:
-
-```python
-# In policyengine-us-data or similar
-from l0 import HardConcrete
-
-# Use for household selection in CPS calibration
-gates = HardConcrete(
-    len(household_weights),
-    temperature=0.25,
-    init_mean=0.999  # Start with most households
-)
+## How L0 Regularization Works
 
-# Apply gates during reweighting
-masked_weights = weights * gates()
-```
+Unlike post-hoc pruning (setting small weights to zero), L0 regularization integrates sparsity into the training objective:
 
-## Documentation
+1. **Stochastic gates**: Each weight has a learned gate parameter controlling activation probability
+2. **Hard Concrete distribution**: Enables differentiable 0/1 gate values during training
+3. **Expected L0 penalty**: Minimizes the expected number of active gates
+4. **Temperature annealing**: Gradually sharpens gates from soft to hard decisions
 
-Full documentation available at: https://policyengine.github.io/L0/
+The result: the network learns *which* weights should be zero as part of optimization, not as a post-processing step.
 
 ## Testing
 
-Run tests with:
 ```bash
 pytest tests/ -v --cov=l0
 ```
 
-## Acknowledgments
-
-This implementation is inspired by and builds upon the [original L0 regularization code](https://github.com/AMLab-Amsterdam/L0_regularization) by AMLab Amsterdam, which accompanied the paper by Louizos et al. (2018).
-
 ## Citation
 
-If you use this package, please cite:
+If you use this package, please cite the original paper:
 
 ```bibtex
 @article{louizos2017learning,
@@ -213,4 +252,4 @@ If you use this package, please cite:
 
 ## License
 
-MIT License - see [LICENSE](LICENSE) file for details.
+MIT License - see [LICENSE](LICENSE) for details.

l0_python-0.4.2/README.md

@@ -0,0 +1,190 @@
+# L0 Regularization
+
+[![PyPI version](https://badge.fury.io/py/l0-python.svg)](https://pypi.org/project/l0-python/)
+[![CI](https://github.com/PolicyEngine/L0/actions/workflows/push.yml/badge.svg)](https://github.com/PolicyEngine/L0/actions)
+
+A PyTorch implementation of L0 regularization based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312), with a **critical bug fix** to the original authors' implementation.
+
+## Why This Package?
+
+The original L0 implementation from AMLab-Amsterdam contains a bug in test-time gate computation where the temperature parameter is incorrectly omitted. This prevents proper sparsity and degrades performance. Our implementation corrects this error:
+
+```python
+# Original (buggy): gates never fully close
+pi = sigmoid(log_alpha)
+
+# Corrected: temperature division required for proper sparsity
+pi = sigmoid(log_alpha / temperature)
+```
+
+This fix enables gates to achieve true 0/1 values, producing exact sparsity as intended by the L0 formulation.
+
+## Installation
+
+```bash
+pip install l0-python
+```
+
+For development:
+```bash
+git clone https://github.com/PolicyEngine/L0.git
+cd L0
+pip install -e .[dev]
+```
+
+## Primary Use Case: Survey Calibration
+
+This package was developed for PolicyEngine's survey calibration system, where we select a sparse subset of survey households while matching population targets.
+
+```python
+import numpy as np
+from scipy import sparse as sp
+from l0.calibration import SparseCalibrationWeights
+
+# Setup: Q targets, N households
+Q, N = 200, 10000
+M = sp.random(Q, N, density=0.3, format="csr")  # Household characteristics
+y = np.random.uniform(1e6, 1e8, size=Q)          # Population targets
+
+# Initialize model
+model = SparseCalibrationWeights(
+    n_features=N,
+    beta=0.35,
+    gamma=-0.1,
+    zeta=1.1,
+    init_keep_prob=0.5,      # Start with 50% active probability
+    init_weights=1.0,        # Or pass array of initial weights
+    log_weight_jitter_sd=0.05,
+    device="cuda",           # GPU acceleration
+)
+
+# Train with L0+L2 regularization
+model.fit(
+    M=M,
+    y=y,
+    lambda_l0=1e-6,          # Controls sparsity level
+    lambda_l2=1e-8,          # Prevents weight explosion
+    lr=0.15,
+    epochs=2000,
+    loss_type="relative",    # Scale-invariant loss
+    verbose=True,
+    verbose_freq=100,
+)
+
+# Get results
+active = model.get_active_weights()
+print(f"Selected {active['count']} of {N} households ({100*active['count']/N:.1f}%)")
+print(f"Sparsity: {model.get_sparsity():.1%}")
+
+# Predict calibrated totals
+y_pred = model.predict(M)
+```
+
+### Key Features for Calibration
+
+- **Non-negative weights**: All weights constrained to be positive via log-space parameterization
+- **Sparse solutions**: L0 penalty directly minimizes the count of active weights
+- **Relative loss**: Scale-invariant loss for targets spanning orders of magnitude
+- **Group-wise averaging**: Balance loss contributions across target groups with different cardinalities
+- **GPU support**: CUDA acceleration for large-scale problems
+
+## Neural Network Sparsification
+
+The package also supports traditional neural network pruning:
+
+```python
+import torch
+from l0 import L0Linear, compute_l0l2_penalty, TemperatureScheduler, update_temperatures
+
+class SparseModel(torch.nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.fc1 = L0Linear(784, 256, init_sparsity=0.5)
+        self.fc2 = L0Linear(256, 10, init_sparsity=0.7)
+
+    def forward(self, x):
+        x = torch.relu(self.fc1(x))
+        return self.fc2(x)
+
+model = SparseModel()
+optimizer = torch.optim.Adam(model.parameters())
+scheduler = TemperatureScheduler(initial_temp=1.0, final_temp=0.1)
+
+for epoch in range(100):
+    temp = scheduler.get_temperature(epoch)
+    update_temperatures(model, temp)
+
+    output = model(input_data)
+    ce_loss = criterion(output, target)
+    penalty = compute_l0l2_penalty(model, l0_lambda=1e-3, l2_lambda=1e-4)
+    loss = ce_loss + penalty
+
+    optimizer.zero_grad()
+    loss.backward()
+    optimizer.step()
+```
+
+### Available Layers
+
+- `L0Linear`: Fully connected layer with L0 gates on weights
+- `L0Conv2d`: 2D convolution with channel-wise L0 gates
+- `L0DepthwiseConv2d`: Depthwise convolution with L0 gates
+- `SparseMLP`: Multi-layer perceptron with built-in L0 regularization
+
+## Intelligent Sampling Gates
+
+Standalone gates for sample/feature selection:
+
+```python
+from l0 import SampleGate, FeatureGate, HybridGate
+
+# Select samples via learned gates
+gate = SampleGate(n_samples=10000, target_samples=1000)
+selected_data, indices = gate.select_samples(data)
+
+# Select features
+gate = FeatureGate(n_features=1000, max_features=50)
+selected_data, indices = gate.select_features(data)
+
+# Hybrid: combine L0 selection with random sampling
+hybrid = HybridGate(
+    n_items=10000,
+    l0_fraction=0.25,    # 25% via learned L0 gates
+    random_fraction=0.75 # 75% random for coverage
+)
+selected, indices, types = hybrid.select(data)
+```
+
+## How L0 Regularization Works
+
+Unlike post-hoc pruning (setting small weights to zero), L0 regularization integrates sparsity into the training objective:
+
+1. **Stochastic gates**: Each weight has a learned gate parameter controlling activation probability
+2. **Hard Concrete distribution**: Enables differentiable 0/1 gate values during training
+3. **Expected L0 penalty**: Minimizes the expected number of active gates
+4. **Temperature annealing**: Gradually sharpens gates from soft to hard decisions
+
+The result: the network learns *which* weights should be zero as part of optimization, not as a post-processing step.
+
+## Testing
+
+```bash
+pytest tests/ -v --cov=l0
+```
+
+## Citation
+
+If you use this package, please cite the original paper:
+
+```bibtex
+@article{louizos2017learning,
+  title={Learning Sparse Neural Networks through L0 Regularization},
+  author={Louizos, Christos and Welling, Max and Kingma, Diederik P},
+  journal={arXiv preprint arXiv:1712.01312},
+  year={2017}
+}
+```
+
+## License
+
+MIT License - see [LICENSE](LICENSE) for details.

{l0_python-0.4.0 → l0_python-0.4.2}/l0/calibration.py

@@ -444,9 +444,9 @@ class SparseCalibrationWeights(nn.Module):
                                 rel_errors[group_mask].mean().item()
                             )
                             group_losses.append(group_mean_err)
-                        mean_group_loss = np.mean(group_losses)
+                        mean_group_mare = np.mean(group_losses)
                     else:
-                        mean_group_loss = mean_rel_err
+                        mean_group_mare = mean_rel_err
 
                     # Calculate sparsity percentage
                     sparsity_pct = 100 * (
@@ -461,7 +461,7 @@ class SparseCalibrationWeights(nn.Module):
                     if target_groups is not None:
                         print(
                             f"Epoch {epoch+1:4d}: "
-                            f"mean_group_loss={mean_group_loss:.1%}, "
+                            f"mean_group_mare={mean_group_mare:.4%}, "
                             f"max_error={max_rel_err:.1%}, "
                             f"total_loss={actual_total_loss:.3f}, "
                             f"active={active_info['count']:4d}/{self.n_features} ({sparsity_pct:.1f}% sparse)"
@@ -469,7 +469,7 @@ class SparseCalibrationWeights(nn.Module):
                     else:
                         print(
                             f"Epoch {epoch+1:4d}: "
-                            f"mean_error={mean_rel_err:.1%}, "
+                            f"mean_error={mean_rel_err:.4%}, "
                             f"max_error={max_rel_err:.1%}, "
                             f"total_loss={actual_total_loss:.3f}, "
                             f"active={active_info['count']:4d}/{self.n_features} ({sparsity_pct:.1f}% sparse)"

{l0_python-0.4.0 → l0_python-0.4.2}/l0_python.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: l0-python
-Version: 0.4.0
+Version: 0.4.2
 Summary: L0 regularization for sparse neural networks and intelligent sampling
 Author-email: PolicyEngine <hello@policyengine.org>
 License: MIT License
@@ -51,7 +51,7 @@ Requires-Dist: scikit-learn>=1.0; extra == "dev"
 Requires-Dist: yaml-changelog>=0.3.0; extra == "dev"
 Requires-Dist: twine>=4.0.0; extra == "dev"
 Provides-Extra: docs
-Requires-Dist: mystmd>=1.3; extra == "docs"
+Requires-Dist: jupyter-book>=1.0; extra == "docs"
 Requires-Dist: jupyter>=1.0; extra == "docs"
 Requires-Dist: notebook>=7.0; extra == "docs"
 Requires-Dist: ipykernel>=6.0; extra == "docs"
@@ -65,21 +65,29 @@ Dynamic: license-file
 
 # L0 Regularization
 
-A PyTorch implementation of L0 regularization for neural network sparsification and intelligent sampling, based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312).
+[![PyPI version](https://badge.fury.io/py/l0-python.svg)](https://pypi.org/project/l0-python/)
+[![CI](https://github.com/PolicyEngine/L0/actions/workflows/push.yml/badge.svg)](https://github.com/PolicyEngine/L0/actions)
 
-## Features
+A PyTorch implementation of L0 regularization based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312), with a **critical bug fix** to the original authors' implementation.
 
-- **Hard Concrete Distribution**: Differentiable approximation of L0 norm
-- **Sparse Neural Network Layers**: L0Linear, L0Conv2d with automatic pruning
-- **Intelligent Sampling**: Sample/feature selection gates for calibration
-- **L0L2 Combined Penalty**: Recommended approach to prevent overfitting
-- **Temperature Scheduling**: Annealing for improved convergence
-- **TDD Development**: Comprehensive test coverage
+## Why This Package?
+
+The original L0 implementation from AMLab-Amsterdam contains a bug in test-time gate computation where the temperature parameter is incorrectly omitted. This prevents proper sparsity and degrades performance. Our implementation corrects this error:
+
+```python
+# Original (buggy): gates never fully close
+pi = sigmoid(log_alpha)
+
+# Corrected: temperature division required for proper sparsity
+pi = sigmoid(log_alpha / temperature)
+```
+
+This fix enables gates to achieve true 0/1 values, producing exact sparsity as intended by the L0 formulation.
 
 ## Installation
 
 ```bash
-pip install l0
+pip install l0-python
 ```
 
 For development:
@@ -89,21 +97,76 @@ cd L0
 pip install -e .[dev]
 ```
 
-## Quick Start
+## Primary Use Case: Survey Calibration
+
+This package was developed for PolicyEngine's survey calibration system, where we select a sparse subset of survey households while matching population targets.
+
+```python
+import numpy as np
+from scipy import sparse as sp
+from l0.calibration import SparseCalibrationWeights
+
+# Setup: Q targets, N households
+Q, N = 200, 10000
+M = sp.random(Q, N, density=0.3, format="csr")  # Household characteristics
+y = np.random.uniform(1e6, 1e8, size=Q)          # Population targets
+
+# Initialize model
+model = SparseCalibrationWeights(
+    n_features=N,
+    beta=0.35,
+    gamma=-0.1,
+    zeta=1.1,
+    init_keep_prob=0.5,      # Start with 50% active probability
+    init_weights=1.0,        # Or pass array of initial weights
+    log_weight_jitter_sd=0.05,
+    device="cuda",           # GPU acceleration
+)
+
+# Train with L0+L2 regularization
+model.fit(
+    M=M,
+    y=y,
+    lambda_l0=1e-6,          # Controls sparsity level
+    lambda_l2=1e-8,          # Prevents weight explosion
+    lr=0.15,
+    epochs=2000,
+    loss_type="relative",    # Scale-invariant loss
+    verbose=True,
+    verbose_freq=100,
+)
+
+# Get results
+active = model.get_active_weights()
+print(f"Selected {active['count']} of {N} households ({100*active['count']/N:.1f}%)")
+print(f"Sparsity: {model.get_sparsity():.1%}")
+
+# Predict calibrated totals
+y_pred = model.predict(M)
+```
+
+### Key Features for Calibration
+
+- **Non-negative weights**: All weights constrained to be positive via log-space parameterization
+- **Sparse solutions**: L0 penalty directly minimizes the count of active weights
+- **Relative loss**: Scale-invariant loss for targets spanning orders of magnitude
+- **Group-wise averaging**: Balance loss contributions across target groups with different cardinalities
+- **GPU support**: CUDA acceleration for large-scale problems
+
+## Neural Network Sparsification
 
-### Neural Network Sparsification
+The package also supports traditional neural network pruning:
 
 ```python
 import torch
 from l0 import L0Linear, compute_l0l2_penalty, TemperatureScheduler, update_temperatures
 
-# Create a sparse model
 class SparseModel(torch.nn.Module):
     def __init__(self):
         super().__init__()
         self.fc1 = L0Linear(784, 256, init_sparsity=0.5)
         self.fc2 = L0Linear(256, 10, init_sparsity=0.7)
-    
+
     def forward(self, x):
         x = torch.relu(self.fc1(x))
         return self.fc2(x)
@@ -112,95 +175,71 @@ model = SparseModel()
 optimizer = torch.optim.Adam(model.parameters())
 scheduler = TemperatureScheduler(initial_temp=1.0, final_temp=0.1)
 
-# Training loop
 for epoch in range(100):
-    # Update temperature
     temp = scheduler.get_temperature(epoch)
     update_temperatures(model, temp)
-    
-    # Forward pass
+
     output = model(input_data)
     ce_loss = criterion(output, target)
-    
-    # Add L0L2 penalty
     penalty = compute_l0l2_penalty(model, l0_lambda=1e-3, l2_lambda=1e-4)
     loss = ce_loss + penalty
-    
-    # Backward pass
+
     optimizer.zero_grad()
     loss.backward()
     optimizer.step()
 ```
 
-### Intelligent Sample Selection
+### Available Layers
+
+- `L0Linear`: Fully connected layer with L0 gates on weights
+- `L0Conv2d`: 2D convolution with channel-wise L0 gates
+- `L0DepthwiseConv2d`: Depthwise convolution with L0 gates
+- `SparseMLP`: Multi-layer perceptron with built-in L0 regularization
+
+## Intelligent Sampling Gates
+
+Standalone gates for sample/feature selection:
 
 ```python
-from l0 import SampleGate, HybridGate
+from l0 import SampleGate, FeatureGate, HybridGate
 
-# Pure L0 selection
+# Select samples via learned gates
 gate = SampleGate(n_samples=10000, target_samples=1000)
 selected_data, indices = gate.select_samples(data)
 
-# Hybrid selection (25% L0, 75% random)
+# Select features
+gate = FeatureGate(n_features=1000, max_features=50)
+selected_data, indices = gate.select_features(data)
+
+# Hybrid: combine L0 selection with random sampling
 hybrid = HybridGate(
     n_items=10000,
-    l0_fraction=0.25,
-    random_fraction=0.75,
-    target_items=1000
+    l0_fraction=0.25,    # 25% via learned L0 gates
+    random_fraction=0.75 # 75% random for coverage
 )
 selected, indices, types = hybrid.select(data)
 ```
 
-### Feature Selection
-
-```python
-from l0 import FeatureGate
-
-# Select top features
-gate = FeatureGate(n_features=1000, max_features=50)
-selected_data, feature_indices = gate.select_features(data)
-
-# Get feature importance
-importance = gate.get_feature_importance()
-```
-
-## Integration with PolicyEngine
-
-This package is designed to work with PolicyEngine's calibration system:
-
-```python
-# In policyengine-us-data or similar
-from l0 import HardConcrete
-
-# Use for household selection in CPS calibration
-gates = HardConcrete(
-    len(household_weights),
-    temperature=0.25,
-    init_mean=0.999  # Start with most households
-)
+## How L0 Regularization Works
 
-# Apply gates during reweighting
-masked_weights = weights * gates()
-```
+Unlike post-hoc pruning (setting small weights to zero), L0 regularization integrates sparsity into the training objective:
 
-## Documentation
+1. **Stochastic gates**: Each weight has a learned gate parameter controlling activation probability
+2. **Hard Concrete distribution**: Enables differentiable 0/1 gate values during training
+3. **Expected L0 penalty**: Minimizes the expected number of active gates
+4. **Temperature annealing**: Gradually sharpens gates from soft to hard decisions
 
-Full documentation available at: https://policyengine.github.io/L0/
+The result: the network learns *which* weights should be zero as part of optimization, not as a post-processing step.
 
 ## Testing
 
-Run tests with:
 ```bash
 pytest tests/ -v --cov=l0
 ```
 
-## Acknowledgments
-
-This implementation is inspired by and builds upon the [original L0 regularization code](https://github.com/AMLab-Amsterdam/L0_regularization) by AMLab Amsterdam, which accompanied the paper by Louizos et al. (2018).
-
 ## Citation
 
-If you use this package, please cite:
+If you use this package, please cite the original paper:
 
 ```bibtex
 @article{louizos2017learning,
@@ -213,4 +252,4 @@ If you use this package, please cite:
 
 ## License
 
-MIT License - see [LICENSE](LICENSE) file for details.
+MIT License - see [LICENSE](LICENSE) for details.

{l0_python-0.4.0 → l0_python-0.4.2}/l0_python.egg-info/requires.txt

@@ -12,7 +12,7 @@ yaml-changelog>=0.3.0
 twine>=4.0.0
 
 [docs]
-mystmd>=1.3
+jupyter-book>=1.0
 jupyter>=1.0
 notebook>=7.0
 ipykernel>=6.0

{l0_python-0.4.0 → l0_python-0.4.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "l0-python"
-version = "0.4.0"
+version = "0.4.2"
 description = "L0 regularization for sparse neural networks and intelligent sampling"
 readme = "README.md"
 authors = [
@@ -38,7 +38,7 @@ dev = [
     "twine>=4.0.0",
 ]
 docs = [
-    "mystmd>=1.3",
+    "jupyter-book>=1.0",
     "jupyter>=1.0",
     "notebook>=7.0",
     "ipykernel>=6.0",

{l0_python-0.4.0 → l0_python-0.4.2}/tests/test_calibration.py

@@ -247,6 +247,232 @@ class TestSparseCalibrationWeights:
                 weights_with_l2.max() <= weights_no_l2.max() * 2.0
             ), "L2 should prevent extreme weights"
 
+    def test_pure_l2_penalty(self):
+        """Test pure L2 regularization without L0."""
+        N = 50
+        Q = 30
+
+        np.random.seed(42)
+        torch.manual_seed(42)
+
+        # Create underdetermined problem where L2 helps regularize
+        M = sp.random(Q, N, density=0.5, format="csr")
+        M.data = np.abs(M.data) * 2  # Scale up to create larger weights
+        y = np.ones(Q) * 10  # Simple target
+
+        # Train without any regularization
+        model_no_reg = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,  # Start with larger weights
+            init_keep_prob=0.999,  # Keep all weights active
+        )
+        model_no_reg.fit(
+            M,
+            y,
+            lambda_l0=0.0,  # No L0
+            lambda_l2=0.0,  # No L2
+            epochs=1000,
+            lr=0.02,
+            verbose=False,
+        )
+
+        # Train with strong L2 regularization
+        model_l2_only = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,  # Same starting point
+            init_keep_prob=0.999,  # Keep all weights active
+        )
+        model_l2_only.fit(
+            M,
+            y,
+            lambda_l0=0.0,  # No L0
+            lambda_l2=10.0,  # Very strong L2 to see clear effect
+            epochs=1000,
+            lr=0.02,
+            verbose=False,
+        )
+
+        with torch.no_grad():
+            weights_no_reg = model_no_reg.get_weights(deterministic=True)
+            weights_l2 = model_l2_only.get_weights(deterministic=True)
+
+            # Get non-zero weights
+            active_no_reg = weights_no_reg[weights_no_reg > 1e-8]
+            active_l2 = weights_l2[weights_l2 > 1e-8]
+
+            # L2 should keep most weights active (no sparsity from L2)
+            assert (
+                len(active_l2) >= N * 0.9
+            ), f"Pure L2 should not induce sparsity, got {len(active_l2)}/{N}"
+
+            # L2 norm should be smaller with regularization
+            l2_norm_no_reg = (weights_no_reg**2).sum().sqrt()
+            l2_norm_with_l2 = (weights_l2**2).sum().sqrt()
+            assert (
+                l2_norm_with_l2 < l2_norm_no_reg
+            ), f"L2 regularization should reduce L2 norm: {l2_norm_with_l2:.2f} vs {l2_norm_no_reg:.2f}"
+
+            # Check coefficient of variation (CV = std/mean) is lower with L2
+            # This captures that L2 shrinks weights toward each other
+            if active_no_reg.mean() > 1e-6 and active_l2.mean() > 1e-6:
+                cv_no_reg = active_no_reg.std() / active_no_reg.mean()
+                cv_l2 = active_l2.std() / active_l2.mean()
+                assert (
+                    cv_l2 < cv_no_reg * 1.2
+                ), f"L2 should reduce relative variation: CV {cv_l2:.2f} vs {cv_no_reg:.2f}"
+
+            # L2 should prevent extreme weights
+            assert (
+                weights_l2.max() < weights_no_reg.max() * 1.5
+            ), f"L2 should limit max weights: {weights_l2.max():.2f} vs {weights_no_reg.max():.2f}"
+
+            # Both should still fit reasonably well
+            y_pred_no_reg = model_no_reg.predict(M).cpu().numpy()
+            y_pred_l2 = model_l2_only.predict(M).cpu().numpy()
+
+            error_no_reg = np.abs((y - y_pred_no_reg) / y).mean()
+            error_l2 = np.abs((y - y_pred_l2) / y).mean()
+
+            # L2 model may have slightly worse fit due to regularization
+            assert (
+                error_l2 < 0.5
+            ), f"L2 model should still fit reasonably: {error_l2:.3f}"
+            # But the trade-off is worth it for regularization
+
+    def test_l0_l2_combination(self):
+        """Test that combining L0 and L2 gives both sparsity and regularization."""
+        N = 100  # features
+        Q = 50  # targets
+
+        np.random.seed(42)
+        torch.manual_seed(42)
+
+        # Create problem with potential for overfitting
+        M = sp.random(Q, N, density=0.4, format="csr")
+        M.data = np.abs(M.data) * 3
+        y = np.random.uniform(5, 20, size=Q)
+
+        # Model 1: Only L0 (sparsity without weight regularization)
+        model_l0_only = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,
+            init_keep_prob=0.5,  # Start with 50% probability
+        )
+        model_l0_only.fit(
+            M,
+            y,
+            lambda_l0=0.01,  # Stronger L0 for sparsity
+            lambda_l2=0.0,  # No L2
+            epochs=2000,
+            lr=0.02,
+            verbose=False,
+        )
+
+        # Model 2: Only L2 (weight regularization without sparsity)
+        model_l2_only = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,
+            init_keep_prob=0.999,  # Keep all weights active
+        )
+        model_l2_only.fit(
+            M,
+            y,
+            lambda_l0=0.0,  # No L0
+            lambda_l2=0.1,  # Moderate L2
+            epochs=1500,
+            lr=0.02,
+            verbose=False,
+        )
+
+        # Model 3: Combined L0+L2 (both sparsity and weight regularization)
+        model_l0_l2 = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,
+            init_keep_prob=0.5,  # Same starting point as L0-only
+        )
+        model_l0_l2.fit(
+            M,
+            y,
+            lambda_l0=0.01,  # Same L0 as model 1
+            lambda_l2=0.1,  # Add L2 regularization
+            epochs=2000,
+            lr=0.02,
+            verbose=False,
+        )
+
+        with torch.no_grad():
+            # Get weights and stats for all models
+            weights_l0_only = model_l0_only.get_weights(deterministic=True)
+            weights_l2_only = model_l2_only.get_weights(deterministic=True)
+            weights_l0_l2 = model_l0_l2.get_weights(deterministic=True)
+
+            # Count active weights
+            active_l0_only = (weights_l0_only > 1e-6).sum().item()
+            active_l2_only = (weights_l2_only > 1e-6).sum().item()
+            active_l0_l2 = (weights_l0_l2 > 1e-6).sum().item()
+
+            # L0-only should have sparsity
+            assert (
+                active_l0_only < N * 0.8
+            ), f"L0 should induce sparsity: {active_l0_only}/{N} active"
+
+            # L2-only should have no/little sparsity
+            assert (
+                active_l2_only > N * 0.9
+            ), f"L2 alone shouldn't induce sparsity: {active_l2_only}/{N} active"
+
+            # L0+L2 should have sparsity (from L0)
+            assert (
+                active_l0_l2 < N * 0.8
+            ), f"L0+L2 should have sparsity: {active_l0_l2}/{N} active"
+
+            # Among active weights, L0+L2 should have smaller magnitudes than L0-only (from L2)
+            active_mask_l0_only = weights_l0_only > 1e-6
+            active_mask_l0_l2 = weights_l0_l2 > 1e-6
+
+            if active_mask_l0_only.any() and active_mask_l0_l2.any():
+                # Compare L2 norms of active weights
+                l2_norm_l0_only = (
+                    (weights_l0_only[active_mask_l0_only] ** 2).sum().sqrt()
+                )
+                l2_norm_l0_l2 = (
+                    (weights_l0_l2[active_mask_l0_l2] ** 2).sum().sqrt()
+                )
+
+                # L0+L2 should have smaller weight norms than L0-only
+                # (L2 regularization effect on top of sparsity)
+                assert (
+                    l2_norm_l0_l2 < l2_norm_l0_only * 1.2
+                ), f"L0+L2 should have controlled weights: {l2_norm_l0_l2:.2f} vs L0-only {l2_norm_l0_only:.2f}"
+
+            # Check prediction quality for all models
+            y_pred_l0_only = model_l0_only.predict(M).cpu().numpy()
+            y_pred_l2_only = model_l2_only.predict(M).cpu().numpy()
+            y_pred_l0_l2 = model_l0_l2.predict(M).cpu().numpy()
+
+            error_l0_only = np.abs((y - y_pred_l0_only) / (y + 1)).mean()
+            error_l2_only = np.abs((y - y_pred_l2_only) / (y + 1)).mean()
+            error_l0_l2 = np.abs((y - y_pred_l0_l2) / (y + 1)).mean()
+
+            # All should fit reasonably well (L2-only may have slightly worse fit due to regularization)
+            assert (
+                error_l0_only < 0.35
+            ), f"L0-only should fit well: {error_l0_only:.3f}"
+            assert (
+                error_l2_only < 0.35
+            ), f"L2-only should fit well: {error_l2_only:.3f}"
+            assert (
+                error_l0_l2 < 0.35
+            ), f"L0+L2 should fit well: {error_l0_l2:.3f}"
+
+            print(
+                f"\nL0-only: {active_l0_only}/{N} active, error={error_l0_only:.3f}"
+            )
+            print(
+                f"L2-only: {active_l2_only}/{N} active, error={error_l2_only:.3f}"
+            )
+            print(f"L0+L2: {active_l0_l2}/{N} active, error={error_l0_l2:.3f}")
+
     def test_group_wise_averaging(self):
         """Test that group-wise averaging balances loss contributions."""
         N = 100  # features (households)

l0_python-0.4.0/README.md

@@ -1,151 +0,0 @@
-# L0 Regularization
-
-A PyTorch implementation of L0 regularization for neural network sparsification and intelligent sampling, based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312).
-
-## Features
-
-- **Hard Concrete Distribution**: Differentiable approximation of L0 norm
-- **Sparse Neural Network Layers**: L0Linear, L0Conv2d with automatic pruning
-- **Intelligent Sampling**: Sample/feature selection gates for calibration
-- **L0L2 Combined Penalty**: Recommended approach to prevent overfitting
-- **Temperature Scheduling**: Annealing for improved convergence
-- **TDD Development**: Comprehensive test coverage
-
-## Installation
-
-```bash
-pip install l0
-```
-
-For development:
-```bash
-git clone https://github.com/PolicyEngine/L0.git
-cd L0
-pip install -e .[dev]
-```
-
-## Quick Start
-
-### Neural Network Sparsification
-
-```python
-import torch
-from l0 import L0Linear, compute_l0l2_penalty, TemperatureScheduler, update_temperatures
-
-# Create a sparse model
-class SparseModel(torch.nn.Module):
-    def __init__(self):
-        super().__init__()
-        self.fc1 = L0Linear(784, 256, init_sparsity=0.5)
-        self.fc2 = L0Linear(256, 10, init_sparsity=0.7)
-    
-    def forward(self, x):
-        x = torch.relu(self.fc1(x))
-        return self.fc2(x)
-
-model = SparseModel()
-optimizer = torch.optim.Adam(model.parameters())
-scheduler = TemperatureScheduler(initial_temp=1.0, final_temp=0.1)
-
-# Training loop
-for epoch in range(100):
-    # Update temperature
-    temp = scheduler.get_temperature(epoch)
-    update_temperatures(model, temp)
-    
-    # Forward pass
-    output = model(input_data)
-    ce_loss = criterion(output, target)
-    
-    # Add L0L2 penalty
-    penalty = compute_l0l2_penalty(model, l0_lambda=1e-3, l2_lambda=1e-4)
-    loss = ce_loss + penalty
-    
-    # Backward pass
-    optimizer.zero_grad()
-    loss.backward()
-    optimizer.step()
-```
-
-### Intelligent Sample Selection
-
-```python
-from l0 import SampleGate, HybridGate
-
-# Pure L0 selection
-gate = SampleGate(n_samples=10000, target_samples=1000)
-selected_data, indices = gate.select_samples(data)
-
-# Hybrid selection (25% L0, 75% random)
-hybrid = HybridGate(
-    n_items=10000,
-    l0_fraction=0.25,
-    random_fraction=0.75,
-    target_items=1000
-)
-selected, indices, types = hybrid.select(data)
-```
-
-### Feature Selection
-
-```python
-from l0 import FeatureGate
-
-# Select top features
-gate = FeatureGate(n_features=1000, max_features=50)
-selected_data, feature_indices = gate.select_features(data)
-
-# Get feature importance
-importance = gate.get_feature_importance()
-```
-
-## Integration with PolicyEngine
-
-This package is designed to work with PolicyEngine's calibration system:
-
-```python
-# In policyengine-us-data or similar
-from l0 import HardConcrete
-
-# Use for household selection in CPS calibration
-gates = HardConcrete(
-    len(household_weights),
-    temperature=0.25,
-    init_mean=0.999  # Start with most households
-)
-
-# Apply gates during reweighting
-masked_weights = weights * gates()
-```
-
-## Documentation
-
-Full documentation available at: https://policyengine.github.io/L0/
-
-## Testing
-
-Run tests with:
-```bash
-pytest tests/ -v --cov=l0
-```
-
-## Acknowledgments
-
-This implementation is inspired by and builds upon the [original L0 regularization code](https://github.com/AMLab-Amsterdam/L0_regularization) by AMLab Amsterdam, which accompanied the paper by Louizos et al. (2018).
-
-## Citation
-
-If you use this package, please cite:
-
-```bibtex
-@article{louizos2017learning,
-  title={Learning Sparse Neural Networks through L0 Regularization},
-  author={Louizos, Christos and Welling, Max and Kingma, Diederik P},
-  journal={arXiv preprint arXiv:1712.01312},
-  year={2017}
-}
-```
-
-## License
-
-MIT License - see [LICENSE](LICENSE) file for details.