l0-python 0.4.1__tar.gz → 0.4.3__tar.gz

{l0_python-0.4.1/l0_python.egg-info → l0_python-0.4.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: l0-python
-Version: 0.4.1
+Version: 0.4.3
 Summary: L0 regularization for sparse neural networks and intelligent sampling
 Author-email: PolicyEngine <hello@policyengine.org>
 License: MIT License
@@ -65,21 +65,15 @@ Dynamic: license-file
 
 # L0 Regularization
 
-A PyTorch implementation of L0 regularization for neural network sparsification and intelligent sampling, based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312).
+[![PyPI version](https://badge.fury.io/py/l0-python.svg)](https://pypi.org/project/l0-python/)
+[![CI](https://github.com/PolicyEngine/L0/actions/workflows/push.yml/badge.svg)](https://github.com/PolicyEngine/L0/actions)
 
-## Features
-
-- **Hard Concrete Distribution**: Differentiable approximation of L0 norm
-- **Sparse Neural Network Layers**: L0Linear, L0Conv2d with automatic pruning
-- **Intelligent Sampling**: Sample/feature selection gates for calibration
-- **L0L2 Combined Penalty**: Recommended approach to prevent overfitting
-- **Temperature Scheduling**: Annealing for improved convergence
-- **TDD Development**: Comprehensive test coverage
+A PyTorch implementation of L0 regularization based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312), designed for survey calibration and sparse regression.
 
 ## Installation
 
 ```bash
-pip install l0
+pip install l0-python
 ```
 
 For development:
@@ -89,119 +83,117 @@ cd L0
 pip install -e .[dev]
 ```
 
-## Quick Start
+## Our Approach to Test-Time Gates
 
-### Neural Network Sparsification
+The original Hard Concrete formulation uses temperature (β) during training to control the sharpness of stochastic gates. At test time, there's a design choice: whether to include temperature in the deterministic gate computation.
 
+We include temperature at test time:
 ```python
-import torch
-from l0 import L0Linear, compute_l0l2_penalty, TemperatureScheduler, update_temperatures
-
-# Create a sparse model
-class SparseModel(torch.nn.Module):
-    def __init__(self):
-        super().__init__()
-        self.fc1 = L0Linear(784, 256, init_sparsity=0.5)
-        self.fc2 = L0Linear(256, 10, init_sparsity=0.7)
-    
-    def forward(self, x):
-        x = torch.relu(self.fc1(x))
-        return self.fc2(x)
-
-model = SparseModel()
-optimizer = torch.optim.Adam(model.parameters())
-scheduler = TemperatureScheduler(initial_temp=1.0, final_temp=0.1)
-
-# Training loop
-for epoch in range(100):
-    # Update temperature
-    temp = scheduler.get_temperature(epoch)
-    update_temperatures(model, temp)
-    
-    # Forward pass
-    output = model(input_data)
-    ce_loss = criterion(output, target)
-    
-    # Add L0L2 penalty
-    penalty = compute_l0l2_penalty(model, l0_lambda=1e-3, l2_lambda=1e-4)
-    loss = ce_loss + penalty
-    
-    # Backward pass
-    optimizer.zero_grad()
-    loss.backward()
-    optimizer.step()
+# Our approach: include temperature
+z = sigmoid(log_alpha / beta) * (zeta - gamma) + gamma
+
+# Alternative: omit temperature
+z = sigmoid(log_alpha) * (zeta - gamma) + gamma
 ```
 
-### Intelligent Sample Selection
+Including temperature produces sharper 0/1 decisions, which we find beneficial for achieving clean sparsity in our applications. See `examples/sparse_regression_demo.py` for a demonstration on a 4-variable regression problem.
+
+## Primary Use Case: Survey Calibration
+
+This package was developed for PolicyEngine's survey calibration, where we select a sparse subset of survey households while matching population targets.
 
 ```python
-from l0 import SampleGate, HybridGate
-
-# Pure L0 selection
-gate = SampleGate(n_samples=10000, target_samples=1000)
-selected_data, indices = gate.select_samples(data)
-
-# Hybrid selection (25% L0, 75% random)
-hybrid = HybridGate(
-    n_items=10000,
-    l0_fraction=0.25,
-    random_fraction=0.75,
-    target_items=1000
+import numpy as np
+from scipy import sparse as sp
+from l0.calibration import SparseCalibrationWeights
+
+# Setup: Q targets, N households
+Q, N = 200, 10000
+M = sp.random(Q, N, density=0.3, format="csr")  # Household characteristics
+y = np.random.uniform(1e6, 1e8, size=Q)          # Population targets
+
+# Initialize model
+model = SparseCalibrationWeights(
+    n_features=N,
+    beta=0.35,
+    gamma=-0.1,
+    zeta=1.1,
+    init_keep_prob=0.5,
+    init_weights=1.0,
+    log_weight_jitter_sd=0.05,
+    device="cuda",
+)
+
+# Train with L0+L2 regularization
+model.fit(
+    M=M,
+    y=y,
+    lambda_l0=1e-6,
+    lambda_l2=1e-8,
+    lr=0.15,
+    epochs=2000,
+    loss_type="relative",
+    verbose=True,
 )
-selected, indices, types = hybrid.select(data)
+
+# Get results
+active = model.get_active_weights()
+print(f"Selected {active['count']} of {N} households")
+print(f"Sparsity: {model.get_sparsity():.1%}")
 ```
 
-### Feature Selection
+### Key Features
+
+- **Non-negative weights**: Constrained via log-space parameterization
+- **L0 sparsity**: Directly minimizes the count of active weights
+- **Relative loss**: Scale-invariant for targets spanning orders of magnitude
+- **Group-wise averaging**: Balance loss across target groups with different sizes
+- **GPU support**: CUDA acceleration for large problems
+
+## Sparse Regression
+
+For sparse linear regression with scipy sparse matrices:
 
 ```python
-from l0 import FeatureGate
+from scipy import sparse as sp
+from l0.sparse import SparseL0Linear
 
-# Select top features
-gate = FeatureGate(n_features=1000, max_features=50)
-selected_data, feature_indices = gate.select_features(data)
+# Sparse design matrix
+X = sp.random(1000, 500, density=0.1, format="csr")
+y = np.random.randn(1000)
 
-# Get feature importance
-importance = gate.get_feature_importance()
+model = SparseL0Linear(n_features=500)
+model.fit(X, y, lambda_l0=0.001, epochs=1000)
+
+# Get sparse coefficients
+coef = model.get_coefficients(threshold=0.01)
 ```
 
-## Integration with PolicyEngine
+## Example: Variable Selection
 
-This package is designed to work with PolicyEngine's calibration system:
+The `examples/sparse_regression_demo.py` script demonstrates L0 regularization on a simple problem where the true coefficients are `[1, 0, -2, 0]`:
 
-```python
-# In policyengine-us-data or similar
-from l0 import HardConcrete
-
-# Use for household selection in CPS calibration
-gates = HardConcrete(
-    len(household_weights),
-    temperature=0.25,
-    init_mean=0.999  # Start with most households
-)
-
-# Apply gates during reweighting
-masked_weights = weights * gates()
+```bash
+python examples/sparse_regression_demo.py
 ```
 
-## Documentation
+Output:
+```
+True coefficients:        [ 1.  0. -2.  0.]
+Recovered coefficients:   [ 1.039  0.    -2.069 -0.   ]
+Gates:                    [1. 0. 1. 0.]
+```
 
-Full documentation available at: https://policyengine.github.io/L0/
+The model correctly identifies that only variables 1 and 3 contribute to the outcome.
 
 ## Testing
 
-Run tests with:
 ```bash
 pytest tests/ -v --cov=l0
 ```
 
-## Acknowledgments
-
-This implementation is inspired by and builds upon the [original L0 regularization code](https://github.com/AMLab-Amsterdam/L0_regularization) by AMLab Amsterdam, which accompanied the paper by Louizos et al. (2018).
-
 ## Citation
 
-If you use this package, please cite:
-
 ```bibtex
 @article{louizos2017learning,
   title={Learning Sparse Neural Networks through L0 Regularization},
@@ -213,4 +205,4 @@ If you use this package, please cite:
 
 ## License
 
-MIT License - see [LICENSE](LICENSE) file for details.
+MIT License - see [LICENSE](LICENSE) for details.

l0_python-0.4.3/README.md

@@ -0,0 +1,143 @@
+# L0 Regularization
+
+[![PyPI version](https://badge.fury.io/py/l0-python.svg)](https://pypi.org/project/l0-python/)
+[![CI](https://github.com/PolicyEngine/L0/actions/workflows/push.yml/badge.svg)](https://github.com/PolicyEngine/L0/actions)
+
+A PyTorch implementation of L0 regularization based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312), designed for survey calibration and sparse regression.
+
+## Installation
+
+```bash
+pip install l0-python
+```
+
+For development:
+```bash
+git clone https://github.com/PolicyEngine/L0.git
+cd L0
+pip install -e .[dev]
+```
+
+## Our Approach to Test-Time Gates
+
+The original Hard Concrete formulation uses temperature (β) during training to control the sharpness of stochastic gates. At test time, there's a design choice: whether to include temperature in the deterministic gate computation.
+
+We include temperature at test time:
+```python
+# Our approach: include temperature
+z = sigmoid(log_alpha / beta) * (zeta - gamma) + gamma
+
+# Alternative: omit temperature
+z = sigmoid(log_alpha) * (zeta - gamma) + gamma
+```
+
+Including temperature produces sharper 0/1 decisions, which we find beneficial for achieving clean sparsity in our applications. See `examples/sparse_regression_demo.py` for a demonstration on a 4-variable regression problem.
+
+## Primary Use Case: Survey Calibration
+
+This package was developed for PolicyEngine's survey calibration, where we select a sparse subset of survey households while matching population targets.
+
+```python
+import numpy as np
+from scipy import sparse as sp
+from l0.calibration import SparseCalibrationWeights
+
+# Setup: Q targets, N households
+Q, N = 200, 10000
+M = sp.random(Q, N, density=0.3, format="csr")  # Household characteristics
+y = np.random.uniform(1e6, 1e8, size=Q)          # Population targets
+
+# Initialize model
+model = SparseCalibrationWeights(
+    n_features=N,
+    beta=0.35,
+    gamma=-0.1,
+    zeta=1.1,
+    init_keep_prob=0.5,
+    init_weights=1.0,
+    log_weight_jitter_sd=0.05,
+    device="cuda",
+)
+
+# Train with L0+L2 regularization
+model.fit(
+    M=M,
+    y=y,
+    lambda_l0=1e-6,
+    lambda_l2=1e-8,
+    lr=0.15,
+    epochs=2000,
+    loss_type="relative",
+    verbose=True,
+)
+
+# Get results
+active = model.get_active_weights()
+print(f"Selected {active['count']} of {N} households")
+print(f"Sparsity: {model.get_sparsity():.1%}")
+```
+
+### Key Features
+
+- **Non-negative weights**: Constrained via log-space parameterization
+- **L0 sparsity**: Directly minimizes the count of active weights
+- **Relative loss**: Scale-invariant for targets spanning orders of magnitude
+- **Group-wise averaging**: Balance loss across target groups with different sizes
+- **GPU support**: CUDA acceleration for large problems
+
+## Sparse Regression
+
+For sparse linear regression with scipy sparse matrices:
+
+```python
+from scipy import sparse as sp
+from l0.sparse import SparseL0Linear
+
+# Sparse design matrix
+X = sp.random(1000, 500, density=0.1, format="csr")
+y = np.random.randn(1000)
+
+model = SparseL0Linear(n_features=500)
+model.fit(X, y, lambda_l0=0.001, epochs=1000)
+
+# Get sparse coefficients
+coef = model.get_coefficients(threshold=0.01)
+```
+
+## Example: Variable Selection
+
+The `examples/sparse_regression_demo.py` script demonstrates L0 regularization on a simple problem where the true coefficients are `[1, 0, -2, 0]`:
+
+```bash
+python examples/sparse_regression_demo.py
+```
+
+Output:
+```
+True coefficients:        [ 1.  0. -2.  0.]
+Recovered coefficients:   [ 1.039  0.    -2.069 -0.   ]
+Gates:                    [1. 0. 1. 0.]
+```
+
+The model correctly identifies that only variables 1 and 3 contribute to the outcome.
+
+## Testing
+
+```bash
+pytest tests/ -v --cov=l0
+```
+
+## Citation
+
+```bibtex
+@article{louizos2017learning,
+  title={Learning Sparse Neural Networks through L0 Regularization},
+  author={Louizos, Christos and Welling, Max and Kingma, Diederik P},
+  journal={arXiv preprint arXiv:1712.01312},
+  year={2017}
+}
+```
+
+## License
+
+MIT License - see [LICENSE](LICENSE) for details.

{l0_python-0.4.1 → l0_python-0.4.3}/l0/calibration.py

@@ -444,9 +444,9 @@ class SparseCalibrationWeights(nn.Module):
                                 rel_errors[group_mask].mean().item()
                             )
                             group_losses.append(group_mean_err)
-                        mean_group_loss = np.mean(group_losses)
+                        mean_group_mare = np.mean(group_losses)
                     else:
-                        mean_group_loss = mean_rel_err
+                        mean_group_mare = mean_rel_err
 
                     # Calculate sparsity percentage
                     sparsity_pct = 100 * (
@@ -461,7 +461,7 @@ class SparseCalibrationWeights(nn.Module):
                     if target_groups is not None:
                         print(
                             f"Epoch {epoch+1:4d}: "
-                            f"mean_group_loss={mean_group_loss:.1%}, "
+                            f"mean_group_mare={mean_group_mare:.4%}, "
                             f"max_error={max_rel_err:.1%}, "
                             f"total_loss={actual_total_loss:.3f}, "
                             f"active={active_info['count']:4d}/{self.n_features} ({sparsity_pct:.1f}% sparse)"
@@ -469,7 +469,7 @@ class SparseCalibrationWeights(nn.Module):
                     else:
                         print(
                             f"Epoch {epoch+1:4d}: "
-                            f"mean_error={mean_rel_err:.1%}, "
+                            f"mean_error={mean_rel_err:.4%}, "
                             f"max_error={max_rel_err:.1%}, "
                             f"total_loss={actual_total_loss:.3f}, "
                             f"active={active_info['count']:4d}/{self.n_features} ({sparsity_pct:.1f}% sparse)"

{l0_python-0.4.1 → l0_python-0.4.3/l0_python.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: l0-python
-Version: 0.4.1
+Version: 0.4.3
 Summary: L0 regularization for sparse neural networks and intelligent sampling
 Author-email: PolicyEngine <hello@policyengine.org>
 License: MIT License
@@ -65,21 +65,15 @@ Dynamic: license-file
 
 # L0 Regularization
 
-A PyTorch implementation of L0 regularization for neural network sparsification and intelligent sampling, based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312).
+[![PyPI version](https://badge.fury.io/py/l0-python.svg)](https://pypi.org/project/l0-python/)
+[![CI](https://github.com/PolicyEngine/L0/actions/workflows/push.yml/badge.svg)](https://github.com/PolicyEngine/L0/actions)
 
-## Features
-
-- **Hard Concrete Distribution**: Differentiable approximation of L0 norm
-- **Sparse Neural Network Layers**: L0Linear, L0Conv2d with automatic pruning
-- **Intelligent Sampling**: Sample/feature selection gates for calibration
-- **L0L2 Combined Penalty**: Recommended approach to prevent overfitting
-- **Temperature Scheduling**: Annealing for improved convergence
-- **TDD Development**: Comprehensive test coverage
+A PyTorch implementation of L0 regularization based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312), designed for survey calibration and sparse regression.
 
 ## Installation
 
 ```bash
-pip install l0
+pip install l0-python
 ```
 
 For development:
@@ -89,119 +83,117 @@ cd L0
 pip install -e .[dev]
 ```
 
-## Quick Start
+## Our Approach to Test-Time Gates
 
-### Neural Network Sparsification
+The original Hard Concrete formulation uses temperature (β) during training to control the sharpness of stochastic gates. At test time, there's a design choice: whether to include temperature in the deterministic gate computation.
 
+We include temperature at test time:
 ```python
-import torch
-from l0 import L0Linear, compute_l0l2_penalty, TemperatureScheduler, update_temperatures
-
-# Create a sparse model
-class SparseModel(torch.nn.Module):
-    def __init__(self):
-        super().__init__()
-        self.fc1 = L0Linear(784, 256, init_sparsity=0.5)
-        self.fc2 = L0Linear(256, 10, init_sparsity=0.7)
-    
-    def forward(self, x):
-        x = torch.relu(self.fc1(x))
-        return self.fc2(x)
-
-model = SparseModel()
-optimizer = torch.optim.Adam(model.parameters())
-scheduler = TemperatureScheduler(initial_temp=1.0, final_temp=0.1)
-
-# Training loop
-for epoch in range(100):
-    # Update temperature
-    temp = scheduler.get_temperature(epoch)
-    update_temperatures(model, temp)
-    
-    # Forward pass
-    output = model(input_data)
-    ce_loss = criterion(output, target)
-    
-    # Add L0L2 penalty
-    penalty = compute_l0l2_penalty(model, l0_lambda=1e-3, l2_lambda=1e-4)
-    loss = ce_loss + penalty
-    
-    # Backward pass
-    optimizer.zero_grad()
-    loss.backward()
-    optimizer.step()
+# Our approach: include temperature
+z = sigmoid(log_alpha / beta) * (zeta - gamma) + gamma
+
+# Alternative: omit temperature
+z = sigmoid(log_alpha) * (zeta - gamma) + gamma
 ```
 
-### Intelligent Sample Selection
+Including temperature produces sharper 0/1 decisions, which we find beneficial for achieving clean sparsity in our applications. See `examples/sparse_regression_demo.py` for a demonstration on a 4-variable regression problem.
+
+## Primary Use Case: Survey Calibration
+
+This package was developed for PolicyEngine's survey calibration, where we select a sparse subset of survey households while matching population targets.
 
 ```python
-from l0 import SampleGate, HybridGate
-
-# Pure L0 selection
-gate = SampleGate(n_samples=10000, target_samples=1000)
-selected_data, indices = gate.select_samples(data)
-
-# Hybrid selection (25% L0, 75% random)
-hybrid = HybridGate(
-    n_items=10000,
-    l0_fraction=0.25,
-    random_fraction=0.75,
-    target_items=1000
+import numpy as np
+from scipy import sparse as sp
+from l0.calibration import SparseCalibrationWeights
+
+# Setup: Q targets, N households
+Q, N = 200, 10000
+M = sp.random(Q, N, density=0.3, format="csr")  # Household characteristics
+y = np.random.uniform(1e6, 1e8, size=Q)          # Population targets
+
+# Initialize model
+model = SparseCalibrationWeights(
+    n_features=N,
+    beta=0.35,
+    gamma=-0.1,
+    zeta=1.1,
+    init_keep_prob=0.5,
+    init_weights=1.0,
+    log_weight_jitter_sd=0.05,
+    device="cuda",
+)
+
+# Train with L0+L2 regularization
+model.fit(
+    M=M,
+    y=y,
+    lambda_l0=1e-6,
+    lambda_l2=1e-8,
+    lr=0.15,
+    epochs=2000,
+    loss_type="relative",
+    verbose=True,
 )
-selected, indices, types = hybrid.select(data)
+
+# Get results
+active = model.get_active_weights()
+print(f"Selected {active['count']} of {N} households")
+print(f"Sparsity: {model.get_sparsity():.1%}")
 ```
 
-### Feature Selection
+### Key Features
+
+- **Non-negative weights**: Constrained via log-space parameterization
+- **L0 sparsity**: Directly minimizes the count of active weights
+- **Relative loss**: Scale-invariant for targets spanning orders of magnitude
+- **Group-wise averaging**: Balance loss across target groups with different sizes
+- **GPU support**: CUDA acceleration for large problems
+
+## Sparse Regression
+
+For sparse linear regression with scipy sparse matrices:
 
 ```python
-from l0 import FeatureGate
+from scipy import sparse as sp
+from l0.sparse import SparseL0Linear
 
-# Select top features
-gate = FeatureGate(n_features=1000, max_features=50)
-selected_data, feature_indices = gate.select_features(data)
+# Sparse design matrix
+X = sp.random(1000, 500, density=0.1, format="csr")
+y = np.random.randn(1000)
 
-# Get feature importance
-importance = gate.get_feature_importance()
+model = SparseL0Linear(n_features=500)
+model.fit(X, y, lambda_l0=0.001, epochs=1000)
+
+# Get sparse coefficients
+coef = model.get_coefficients(threshold=0.01)
 ```
 
-## Integration with PolicyEngine
+## Example: Variable Selection
 
-This package is designed to work with PolicyEngine's calibration system:
+The `examples/sparse_regression_demo.py` script demonstrates L0 regularization on a simple problem where the true coefficients are `[1, 0, -2, 0]`:
 
-```python
-# In policyengine-us-data or similar
-from l0 import HardConcrete
-
-# Use for household selection in CPS calibration
-gates = HardConcrete(
-    len(household_weights),
-    temperature=0.25,
-    init_mean=0.999  # Start with most households
-)
-
-# Apply gates during reweighting
-masked_weights = weights * gates()
+```bash
+python examples/sparse_regression_demo.py
 ```
 
-## Documentation
+Output:
+```
+True coefficients:        [ 1.  0. -2.  0.]
+Recovered coefficients:   [ 1.039  0.    -2.069 -0.   ]
+Gates:                    [1. 0. 1. 0.]
+```
 
-Full documentation available at: https://policyengine.github.io/L0/
+The model correctly identifies that only variables 1 and 3 contribute to the outcome.
 
 ## Testing
 
-Run tests with:
 ```bash
 pytest tests/ -v --cov=l0
 ```
 
-## Acknowledgments
-
-This implementation is inspired by and builds upon the [original L0 regularization code](https://github.com/AMLab-Amsterdam/L0_regularization) by AMLab Amsterdam, which accompanied the paper by Louizos et al. (2018).
-
 ## Citation
 
-If you use this package, please cite:
-
 ```bibtex
 @article{louizos2017learning,
   title={Learning Sparse Neural Networks through L0 Regularization},
@@ -213,4 +205,4 @@ If you use this package, please cite:
 
 ## License
 
-MIT License - see [LICENSE](LICENSE) file for details.
+MIT License - see [LICENSE](LICENSE) for details.

{l0_python-0.4.1 → l0_python-0.4.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "l0-python"
-version = "0.4.1"
+version = "0.4.3"
 description = "L0 regularization for sparse neural networks and intelligent sampling"
 readme = "README.md"
 authors = [

{l0_python-0.4.1 → l0_python-0.4.3}/tests/test_calibration.py

@@ -247,6 +247,232 @@ class TestSparseCalibrationWeights:
                 weights_with_l2.max() <= weights_no_l2.max() * 2.0
             ), "L2 should prevent extreme weights"
 
+    def test_pure_l2_penalty(self):
+        """Test pure L2 regularization without L0."""
+        N = 50
+        Q = 30
+
+        np.random.seed(42)
+        torch.manual_seed(42)
+
+        # Create underdetermined problem where L2 helps regularize
+        M = sp.random(Q, N, density=0.5, format="csr")
+        M.data = np.abs(M.data) * 2  # Scale up to create larger weights
+        y = np.ones(Q) * 10  # Simple target
+
+        # Train without any regularization
+        model_no_reg = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,  # Start with larger weights
+            init_keep_prob=0.999,  # Keep all weights active
+        )
+        model_no_reg.fit(
+            M,
+            y,
+            lambda_l0=0.0,  # No L0
+            lambda_l2=0.0,  # No L2
+            epochs=1000,
+            lr=0.02,
+            verbose=False,
+        )
+
+        # Train with strong L2 regularization
+        model_l2_only = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,  # Same starting point
+            init_keep_prob=0.999,  # Keep all weights active
+        )
+        model_l2_only.fit(
+            M,
+            y,
+            lambda_l0=0.0,  # No L0
+            lambda_l2=10.0,  # Very strong L2 to see clear effect
+            epochs=1000,
+            lr=0.02,
+            verbose=False,
+        )
+
+        with torch.no_grad():
+            weights_no_reg = model_no_reg.get_weights(deterministic=True)
+            weights_l2 = model_l2_only.get_weights(deterministic=True)
+
+            # Get non-zero weights
+            active_no_reg = weights_no_reg[weights_no_reg > 1e-8]
+            active_l2 = weights_l2[weights_l2 > 1e-8]
+
+            # L2 should keep most weights active (no sparsity from L2)
+            assert (
+                len(active_l2) >= N * 0.9
+            ), f"Pure L2 should not induce sparsity, got {len(active_l2)}/{N}"
+
+            # L2 norm should be smaller with regularization
+            l2_norm_no_reg = (weights_no_reg**2).sum().sqrt()
+            l2_norm_with_l2 = (weights_l2**2).sum().sqrt()
+            assert (
+                l2_norm_with_l2 < l2_norm_no_reg
+            ), f"L2 regularization should reduce L2 norm: {l2_norm_with_l2:.2f} vs {l2_norm_no_reg:.2f}"
+
+            # Check coefficient of variation (CV = std/mean) is lower with L2
+            # This captures that L2 shrinks weights toward each other
+            if active_no_reg.mean() > 1e-6 and active_l2.mean() > 1e-6:
+                cv_no_reg = active_no_reg.std() / active_no_reg.mean()
+                cv_l2 = active_l2.std() / active_l2.mean()
+                assert (
+                    cv_l2 < cv_no_reg * 1.2
+                ), f"L2 should reduce relative variation: CV {cv_l2:.2f} vs {cv_no_reg:.2f}"
+
+            # L2 should prevent extreme weights
+            assert (
+                weights_l2.max() < weights_no_reg.max() * 1.5
+            ), f"L2 should limit max weights: {weights_l2.max():.2f} vs {weights_no_reg.max():.2f}"
+
+            # Both should still fit reasonably well
+            y_pred_no_reg = model_no_reg.predict(M).cpu().numpy()
+            y_pred_l2 = model_l2_only.predict(M).cpu().numpy()
+
+            error_no_reg = np.abs((y - y_pred_no_reg) / y).mean()
+            error_l2 = np.abs((y - y_pred_l2) / y).mean()
+
+            # L2 model may have slightly worse fit due to regularization
+            assert (
+                error_l2 < 0.5
+            ), f"L2 model should still fit reasonably: {error_l2:.3f}"
+            # But the trade-off is worth it for regularization
+
+    def test_l0_l2_combination(self):
+        """Test that combining L0 and L2 gives both sparsity and regularization."""
+        N = 100  # features
+        Q = 50  # targets
+
+        np.random.seed(42)
+        torch.manual_seed(42)
+
+        # Create problem with potential for overfitting
+        M = sp.random(Q, N, density=0.4, format="csr")
+        M.data = np.abs(M.data) * 3
+        y = np.random.uniform(5, 20, size=Q)
+
+        # Model 1: Only L0 (sparsity without weight regularization)
+        model_l0_only = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,
+            init_keep_prob=0.5,  # Start with 50% probability
+        )
+        model_l0_only.fit(
+            M,
+            y,
+            lambda_l0=0.01,  # Stronger L0 for sparsity
+            lambda_l2=0.0,  # No L2
+            epochs=2000,
+            lr=0.02,
+            verbose=False,
+        )
+
+        # Model 2: Only L2 (weight regularization without sparsity)
+        model_l2_only = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,
+            init_keep_prob=0.999,  # Keep all weights active
+        )
+        model_l2_only.fit(
+            M,
+            y,
+            lambda_l0=0.0,  # No L0
+            lambda_l2=0.1,  # Moderate L2
+            epochs=1500,
+            lr=0.02,
+            verbose=False,
+        )
+
+        # Model 3: Combined L0+L2 (both sparsity and weight regularization)
+        model_l0_l2 = SparseCalibrationWeights(
+            n_features=N,
+            init_weights=2.0,
+            init_keep_prob=0.5,  # Same starting point as L0-only
+        )
+        model_l0_l2.fit(
+            M,
+            y,
+            lambda_l0=0.01,  # Same L0 as model 1
+            lambda_l2=0.1,  # Add L2 regularization
+            epochs=2000,
+            lr=0.02,
+            verbose=False,
+        )
+
+        with torch.no_grad():
+            # Get weights and stats for all models
+            weights_l0_only = model_l0_only.get_weights(deterministic=True)
+            weights_l2_only = model_l2_only.get_weights(deterministic=True)
+            weights_l0_l2 = model_l0_l2.get_weights(deterministic=True)
+
+            # Count active weights
+            active_l0_only = (weights_l0_only > 1e-6).sum().item()
+            active_l2_only = (weights_l2_only > 1e-6).sum().item()
+            active_l0_l2 = (weights_l0_l2 > 1e-6).sum().item()
+
+            # L0-only should have sparsity
+            assert (
+                active_l0_only < N * 0.8
+            ), f"L0 should induce sparsity: {active_l0_only}/{N} active"
+
+            # L2-only should have no/little sparsity
+            assert (
+                active_l2_only > N * 0.9
+            ), f"L2 alone shouldn't induce sparsity: {active_l2_only}/{N} active"
+
+            # L0+L2 should have sparsity (from L0)
+            assert (
+                active_l0_l2 < N * 0.8
+            ), f"L0+L2 should have sparsity: {active_l0_l2}/{N} active"
+
+            # Among active weights, L0+L2 should have smaller magnitudes than L0-only (from L2)
+            active_mask_l0_only = weights_l0_only > 1e-6
+            active_mask_l0_l2 = weights_l0_l2 > 1e-6
+
+            if active_mask_l0_only.any() and active_mask_l0_l2.any():
+                # Compare L2 norms of active weights
+                l2_norm_l0_only = (
+                    (weights_l0_only[active_mask_l0_only] ** 2).sum().sqrt()
+                )
+                l2_norm_l0_l2 = (
+                    (weights_l0_l2[active_mask_l0_l2] ** 2).sum().sqrt()
+                )
+
+                # L0+L2 should have smaller weight norms than L0-only
+                # (L2 regularization effect on top of sparsity)
+                assert (
+                    l2_norm_l0_l2 < l2_norm_l0_only * 1.2
+                ), f"L0+L2 should have controlled weights: {l2_norm_l0_l2:.2f} vs L0-only {l2_norm_l0_only:.2f}"
+
+            # Check prediction quality for all models
+            y_pred_l0_only = model_l0_only.predict(M).cpu().numpy()
+            y_pred_l2_only = model_l2_only.predict(M).cpu().numpy()
+            y_pred_l0_l2 = model_l0_l2.predict(M).cpu().numpy()
+
+            error_l0_only = np.abs((y - y_pred_l0_only) / (y + 1)).mean()
+            error_l2_only = np.abs((y - y_pred_l2_only) / (y + 1)).mean()
+            error_l0_l2 = np.abs((y - y_pred_l0_l2) / (y + 1)).mean()
+
+            # All should fit reasonably well (L2-only may have slightly worse fit due to regularization)
+            assert (
+                error_l0_only < 0.35
+            ), f"L0-only should fit well: {error_l0_only:.3f}"
+            assert (
+                error_l2_only < 0.35
+            ), f"L2-only should fit well: {error_l2_only:.3f}"
+            assert (
+                error_l0_l2 < 0.35
+            ), f"L0+L2 should fit well: {error_l0_l2:.3f}"
+
+            print(
+                f"\nL0-only: {active_l0_only}/{N} active, error={error_l0_only:.3f}"
+            )
+            print(
+                f"L2-only: {active_l2_only}/{N} active, error={error_l2_only:.3f}"
+            )
+            print(f"L0+L2: {active_l0_l2}/{N} active, error={error_l0_l2:.3f}")
+
     def test_group_wise_averaging(self):
         """Test that group-wise averaging balances loss contributions."""
         N = 100  # features (households)

l0_python-0.4.1/README.md

@@ -1,151 +0,0 @@
-# L0 Regularization
-
-A PyTorch implementation of L0 regularization for neural network sparsification and intelligent sampling, based on [Louizos, Welling, & Kingma (2017)](https://arxiv.org/abs/1712.01312).
-
-## Features
-
-- **Hard Concrete Distribution**: Differentiable approximation of L0 norm
-- **Sparse Neural Network Layers**: L0Linear, L0Conv2d with automatic pruning
-- **Intelligent Sampling**: Sample/feature selection gates for calibration
-- **L0L2 Combined Penalty**: Recommended approach to prevent overfitting
-- **Temperature Scheduling**: Annealing for improved convergence
-- **TDD Development**: Comprehensive test coverage
-
-## Installation
-
-```bash
-pip install l0
-```
-
-For development:
-```bash
-git clone https://github.com/PolicyEngine/L0.git
-cd L0
-pip install -e .[dev]
-```
-
-## Quick Start
-
-### Neural Network Sparsification
-
-```python
-import torch
-from l0 import L0Linear, compute_l0l2_penalty, TemperatureScheduler, update_temperatures
-
-# Create a sparse model
-class SparseModel(torch.nn.Module):
-    def __init__(self):
-        super().__init__()
-        self.fc1 = L0Linear(784, 256, init_sparsity=0.5)
-        self.fc2 = L0Linear(256, 10, init_sparsity=0.7)
-    
-    def forward(self, x):
-        x = torch.relu(self.fc1(x))
-        return self.fc2(x)
-
-model = SparseModel()
-optimizer = torch.optim.Adam(model.parameters())
-scheduler = TemperatureScheduler(initial_temp=1.0, final_temp=0.1)
-
-# Training loop
-for epoch in range(100):
-    # Update temperature
-    temp = scheduler.get_temperature(epoch)
-    update_temperatures(model, temp)
-    
-    # Forward pass
-    output = model(input_data)
-    ce_loss = criterion(output, target)
-    
-    # Add L0L2 penalty
-    penalty = compute_l0l2_penalty(model, l0_lambda=1e-3, l2_lambda=1e-4)
-    loss = ce_loss + penalty
-    
-    # Backward pass
-    optimizer.zero_grad()
-    loss.backward()
-    optimizer.step()
-```
-
-### Intelligent Sample Selection
-
-```python
-from l0 import SampleGate, HybridGate
-
-# Pure L0 selection
-gate = SampleGate(n_samples=10000, target_samples=1000)
-selected_data, indices = gate.select_samples(data)
-
-# Hybrid selection (25% L0, 75% random)
-hybrid = HybridGate(
-    n_items=10000,
-    l0_fraction=0.25,
-    random_fraction=0.75,
-    target_items=1000
-)
-selected, indices, types = hybrid.select(data)
-```
-
-### Feature Selection
-
-```python
-from l0 import FeatureGate
-
-# Select top features
-gate = FeatureGate(n_features=1000, max_features=50)
-selected_data, feature_indices = gate.select_features(data)
-
-# Get feature importance
-importance = gate.get_feature_importance()
-```
-
-## Integration with PolicyEngine
-
-This package is designed to work with PolicyEngine's calibration system:
-
-```python
-# In policyengine-us-data or similar
-from l0 import HardConcrete
-
-# Use for household selection in CPS calibration
-gates = HardConcrete(
-    len(household_weights),
-    temperature=0.25,
-    init_mean=0.999  # Start with most households
-)
-
-# Apply gates during reweighting
-masked_weights = weights * gates()
-```
-
-## Documentation
-
-Full documentation available at: https://policyengine.github.io/L0/
-
-## Testing
-
-Run tests with:
-```bash
-pytest tests/ -v --cov=l0
-```
-
-## Acknowledgments
-
-This implementation is inspired by and builds upon the [original L0 regularization code](https://github.com/AMLab-Amsterdam/L0_regularization) by AMLab Amsterdam, which accompanied the paper by Louizos et al. (2018).
-
-## Citation
-
-If you use this package, please cite:
-
-```bibtex
-@article{louizos2017learning,
-  title={Learning Sparse Neural Networks through L0 Regularization},
-  author={Louizos, Christos and Welling, Max and Kingma, Diederik P},
-  journal={arXiv preprint arXiv:1712.01312},
-  year={2017}
-}
-```
-
-## License
-
-MIT License - see [LICENSE](LICENSE) file for details.