@crownpeak/dqm-react-component-dev-mcp 1.2.0

package/data/probe/examples/reranker/README.md

@@ -0,0 +1,347 @@
+# BERT Reranker Example
+
+A complete working Rust implementation of a BERT-based document reranker using the Candle framework. This example demonstrates how to use transformer models for document reranking tasks, specifically using the ms-marco-MiniLM-L-2-v2 model.
+
+## Overview
+
+This implementation provides a cross-encoder based reranker that:
+- Loads pre-trained BERT models from HuggingFace Hub
+- Processes query-document pairs through the transformer
+- Computes relevance scores for ranking
+- Sorts documents by relevance to the query
+
+## Features
+
+- **Pure Rust Implementation**: Uses the Candle framework for ML inference
+- **HuggingFace Integration**: Automatic model and tokenizer downloading
+- **Cross-Encoder Architecture**: Proper query-document pair encoding
+- **Flexible Model Support**: Works with various BERT-based reranking models
+- **Interactive Mode**: Test reranking with custom queries and documents
+- **Command Line Interface**: Easy to use from command line or scripts
+
+## Installation and Setup
+
+### Prerequisites
+
+- Rust 1.70 or later
+- Internet connection (for model downloading)
+
+### Building the Project
+
+```bash
+cd examples/reranker
+cargo build --release
+```
+
+## Usage
+
+### Basic Usage with Default Documents
+
+```bash
+# Use the default ms-marco-MiniLM-L-2-v2 model
+cargo run --release -- --query "machine learning"
+
+# Or run the binary directly after building
+./target/release/reranker --query "rust programming"
+```
+
+### Using Custom Documents
+
+```bash
+cargo run --release -- \
+    --query "natural language processing" \
+    --documents "BERT is a transformer model,Python is a programming language,NLP involves text processing,Rust is systems programming"
+```
+
+### Interactive Mode
+
+```bash
+cargo run --release -- --query "your query here" --interactive
+```
+
+This will prompt you to enter documents one by one, then rerank them.
+
+### Using Different Models
+
+```bash
+# Use a different cross-encoder model
+cargo run --release -- \
+    --model "cross-encoder/ms-marco-MiniLM-L-6-v2" \
+    --query "information retrieval"
+
+# Use PyTorch weights instead of SafeTensors
+cargo run --release -- \
+    --model "cross-encoder/ms-marco-MiniLM-L-2-v2" \
+    --use-pth \
+    --query "document ranking"
+```
+
+## Supported Models
+
+This implementation works with cross-encoder models from HuggingFace Hub. Recommended models:
+
+- `cross-encoder/ms-marco-MiniLM-L-2-v2` (default, fast and efficient)
+- `cross-encoder/ms-marco-MiniLM-L-6-v2` (larger, potentially more accurate)
+- `cross-encoder/ms-marco-MiniLM-L-12-v2` (largest, highest accuracy)
+
+## Command Line Options
+
+- `--model, -m`: HuggingFace model ID (default: `cross-encoder/ms-marco-MiniLM-L-2-v2`)
+- `--revision, -r`: Model revision/branch (default: `main`)
+- `--use-pth`: Use PyTorch weights instead of SafeTensors
+- `--query, -q`: Search query (required)
+- `--documents, -d`: Comma-separated list of documents to rerank
+- `--interactive, -i`: Run in interactive mode
+
+## Example Output
+
+```
+Initializing BERT Reranker...
+Model: cross-encoder/ms-marco-MiniLM-L-2-v2
+Revision: main
+Using PyTorch weights: false
+
+=== Example Usage ===
+Query: machine learning
+Documents to rerank:
+  1. Rust is a systems programming language focused on safety and performance.
+  2. Python is a high-level programming language known for its simplicity.
+  3. Machine learning involves training algorithms on data to make predictions.
+  4. BERT is a transformer-based model for natural language understanding.
+  5. The Candle framework provides machine learning capabilities in Rust.
+  6. Cross-encoders are used for reranking tasks in information retrieval.
+  7. Tokenization is the process of breaking text into individual tokens.
+  8. Neural networks consist of interconnected nodes that process information.
+
+Loading BERT reranker model: cross-encoder/ms-marco-MiniLM-L-2-v2
+Config file: "/Users/username/.cache/huggingface/hub/models--cross-encoder--ms-marco-MiniLM-L-2-v2/snapshots/main/config.json"
+Tokenizer file: "/Users/username/.cache/huggingface/hub/models--cross-encoder--ms-marco-MiniLM-L-2-v2/snapshots/main/tokenizer.json"
+Weights file: "/Users/username/.cache/huggingface/hub/models--cross-encoder--ms-marco-MiniLM-L-2-v2/snapshots/main/model.safetensors"
+BERT model loaded successfully
+
+Reranking 8 documents for query: 'machine learning'
+Reranking completed
+
+=== Reranking Results ===
+Documents ranked by relevance to query:
+1. #3: 2.8934 - Machine learning involves training algorithms on data to make predictions.
+2. #5: 2.1203 - The Candle framework provides machine learning capabilities in Rust.
+3. #4: 1.9876 - BERT is a transformer-based model for natural language understanding.
+4. #8: 1.7432 - Neural networks consist of interconnected nodes that process information.
+5. #6: 1.5621 - Cross-encoders are used for reranking tasks in information retrieval.
+6. #2: 0.9834 - Python is a high-level programming language known for its simplicity.
+7. #7: 0.8976 - Tokenization is the process of breaking text into individual tokens.
+8. #1: 0.7654 - Rust is a systems programming language focused on safety and performance.
+```
+
+## Architecture Details
+
+### Cross-Encoder Approach
+
+This implementation uses a cross-encoder architecture where:
+1. Query and document are concatenated with a `[SEP]` token
+2. The combined text is tokenized and fed through BERT
+3. The `[CLS]` token embedding is used to compute a relevance score
+4. Documents are ranked by their scores
+
+### Model Components
+
+- **Tokenizer**: Converts text to tokens using HuggingFace tokenizers
+- **BERT Model**: Transformer encoder for processing text
+- **Scoring**: Uses the CLS token embedding sum as relevance score
+
+### Performance Considerations
+
+- **CPU Inference**: Runs on CPU by default (GPU support can be added)
+- **Memory Usage**: Models are loaded once and reused for multiple queries
+- **Caching**: HuggingFace Hub automatically caches downloaded models
+
+## Extending the Example
+
+### Adding GPU Support
+
+To enable GPU acceleration, modify the device initialization:
+
+```rust
+let device = Device::new_cuda(0)?; // Use GPU 0
+// or
+let device = Device::new_metal(0)?; // Use Metal on macOS
+```
+
+### Custom Scoring Functions
+
+The current implementation uses a simple sum of CLS embeddings. For production use, consider:
+- Adding a linear classification head
+- Using cosine similarity between query and document embeddings
+- Implementing attention-based scoring mechanisms
+
+### Batch Processing
+
+For better performance with multiple documents, implement batch processing:
+
+```rust
+// Process multiple query-document pairs simultaneously
+fn batch_rerank(&self, query: &str, documents: &[&str]) -> Result<Vec<f32>> {
+    // Implementation for batch processing
+}
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Model Download Failures**
+   - Check internet connection
+   - Verify model ID exists on HuggingFace Hub
+   - Try using `--use-pth` flag if SafeTensors download fails
+   - **For testing**: Use the demo version (`./target/release/demo`) which doesn't require model downloads
+
+2. **Memory Issues**
+   - Use smaller models (L-2 instead of L-12)
+   - Process documents in smaller batches
+   - Reduce sequence length in tokenizer
+
+3. **Performance Issues**
+   - Enable GPU support if available
+   - Use release builds (`cargo build --release`)
+   - Consider model quantization for faster inference
+
+4. **HuggingFace Hub API Issues**
+   - Some models may have download restrictions or require authentication
+   - The demo version provides the same interface without requiring model downloads
+   - Check HuggingFace Hub status if experiencing consistent download failures
+
+### Debug Mode
+
+Enable debug logging by setting the `RUST_LOG` environment variable:
+
+```bash
+RUST_LOG=debug cargo run --release -- --query "your query"
+```
+
+## Integration with Code Search
+
+This reranker can be integrated with the main probe code search tool to improve result relevance:
+
+```rust
+// Example integration
+let search_results = probe::search("function authentication")?;
+let documents: Vec<&str> = search_results.iter().map(|r| r.content.as_str()).collect();
+let reranked = reranker.rerank("user authentication", &documents)?;
+```
+
+## Testing
+
+### Demo Version (No Model Download Required)
+
+For quick testing without downloading models, use the demo version:
+
+```bash
+# Build both the real and demo versions
+cargo build --release
+
+# Test the demo version with mock reranking
+./target/release/demo --query "machine learning"
+
+# Test interactive mode
+./target/release/demo --query "neural networks" --interactive
+```
+
+The demo version uses simple word overlap instead of BERT models and demonstrates the complete interface.
+
+### Testing with Real Models
+
+Run the test suite:
+
+```bash
+cargo test
+```
+
+For integration tests with actual models:
+
+```bash
+cargo test --release -- --ignored
+```
+
+## Contributing
+
+This is an example implementation demonstrating BERT reranking with Candle. For production use, consider:
+- Adding comprehensive error handling
+- Implementing proper cross-encoder head architecture
+- Adding support for different similarity metrics
+- Optimizing for batch processing and GPU acceleration
+
+## Python Cross-Encoder Testing
+
+For debugging and comparing Python vs Rust implementations, several Python testing tools are provided:
+
+### Comprehensive Testing Script
+
+```bash
+# Run comprehensive cross-encoder testing
+./test_cross_encoder.sh
+
+# Or run Python script directly (requires dependencies)
+python3 test_cross_encoder.py
+```
+
+This script:
+- Tests both `transformers` and `sentence-transformers` libraries
+- Shows detailed tokenization analysis (token IDs, attention masks, special tokens)
+- Compares scores between relevant and irrelevant queries
+- Saves results to JSON for further analysis
+- Provides debugging recommendations for Rust implementation
+
+### Quick Debugging Script
+
+```bash
+# Run focused debugging tests
+python3 debug_scoring.py
+```
+
+This minimal script:
+- Tests hardcoded query-document pairs
+- Shows raw logits and final scores
+- Easy to modify for specific test cases
+- Highlights score differences and discrimination quality
+
+### Dependencies
+
+Install Python dependencies:
+
+```bash
+pip3 install -r requirements.txt
+```
+
+Required packages:
+- `torch` - PyTorch for model inference
+- `transformers` - HuggingFace transformers library
+- `sentence-transformers` - Cross-encoder wrapper (optional but recommended)
+- `numpy` - Numerical operations
+
+### Test Cases
+
+Both scripts test these scenarios by default:
+- **Relevant Query**: "how does authentication work" 
+- **Irrelevant Query**: "foobar random nonsense gibberish"
+- **Document**: Authentication-related text snippet
+
+Expected behavior:
+- Relevant query should score >0.5 (high relevance)
+- Irrelevant query should score <0.5 (low relevance)
+- Score difference should be significant (>0.1)
+
+### Debugging Rust Implementation
+
+Use these Python scripts to debug Rust cross-encoder issues:
+
+1. **Compare tokenization**: Check if token IDs match between Python and Rust
+2. **Compare raw logits**: Verify model outputs before activation functions
+3. **Compare final scores**: Check if score calculation methods are identical
+4. **Model configuration**: Ensure same model version and weights are loaded
+
+The Python scripts provide detailed output to help identify where discrepancies occur.
+
+## License
+
+This example follows the same license as the main probe project.

package/data/probe/examples/reranker/RUST_BERT_COMPARISON.md

@@ -0,0 +1,82 @@
+# Rust-BERT vs Candle for Cross-Encoders
+
+## Summary
+
+After investigating rust-bert for cross-encoder support, here are the key findings:
+
+### rust-bert Limitations for Cross-Encoders
+
+1. **No Native Cross-Encoder Support**: rust-bert doesn't have a dedicated cross-encoder pipeline
+2. **Classification Focus**: The sequence classification pipeline expects discrete labels (POSITIVE/NEGATIVE), not continuous relevance scores
+3. **Model Format**: Requires TorchScript (.ot) format, not standard PyTorch .bin files
+4. **Architecture Mismatch**: Cross-encoders output a single score, but rust-bert's classification expects label probabilities
+
+### Our Candle Implementation Advantages
+
+1. **Direct PyTorch Support**: Loads .bin files directly from HuggingFace
+2. **Custom Architecture**: We implement the exact cross-encoder architecture
+3. **Raw Scores**: Returns raw logits for scoring, which is what cross-encoders need
+4. **Flexibility**: Full control over tokenization and model behavior
+
+## Model Availability
+
+The MS-MARCO models on HuggingFace include:
+- PyTorch formats: `pytorch_model.bin`, `model.safetensors`
+- ONNX formats: Multiple optimized ONNX versions
+- No TorchScript (.ot) versions available
+
+## Conversion Options
+
+### 1. PyTorch to TorchScript
+```python
+# See convert_to_torchscript.py
+traced_model = torch.jit.trace(model, example_inputs)
+traced_model.save("rust_model.ot")
+```
+
+### 2. Use ONNX Runtime
+Instead of rust-bert, consider using ONNX Runtime with Rust bindings:
+```toml
+[dependencies]
+ort = "1.16"  # ONNX Runtime for Rust
+```
+
+### 3. Continue with Candle
+Our current Candle implementation is actually well-suited for cross-encoders.
+
+## Recommendation
+
+**Stay with Candle** for cross-encoder implementation because:
+
+1. It already works correctly with HuggingFace models
+2. No conversion needed
+3. Better control over the scoring pipeline
+4. The issue isn't with Candle - it's that TinyBERT (4M params) is too small
+
+**To improve results:**
+1. Switch to a larger model (MiniLM-L-6-v2 with 85M params)
+2. Make the model configurable via CLI
+3. Consider adding ONNX support as an alternative backend
+
+## Code Comparison
+
+### rust-bert Approach (Would Require Modifications)
+```rust
+// rust-bert expects classification, not scoring
+let config = SequenceClassificationConfig { ... };
+let model = SequenceClassificationModel::new(config)?;
+let output = model.predict(&[text]); // Returns Label with probability
+```
+
+### Our Candle Approach (Current)
+```rust
+// Direct cross-encoder implementation
+let bert_outputs = self.bert.forward(&input_ids, &attention_mask, token_type_ids.as_ref())?;
+let cls_output = bert_outputs.i((.., 0, ..))?;
+let logits = self.classifier.forward(&cls_output)?;
+let score = logits.i((0, 0))?.to_scalar::<f32>()?; // Raw relevance score
+```
+
+## Conclusion
+
+rust-bert isn't suitable for cross-encoder models without significant modifications. Our Candle implementation is the right approach. The scoring issues are due to model size (TinyBERT), not the implementation framework.

package/data/probe/examples/reranker/TOKENIZATION_GUIDE.md

@@ -0,0 +1,120 @@
+# Cross-Encoder Tokenization Guide for Rust Implementation
+
+## Critical Points for Correct Implementation
+
+### 1. **Use `encode_pair()` NOT Manual Concatenation**
+
+❌ **WRONG** (Manual concatenation):
+```rust
+let text = format!("{} [SEP] {}", query, document);
+let encoding = tokenizer.encode(text, true)?;
+```
+
+✅ **CORRECT** (Tokenizer pair encoding):
+```rust
+let encoding = tokenizer.encode((query, document), true)?;
+```
+
+### 2. **Why This Matters**
+
+When you use `encode_pair()`, the tokenizer:
+- Automatically adds [CLS] at the start
+- Adds [SEP] after the query
+- Adds [SEP] at the end (for BERT)
+- **Correctly sets token_type_ids**: 0 for query, 1 for document
+- Handles special tokens properly
+
+Manual concatenation will:
+- Add extra [SEP] tokens (you get [SEP] [SEP] in the middle)
+- Set ALL token_type_ids to 0 (incorrect!)
+- Produce different tokenization due to whitespace handling
+
+### 3. **Expected Token Structure**
+
+For input: 
+- Query: "how does authentication work"
+- Document: "Authentication is the process..."
+
+The correct tokenization should be:
+```
+[CLS] how does authentication work [SEP] authentication is the process ... [SEP]
+  0    0   0        0            0    0          1          1   1    1  ...  1
+```
+
+Token type IDs:
+- 0 = Query segment (including [CLS] and first [SEP])
+- 1 = Document segment (including final [SEP])
+
+### 4. **Special Token IDs (for BERT)**
+
+```
+[CLS] = 101
+[SEP] = 102
+[PAD] = 0
+```
+
+### 5. **Verification in Rust**
+
+```rust
+// After tokenization, check:
+let token_ids = encoding.get_ids();
+let type_ids = encoding.get_type_ids();
+
+// First token should be [CLS] (101)
+assert_eq!(token_ids[0], 101);
+
+// Look for [SEP] tokens (102)
+let sep_positions: Vec<_> = token_ids.iter()
+    .enumerate()
+    .filter(|(_, &id)| id == 102)
+    .map(|(i, _)| i)
+    .collect();
+
+// Should have 2 [SEP] tokens for pair encoding
+assert_eq!(sep_positions.len(), 2);
+
+// Check token type IDs switch from 0 to 1 after first [SEP]
+if let Some(first_sep) = sep_positions.first() {
+    // Tokens before first [SEP] should have type 0
+    // Tokens after should have type 1
+}
+```
+
+### 6. **Common Issues**
+
+1. **Using wrong tokenizer**: Make sure you load tokenizer.json from the same model directory
+2. **Not using pair encoding**: Always use `encode_pair()` for cross-encoders
+3. **Missing token type IDs**: These are crucial for BERT to understand query vs document
+
+### 7. **Score Differences**
+
+If you see different scores between Python and Rust:
+1. First check tokenization matches exactly (same token IDs)
+2. Check token type IDs are correct (0 for query, 1 for document)
+3. Verify attention masks are the same
+4. Ensure model weights loaded correctly
+
+### 8. **Debug Output**
+
+Add this to your Rust code to debug:
+```rust
+println!("Token IDs: {:?}", encoding.get_ids());
+println!("Type IDs: {:?}", encoding.get_type_ids()); 
+println!("Attention mask: {:?}", encoding.get_attention_mask());
+```
+
+Compare with Python:
+```python
+print(f"Token IDs: {encoding['input_ids'][0].tolist()}")
+print(f"Type IDs: {encoding['token_type_ids'][0].tolist()}")
+print(f"Attention: {encoding['attention_mask'][0].tolist()}")
+```
+
+## Summary
+
+The key issue is likely that our Rust implementation was using manual concatenation instead of proper pair encoding. This would result in:
+- Wrong token type IDs (all 0s instead of 0s and 1s)
+- Extra [SEP] tokens
+- Different tokenization
+
+Fixing this should improve the model's ability to distinguish between query and document, leading to better discrimination between relevant and irrelevant queries.

package/data/probe/examples/reranker/check_rust_tokenizer.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""
+Check what tokenizer files our Rust implementation is using
+and compare with Python tokenizer output.
+"""
+
+import json
+import os
+
+print("="*80)
+print("CHECKING RUST TOKENIZER CONFIGURATION")
+print("="*80)
+
+# Check what tokenizer files we have
+tokenizer_path = "models/ms-marco-TinyBERT-L-2-v2/tokenizer.json"
+
+if os.path.exists(tokenizer_path):
+    print(f"✓ Found tokenizer at: {tokenizer_path}")
+    
+    # Load and inspect the tokenizer
+    with open(tokenizer_path, 'r') as f:
+        tokenizer_data = json.load(f)
+    
+    print("\n--- TOKENIZER STRUCTURE ---")
+    print(f"Tokenizer type: {tokenizer_data.get('model', {}).get('type', 'Unknown')}")
+    
+    # Check for special tokens
+    if 'added_tokens' in tokenizer_data:
+        print("\nSpecial tokens:")
+        for token in tokenizer_data['added_tokens'][:10]:  # Show first 10
+            print(f"  {token}")
+    
+    # Check post-processor (important for BERT!)
+    if 'post_processor' in tokenizer_data:
+        post_proc = tokenizer_data['post_processor']
+        print(f"\nPost-processor type: {post_proc.get('type', 'Unknown')}")
+        
+        # For BERT, should be TemplateProcessing
+        if post_proc.get('type') == 'TemplateProcessing':
+            if 'single' in post_proc:
+                print(f"Single sequence template: {post_proc['single']}")
+            if 'pair' in post_proc:
+                print(f"Pair sequence template: {post_proc['pair']}")
+else:
+    print(f"❌ Tokenizer not found at: {tokenizer_path}")
+
+# Now let's create a test to verify Rust tokenization
+print("\n" + "="*80)
+print("RUST TOKENIZATION TEST CASES")
+print("="*80)
+
+# These should match Python exactly
+test_cases = [
+    {
+        "name": "Simple pair",
+        "query": "how does authentication work",
+        "document": "Authentication is the process of verifying the identity of a user.",
+        "method": "pair"  # tokenizer.encode_pair(query, document)
+    },
+    {
+        "name": "Manual concat (wrong)",
+        "text": "how does authentication work [SEP] Authentication is the process of verifying the identity of a user.",
+        "method": "single"  # tokenizer.encode(text)
+    }
+]
+
+print("\nExpected Rust code for correct tokenization:")
+print("```rust")
+print('// CORRECT: Use encode_pair for cross-encoder')
+print('let encoding = tokenizer.encode((query, document), true)?;')
+print('')
+print('// WRONG: Do not manually concatenate')
+print('let text = format!("{} [SEP] {}", query, document);')
+print('let encoding = tokenizer.encode(text, true)?;')
+print("```")
+
+# Key differences to check
+print("\n--- KEY THINGS TO VERIFY IN RUST ---")
+print("1. Token IDs match exactly")
+print("2. Token type IDs are generated correctly:")
+print("   - 0 for query tokens (including [CLS])")
+print("   - 0 for first [SEP]") 
+print("   - 1 for document tokens")
+print("   - 1 for final [SEP] (if present)")
+print("3. Special tokens are in the right positions")
+print("4. Padding is handled correctly")
+
+# Load Python results if available
+if os.path.exists("tokenizer_debug_info.json"):
+    with open("tokenizer_debug_info.json", 'r') as f:
+        python_info = json.load(f)
+    
+    print("\n--- PYTHON REFERENCE ---")
+    print(f"Query: '{python_info['test_case']['query']}'")
+    print(f"Document: '{python_info['test_case']['document']}'")
+    print(f"Correct score: {python_info['test_case']['correct_score']:.6f}")
+    print(f"Manual concat score: {python_info['test_case']['manual_concat_score']:.6f}")
+    
+    # Show first 20 tokens
+    ids = python_info['test_case']['correct_input_ids'][:20]
+    types = python_info['test_case']['correct_token_types'][:20] if python_info['test_case']['correct_token_types'] else None
+    
+    print("\nFirst 20 tokens (Python):")
+    print(f"IDs: {ids}")
+    if types:
+        print(f"Types: {types}")
+    
+    print("\n✅ Your Rust implementation should produce these EXACT token IDs and types!")