@rfxlamia/skillkit 1.0.0 → 1.2.0

package/skills/skills/framework-critical-thinking/references/reflection_trigger.md

@@ -0,0 +1,78 @@
+# Reflection Trigger
+
+## Purpose
+
+The Reflection Trigger provides rule-based conditions for activating self-correction loops. It determines when an agent should pause, evaluate its progress, and potentially adjust its approach.
+
+## Trigger Conditions
+
+### 1. Confidence Threshold Violations
+
+```python
+class ConfidenceTrigger:
+    def __init__(self, thresholds):
+        self.thresholds = {
+            'critical': 0.3,
+            'warning': 0.5,
+            'caution': 0.7
+        }
+
+    def check(self, current_state):
+        confidence = current_state.get('confidence', 1.0)
+        if confidence < self.thresholds['critical']:
+            return TriggerResult(
+                should_reflect=True,
+                priority='critical',
+                reason=f"Confidence {confidence:.2f} critically low"
+            )
+        return TriggerResult(should_reflect=False)
+```
+
+### 2. Repeated Action Patterns
+
+```python
+class PatternTrigger:
+    def check(self, action_history):
+        if len(action_history) < 3:
+            return TriggerResult(should_reflect=False)
+
+        recent = action_history[-3:]
+        if self.are_identical(recent):
+            return TriggerResult(
+                should_reflect=True,
+                priority='high',
+                reason="Same action repeated 3 times"
+            )
+        return TriggerResult(should_reflect=False)
+```
+
+### 3. Latency Spikes
+
+```python
+class LatencyTrigger:
+    def check(self, elapsed_time, expected_time):
+        if elapsed_time > expected_time * 5:
+            return TriggerResult(
+                should_reflect=True,
+                priority='critical',
+                reason="Latency 5x expected - possible infinite loop"
+            )
+        return TriggerResult(should_reflect=False)
+```
+
+## Usage Example
+
+```python
+trigger = ReflectionTrigger(config={'confidence': {'critical': 0.3}})
+
+for step_number, state in enumerate(agent_states):
+    result = trigger.should_reflect(state, step_number)
+    if result.should_reflect:
+        reflection = generate_reflection(state)
+        state = apply_reflection(state, reflection)
+```
+
+---
+
+**Sources:**
+- [Self-Corrective Agent Architecture](https://www.emergentmind.com/topics/self-corrective-agent-architecture)

package/skills/skills/framework-critical-thinking/references/self_verification.md

@@ -0,0 +1,388 @@
+# Self-Verification
+
+## Table of Contents
+
+- [Purpose](#purpose)
+- [Core Techniques](#core-techniques)
+  - [Chain-of-Verification (CoVe)](#1-chain-of-verification-cove)
+  - [Backward Verification](#2-backward-verification)
+  - [Self-Consistency Check](#3-self-consistency-check)
+  - [Cross-Verification](#4-cross-verification-with-external-sources)
+- [Verification Strategies by Task Type](#verification-strategies-by-task-type)
+- [Self-Refine Loop](#self-refine-loop)
+- [Implementation Example](#implementation-example)
+- [Best Practices](#best-practices)
+- [Common Pitfalls](#common-pitfalls)
+
+## Purpose
+
+Self-Verification provides mechanisms for AI agents to validate their own outputs before delivery. It implements techniques like Chain-of-Verification (CoVe), backward verification, and cross-verification to catch errors and hallucinations.
+
+## Core Techniques
+
+### 1. Chain-of-Verification (CoVe)
+
+A multi-stage verification process where the model generates verification questions and answers them to validate its initial output.
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  Stage 1: Baseline Response                                 │
+│  Generate initial answer to the query                       │
+└──────────────────────┬──────────────────────────────────────┘
+                       │
+                       ▼
+┌─────────────────────────────────────────────────────────────┐
+│  Stage 2: Plan Verification                                 │
+│  Generate verification questions based on baseline          │
+└──────────────────────┬──────────────────────────────────────┘
+                       │
+                       ▼
+┌─────────────────────────────────────────────────────────────┐
+│  Stage 3: Execute Verification                              │
+│  Answer verification questions independently                │
+└──────────────────────┬──────────────────────────────────────┘
+                       │
+                       ▼
+┌─────────────────────────────────────────────────────────────┐
+│  Stage 4: Generate Final Verified Response                  │
+│  Incorporate verification results into final output         │
+└─────────────────────────────────────────────────────────────┘
+```
+
+**Implementation:**
+
+```python
+class ChainOfVerification:
+    def verify(self, query, baseline_response):
+        # Stage 2: Generate verification questions
+        verification_questions = self.generate_verification_questions(
+            query, baseline_response
+        )
+
+        # Stage 3: Execute verification
+        verified_facts = []
+        for question in verification_questions:
+            # Answer independently, without reference to baseline
+            answer = self.answer_independently(question)
+            verified_facts.append({
+                'question': question,
+                'answer': answer,
+                'matches_baseline': self.check_consistency(
+                    answer, baseline_response
+                )
+            })
+
+        # Stage 4: Generate final response
+        if all(f['matches_baseline'] for f in verified_facts):
+            return baseline_response
+        else:
+            return self.revise_response(baseline_response, verified_facts)
+
+    def generate_verification_questions(self, query, response):
+        """Extract claims and generate questions to verify them."""
+        claims = self.extract_claims(response)
+        questions = []
+        for claim in claims:
+            questions.extend(self.claim_to_questions(claim))
+        return questions
+```
+
+**Template for Verification Planning:**
+```
+Given the following claim from a response:
+"{claim}"
+
+Generate 2-3 specific questions that would verify the accuracy of this claim.
+Questions should be answerable independently without referencing the original claim.
+
+Verification questions:
+1.
+2.
+3.
+```
+
+### 2. Backward Verification
+
+For mathematical or logical problems, verify by working backward from the conclusion to see if it leads to the original problem conditions.
+
+```python
+class BackwardVerification:
+    def verify(self, problem, solution_steps, final_answer):
+        """Verify by reversing the solution."""
+        # Start from final answer
+        current = final_answer
+
+        # Apply inverse operations in reverse order
+        for step in reversed(solution_steps):
+            inverse_op = self.get_inverse_operation(step.operation)
+            current = inverse_op.apply(current, step.operand)
+
+        # Check if we return to original problem conditions
+        return self.matches_original(current, problem)
+
+    def get_inverse_operation(self, operation):
+        inverses = {
+            'add': 'subtract',
+            'multiply': 'divide',
+            'square': 'square_root',
+            'append': 'remove_last'
+        }
+        return inverses.get(operation)
+```
+
+**Example:**
+```
+Problem: "If x + 5 = 12, what is x?"
+Forward: x = 12 - 5 = 7
+Backward: 7 + 5 = 12 ✓ (matches original equation)
+```
+
+### 3. Self-Consistency Check
+
+Generate multiple reasoning paths and check if they converge to the same answer.
+
+```python
+class SelfConsistency:
+    def __init__(self, sample_count=10, temperature=0.7):
+        self.sample_count = sample_count
+        self.temperature = temperature
+
+    def verify(self, problem):
+        # Generate multiple reasoning paths
+        answers = []
+        for i in range(self.sample_count):
+            reasoning, answer = self.generate_with_temperature(
+                problem, self.temperature
+            )
+            answers.append({
+                'reasoning': reasoning,
+                'answer': answer,
+                'confidence': self.assess_reasoning_quality(reasoning)
+            })
+
+        # Aggregate answers
+        answer_counts = Counter(a['answer'] for a in answers)
+        most_common = answer_counts.most_common(1)[0]
+
+        # Calculate consistency score
+        consistency_score = most_common[1] / len(answers)
+
+        return {
+            'final_answer': most_common[0],
+            'consistency_score': consistency_score,
+            'all_answers': answers,
+            'confidence': self.compute_aggregate_confidence(answers)
+        }
+```
+
+### 4. Cross-Verification with External Sources
+
+When external knowledge bases or tools are available, verify claims against them.
+
+```python
+class CrossVerification:
+    def __init__(self, knowledge_base, tools):
+        self.kb = knowledge_base
+        self.tools = tools
+
+    def verify_claim(self, claim):
+        verification_result = {
+            'claim': claim,
+            'verified': False,
+            'sources': [],
+            'confidence': 0.0
+        }
+
+        # Query knowledge base
+        kb_results = self.kb.query(claim)
+        if kb_results:
+            verification_result['sources'].extend(kb_results)
+            verification_result['kb_match'] = self.check_match(
+                claim, kb_results
+            )
+
+        # Use tools if applicable
+        if self.is_computable(claim):
+            tool_result = self.compute_with_tools(claim)
+            verification_result['tool_result'] = tool_result
+            verification_result['tool_match'] = self.check_match(
+                claim, tool_result
+            )
+
+        # Aggregate verification
+        verification_result['verified'] = self.aggregate_verification(
+            verification_result
+        )
+
+        return verification_result
+```
+
+## Verification Strategies by Task Type
+
+### Factual Claims
+
+Use CoVe with web search or knowledge base verification:
+
+```python
+factual_verification = {
+    'technique': 'CoVe',
+    'question_generation': 'claim_extraction',
+    'verification_source': ['knowledge_base', 'web_search'],
+    'revision_strategy': 'correct_or_hedge'
+}
+```
+
+### Mathematical Reasoning
+
+Use backward verification + computation tools:
+
+```python
+math_verification = {
+    'technique': 'backward_verification',
+    'tools': ['calculator', 'sympy', 'wolfram_alpha'],
+    'check_steps': True,
+    'precision_tolerance': 1e-10
+}
+```
+
+### Logical Arguments
+
+Use consistency checking and counter-example search:
+
+```python
+logic_verification = {
+    'technique': 'consistency_checking',
+    'methods': ['truth_table', 'counter_example_search'],
+    'validate_premises': True,
+    'check_entailment': True
+}
+```
+
+### Code Generation
+
+Use execution and test validation:
+
+```python
+code_verification = {
+    'technique': 'execution_based',
+    'steps': [
+        'syntax_check',
+        'static_analysis',
+        'test_execution',
+        'edge_case_testing'
+    ],
+    'timeout': 30
+}
+```
+
+## Self-Refine Loop
+
+Iterative improvement through self-critique:
+
+```python
+class SelfRefine:
+    def __init__(self, max_iterations=3):
+        self.max_iterations = max_iterations
+
+    def refine(self, initial_output, task_description):
+        current = initial_output
+
+        for iteration in range(self.max_iterations):
+            # Generate critique
+            critique = self.generate_critique(current, task_description)
+
+            # Check if satisfactory
+            if critique['is_satisfactory']:
+                break
+
+            # Refine based on critique
+            current = self.apply_refinement(current, critique)
+
+        return {
+            'final_output': current,
+            'iterations': iteration + 1,
+            'critiques': critique_history
+        }
+
+    def generate_critique(self, output, task):
+        critique_prompt = f"""
+Task: {task}
+Output: {output}
+
+Critique this output. Identify:
+1. Factual errors or hallucinations
+2. Logical inconsistencies
+3. Missing information
+4. Unclear explanations
+5. Areas for improvement
+
+Is this output satisfactory? (Yes/No with explanation)
+"""
+        return self.llm.generate(critique_prompt)
+```
+
+## Implementation Example
+
+```python
+class SelfVerificationSystem:
+    def __init__(self):
+        self.cove = ChainOfVerification()
+        self.backward = BackwardVerification()
+        self.consistency = SelfConsistency()
+        self.refine = SelfRefine()
+
+    def verify_output(self, task_type, problem, output):
+        verification_pipeline = self.get_pipeline(task_type)
+
+        results = {}
+        for technique in verification_pipeline:
+            if technique == 'CoVe':
+                results['cove'] = self.cove.verify(problem, output)
+            elif technique == 'backward':
+                results['backward'] = self.backward.verify(
+                    problem, output['steps'], output['answer']
+                )
+            elif technique == 'consistency':
+                results['consistency'] = self.consistency.verify(problem)
+
+        # Aggregate verification results
+        final_confidence = self.aggregate_confidence(results)
+
+        # Refine if needed
+        if final_confidence < 0.8:
+            refined = self.refine.refine(output, problem)
+            return {
+                'verified_output': refined['final_output'],
+                'confidence': final_confidence,
+                'verification_details': results,
+                'was_refined': True
+            }
+
+        return {
+            'verified_output': output,
+            'confidence': final_confidence,
+            'verification_details': results,
+            'was_refined': False
+        }
+```
+
+## Best Practices
+
+1. **Independent Verification:** Answer verification questions without reference to the baseline response
+2. **Multiple Techniques:** Combine different verification methods for robust validation
+3. **Graceful Degradation:** If verification fails, hedge claims or request human review
+4. **Budget Management:** Limit verification iterations to control computational cost
+5. **Calibration:** Track verification success rates and adjust confidence thresholds
+
+## Common Pitfalls
+
+- **Verification Hallucination:** Model hallucinates during verification phase
+- **Confirmation Bias:** Verification favors the baseline response
+- **Over-Verification:** Spending too much compute on simple, reliable outputs
+- **False Confidence:** High verification scores for incorrect but internally consistent outputs
+
+---
+
+**Sources:**
+- [Chain-of-Verification Reduces Hallucination in LLMs](https://arxiv.org/abs/2309.11495)
+- [Self-Verification Prompting](https://learnprompting.org/docs/advanced/self_criticism/self_verification)

package/skills/skills/framework-critical-thinking/references/uncertainty_quantifier.md

@@ -0,0 +1,207 @@
+# Uncertainty Quantifier
+
+## Purpose
+
+The Uncertainty Quantifier calibrates confidence scores to match actual accuracy, ensuring that stated confidence levels are reliable indicators of correctness.
+
+## Confidence Calibration
+
+### The Calibration Problem
+
+Models often exhibit:
+- **Overconfidence:** Stating high confidence for incorrect answers
+- **Underconfidence:** Stating low confidence for correct answers
+
+### Calibration Methods
+
+#### 1. Temperature Scaling
+
+```python
+class TemperatureScaler:
+    def __init__(self):
+        self.temperature = 1.0
+
+    def fit(self, confidences, accuracies):
+        """Learn optimal temperature on validation set."""
+        def nll_loss(T):
+            scaled = self.scale_confidences(confidences, T)
+            return -np.mean(
+                accuracies * np.log(scaled) +
+                (1 - accuracies) * np.log(1 - scaled)
+            )
+
+        self.temperature = minimize(nll_loss, x0=1.0).x[0]
+
+    def calibrate(self, confidence):
+        """Apply temperature scaling."""
+        return self.scale_confidences(confidence, self.temperature)
+
+    def scale_confidences(self, confidences, temperature):
+        return 1 / (1 + np.exp(-np.log(confidences / (1 - confidences)) / temperature))
+```
+
+#### 2. Platt Scaling
+
+```python
+class PlattScaler:
+    def __init__(self):
+        self.a = 1.0
+        self.b = 0.0
+
+    def fit(self, confidences, accuracies):
+        """Fit sigmoid parameters."""
+        from scipy.optimize import minimize
+
+        def loss(params):
+            a, b = params
+            calibrated = 1 / (1 + np.exp(-(a * confidences + b)))
+            return -np.mean(
+                accuracies * np.log(calibrated + 1e-10) +
+                (1 - accuracies) * np.log(1 - calibrated + 1e-10)
+            )
+
+        result = minimize(loss, x0=[1.0, 0.0])
+        self.a, self.b = result.x
+
+    def calibrate(self, confidence):
+        """Apply Platt scaling."""
+        return 1 / (1 + np.exp(-(self.a * confidence + self.b)))
+```
+
+#### 3. Isotonic Regression
+
+```python
+class IsotonicCalibrator:
+    def __init__(self):
+        from sklearn.isotonic import IsotonicRegression
+        self.iso = IsotonicRegression(out_of_bounds='clip')
+
+    def fit(self, confidences, accuracies):
+        """Fit isotonic regression."""
+        self.iso.fit(confidences, accuracies)
+
+    def calibrate(self, confidence):
+        """Apply isotonic calibration."""
+        return self.iso.predict([confidence])[0]
+```
+
+## Calibration Metrics
+
+### Expected Calibration Error (ECE)
+
+```python
+def expected_calibration_error(confidences, predictions, labels, n_bins=10):
+    """Calculate ECE - lower is better."""
+    bin_boundaries = np.linspace(0, 1, n_bins + 1)
+    bin_lowers = bin_boundaries[:-1]
+    bin_uppers = bin_boundaries[1:]
+
+    ece = 0.0
+    for lower, upper in zip(bin_lowers, bin_uppers):
+        in_bin = (confidences > lower) & (confidences <= upper)
+        prop_in_bin = in_bin.mean()
+
+        if prop_in_bin > 0:
+            accuracy_in_bin = (predictions[in_bin] == labels[in_bin]).mean()
+            avg_confidence_in_bin = confidences[in_bin].mean()
+            ece += np.abs(avg_confidence_in_bin - accuracy_in_bin) * prop_in_bin
+
+    return ece
+```
+
+### Maximum Calibration Error (MCE)
+
+```python
+def maximum_calibration_error(confidences, predictions, labels, n_bins=10):
+    """Calculate MCE - worst-case calibration error."""
+    bin_boundaries = np.linspace(0, 1, n_bins + 1)
+    errors = []
+
+    for i in range(n_bins):
+        lower, upper = bin_boundaries[i], bin_boundaries[i + 1]
+        in_bin = (confidences > lower) & (confidences <= upper)
+
+        if in_bin.sum() > 0:
+            accuracy = (predictions[in_bin] == labels[in_bin]).mean()
+            confidence = confidences[in_bin].mean()
+            errors.append(np.abs(confidence - accuracy))
+
+    return max(errors) if errors else 0.0
+```
+
+### Brier Score
+
+```python
+def brier_score(confidences, outcomes):
+    """Calculate Brier score - lower is better."""
+    return np.mean((confidences - outcomes) ** 2)
+```
+
+## Confidence Intervals
+
+```python
+class ConfidenceIntervalEstimator:
+    def __init__(self, confidence_level=0.95):
+        self.confidence_level = confidence_level
+
+    def estimate_interval(self, predictions, variances):
+        """Estimate confidence intervals."""
+        from scipy import stats
+
+        z_score = stats.norm.ppf((1 + self.confidence_level) / 2)
+
+        lower = predictions - z_score * np.sqrt(variances)
+        upper = predictions + z_score * np.sqrt(variances)
+
+        return {'lower': lower, 'upper': upper, 'confidence': self.confidence_level}
+
+    def monte_carlo_dropout(self, model, input_data, n_samples=100):
+        """Estimate uncertainty using MC dropout."""
+        predictions = []
+        model.train()  # Enable dropout
+
+        for _ in range(n_samples):
+            pred = model(input_data)
+            predictions.append(pred)
+
+        predictions = torch.stack(predictions)
+        mean = predictions.mean(dim=0)
+        variance = predictions.var(dim=0)
+
+        return {'mean': mean, 'variance': variance}
+```
+
+## Usage Example
+
+```python
+# Collect calibration data
+confidences = []
+accuracies = []
+
+for task in validation_set:
+    confidence, prediction, actual = agent.solve_with_confidence(task)
+    confidences.append(confidence)
+    accuracies.append(prediction == actual)
+
+# Fit calibrator
+calibrator = TemperatureScaler()
+calibrator.fit(confidences, accuracies)
+
+# Use calibrated confidence
+raw_confidence = agent.get_confidence(new_task)
+calibrated = calibrator.calibrate(raw_confidence)
+
+print(f"Raw: {raw_confidence:.2f}, Calibrated: {calibrated:.2f}")
+```
+
+## Best Practices
+
+1. **Use held-out calibration set** - Don't calibrate on training data
+2. **Monitor ECE regularly** - Recalibrate if drift detected
+3. **Consider task-specific calibration** - Different tasks may need different calibrators
+4. **Report uncertainty** - Always provide confidence intervals, not just point estimates
+
+---
+
+**Sources:**
+- [On Calibration of Modern Neural Networks](https://arxiv.org/abs/1706.04599)