omgkit 2.5.1 → 2.6.0

package/plugin/skills/ai-engineering/rag-systems/SKILL.md

@@ -0,0 +1,137 @@
+---
+name: rag-systems
+description: Retrieval-Augmented Generation - chunking strategies, embedding, vector search, hybrid retrieval, reranking, query transformation. Use when building RAG pipelines, knowledge bases, or context-augmented applications.
+---
+
+# RAG Systems
+
+Building Retrieval-Augmented Generation systems.
+
+## RAG Architecture
+
+```
+INDEXING (Offline)
+Documents → Chunking → Embedding → Vector DB
+
+QUERYING (Online)
+Query → Embed → Search → Retrieved Docs
+                              ↓
+Response ← LLM ← Context + Query
+```
+
+## Retrieval Algorithms
+
+### Term-Based (BM25)
+```python
+from rank_bm25 import BM25Okapi
+
+tokenized_docs = [doc.split() for doc in documents]
+bm25 = BM25Okapi(tokenized_docs)
+scores = bm25.get_scores(query.split())
+```
+
+### Embedding-Based
+```python
+from sentence_transformers import SentenceTransformer
+import faiss
+
+model = SentenceTransformer('all-MiniLM-L6-v2')
+embeddings = model.encode(documents)
+
+index = faiss.IndexFlatIP(embeddings.shape[1])
+faiss.normalize_L2(embeddings)
+index.add(embeddings)
+
+# Query
+query_emb = model.encode([query])
+faiss.normalize_L2(query_emb)
+distances, indices = index.search(query_emb, k=5)
+```
+
+### Hybrid Retrieval
+```python
+def hybrid_retrieve(query, k=5, alpha=0.5):
+    bm25_scores = normalize(bm25.get_scores(query.split()))
+    dense_scores = normalize(index.search(embed(query), len(docs))[0])
+
+    hybrid = alpha * bm25_scores + (1-alpha) * dense_scores
+    return [docs[i] for i in np.argsort(hybrid)[::-1][:k]]
+```
+
+## Chunking Strategies
+
+### Fixed Size
+```python
+def fixed_chunk(text, size=500, overlap=50):
+    chunks = []
+    for i in range(0, len(text), size - overlap):
+        chunks.append(text[i:i+size])
+    return chunks
+```
+
+### Semantic Chunking
+```python
+def semantic_chunk(text, model, threshold=0.5):
+    sentences = sent_tokenize(text)
+    chunks, current = [], []
+
+    for sent in sentences:
+        current.append(sent)
+        if len(current) > 1:
+            sim = similarity(current[-2], current[-1], model)
+            if sim < threshold:
+                chunks.append(" ".join(current[:-1]))
+                current = [sent]
+
+    if current:
+        chunks.append(" ".join(current))
+    return chunks
+```
+
+## Retrieval Optimization
+
+### Query Expansion
+```python
+def expand_query(query, model):
+    prompt = f"Generate 3 alternative phrasings:\n{query}"
+    return [query] + model.generate(prompt).split("\n")
+```
+
+### HyDE (Hypothetical Document)
+```python
+def hyde(query, model):
+    prompt = f"Write a paragraph answering:\n{query}"
+    return model.generate(prompt)  # Use this for retrieval
+```
+
+### Reranking
+```python
+from sentence_transformers import CrossEncoder
+
+reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
+
+def rerank(query, docs, k=5):
+    pairs = [(query, doc) for doc in docs]
+    scores = reranker.predict(pairs)
+    return sorted(zip(docs, scores), key=lambda x: -x[1])[:k]
+```
+
+## RAG Evaluation
+
+```python
+def rag_metrics(query, response, context, ground_truth):
+    return {
+        "retrieval_precision": precision(retrieved, relevant),
+        "retrieval_recall": recall(retrieved, relevant),
+        "answer_relevance": similarity(response, ground_truth),
+        "faithfulness": check_hallucination(response, context),
+    }
+```
+
+## Best Practices
+
+1. Use hybrid retrieval (BM25 + dense)
+2. Add reranking for quality
+3. Chunk with overlap (10-20%)
+4. Experiment with chunk sizes (200-1000 tokens)
+5. Evaluate retrieval separately from generation

package/plugin/skills/ai-engineering/user-feedback/SKILL.md

@@ -0,0 +1,162 @@
+---
+name: user-feedback
+description: Collecting and using user feedback - explicit/implicit signals, feedback analysis, improvement loops, A/B testing. Use when improving AI systems, understanding user satisfaction, or iterating on quality.
+---
+
+# User Feedback Skill
+
+Leveraging feedback to improve AI systems.
+
+## Feedback Collection
+
+### Explicit Feedback
+```python
+class FeedbackCollector:
+    def collect_explicit(self, response_id, feedback):
+        self.db.save({
+            "type": "explicit",
+            "response_id": response_id,
+            "rating": feedback.get("rating"),      # 1-5
+            "thumbs": feedback.get("thumbs"),      # up/down
+            "comment": feedback.get("comment"),
+            "timestamp": datetime.now()
+        })
+```
+
+### Implicit Feedback
+```python
+def extract_implicit(conversation):
+    signals = []
+
+    for i, turn in enumerate(conversation[1:], 1):
+        prev = conversation[i-1]
+
+        # Negative signals
+        if is_correction(turn, prev):
+            signals.append(("correction", i))
+        if is_repetition(turn, prev):
+            signals.append(("repetition", i))
+        if is_abandonment(turn):
+            signals.append(("abandonment", i))
+
+        # Positive signals
+        if is_acceptance(turn, prev):
+            signals.append(("acceptance", i))
+        if is_follow_up(turn, prev):
+            signals.append(("engagement", i))
+
+    return signals
+```
+
+### Natural Language Feedback
+```python
+def extract_from_text(turn, model):
+    prompt = f"""Extract feedback signal from user message.
+
+Message: {turn}
+
+Sentiment (positive/negative/neutral):
+Specific issue (if any):
+Suggestion (if any):"""
+
+    return model.generate(prompt)
+```
+
+## Feedback Analysis
+
+```python
+class FeedbackAnalyzer:
+    def categorize(self, feedbacks):
+        prompt = f"""Categorize these feedback items:
+
+{json.dumps(feedbacks)}
+
+Categories:
+1. Accuracy issues
+2. Format issues
+3. Relevance issues
+4. Safety issues
+5. Missing features
+
+Summary:"""
+        return self.llm.generate(prompt)
+
+    def find_patterns(self, feedbacks):
+        # Cluster similar complaints
+        embeddings = [self.embed(f["text"]) for f in feedbacks]
+        clusters = self.cluster(embeddings)
+
+        patterns = {}
+        for cluster_id, indices in clusters.items():
+            cluster_feedback = [feedbacks[i] for i in indices]
+            patterns[cluster_id] = {
+                "count": len(cluster_feedback),
+                "summary": self.summarize(cluster_feedback),
+                "examples": cluster_feedback[:3]
+            }
+
+        return patterns
+```
+
+## Improvement Loop
+
+```python
+class FeedbackLoop:
+    def run_cycle(self):
+        # 1. Collect
+        recent = self.db.get_recent(days=7)
+        analysis = self.analyze(recent)
+
+        # 2. Identify improvements
+        if analysis["accuracy_issues"] > threshold:
+            training_data = self.create_training_data(
+                analysis["corrections"]
+            )
+
+            # 3. Improve
+            if len(training_data) > 1000:
+                self.finetune(training_data)
+            else:
+                self.update_prompts(analysis)
+
+        # 4. Evaluate
+        metrics = self.evaluate(self.test_set)
+
+        # 5. Deploy if improved
+        if metrics["quality"] > self.baseline:
+            self.deploy()
+
+        return metrics
+```
+
+## A/B Testing
+
+```python
+class ABTest:
+    def __init__(self, variants):
+        self.variants = variants
+        self.results = {v: {"count": 0, "positive": 0} for v in variants}
+
+    def assign(self, user_id):
+        # Consistent assignment
+        return self.variants[hash(user_id) % len(self.variants)]
+
+    def record(self, user_id, positive):
+        variant = self.assign(user_id)
+        self.results[variant]["count"] += 1
+        if positive:
+            self.results[variant]["positive"] += 1
+
+    def analyze(self):
+        for variant, data in self.results.items():
+            rate = data["positive"] / max(data["count"], 1)
+            print(f"{variant}: {rate:.2%} ({data['count']} samples)")
+```
+
+## Best Practices
+
+1. Collect both explicit and implicit feedback
+2. Analyze patterns, not individual feedback
+3. Close the loop (feedback → improvement)
+4. A/B test changes
+5. Monitor long-term trends