maestro-bundle 1.3.1 → 1.4.0

package/templates/bundle-data-pipeline/skills/rag-pipeline/SKILL.md

@@ -1,54 +1,96 @@
 ---
 name: rag-pipeline
-description: Construir pipeline RAG completo com ingestão, chunking, embedding, indexação e retrieval usando LangChain + pgvector. Use sempre que precisar implementar busca semântica, responder perguntas sobre documentos, ou criar um sistema de retrieval.
+description: Build a complete RAG pipeline with document ingestion, chunking, embedding, vector indexing, and hybrid retrieval using LangChain and pgvector. Use when you need to implement semantic search, answer questions over documents, or create a retrieval-augmented generation system.
+version: 1.0.0
+author: Maestro
 ---
 
 # RAG Pipeline
 
-## Pipeline completo
+Build production-ready retrieval-augmented generation systems with LangChain, pgvector, and hybrid search.
 
-```
-Documentos → Loader → Splitter → Embeddings → pgvector → Retriever → Re-ranker → LLM
+## When to Use
+- User needs to build a Q&A system over internal documents
+- User wants to implement semantic search on a document corpus
+- User needs to ingest and chunk documents for vector indexing
+- User wants hybrid retrieval (semantic + keyword) with re-ranking
+- User needs to set up a pgvector-backed vector store
+
+## Available Operations
+1. Ingest documents (Markdown, PDF, text) with LangChain loaders
+2. Split documents into chunks with RecursiveCharacterTextSplitter
+3. Generate embeddings and store in pgvector
+4. Build hybrid retrieval (semantic + BM25 keyword search)
+5. Add Cohere re-ranking for precision
+6. Create a query chain with LLM and source attribution
+
+## Multi-Step Workflow
+
+### Step 1: Install Dependencies
+```bash
+pip install langchain langchain-openai langchain-community langchain-postgres langchain-cohere pgvector psycopg2-binary unstructured rank-bm25
 ```
 
-## 1. Ingestão
+Set required environment variables:
+```bash
+export OPENAI_API_KEY="sk-..."
+export COHERE_API_KEY="..."
+export DATABASE_URL="postgresql://user:pass@localhost:5432/vectordb"
+```
 
+### Step 2: Ingest Documents
 ```python
 from langchain_community.document_loaders import DirectoryLoader, UnstructuredMarkdownLoader
-from langchain.text_splitter import RecursiveCharacterTextSplitter
 
-# Loader por tipo de documento
+# Load all markdown files from a directory
 loader = DirectoryLoader(
     "./documents/",
     glob="**/*.md",
-    loader_cls=UnstructuredMarkdownLoader
+    loader_cls=UnstructuredMarkdownLoader,
+    show_progress=True
 )
 docs = loader.load()
+print(f"Loaded {len(docs)} documents")
+```
+
+For PDF files:
+```python
+from langchain_community.document_loaders import PyPDFLoader
+
+loader = PyPDFLoader("documents/report.pdf")
+docs = loader.load()
+```
+
+### Step 3: Split into Chunks
+```python
+from langchain.text_splitter import RecursiveCharacterTextSplitter
 
-# Splitter com separadores Markdown
 splitter = RecursiveCharacterTextSplitter(
     chunk_size=1000,
     chunk_overlap=200,
     separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "]
 )
 chunks = splitter.split_documents(docs)
+print(f"Split into {len(chunks)} chunks")
 ```
 
-## 2. Metadados obrigatórios
-
-Cada chunk deve ter:
+### Step 4: Enrich Metadata
 ```python
+from datetime import datetime
+
 for chunk in chunks:
     chunk.metadata.update({
         "source": chunk.metadata.get("source", "unknown"),
-        "doc_type": classify_document(chunk),  # skill, agent_md, prd, code
-        "language": detect_language(chunk),
-        "created_at": datetime.now().isoformat(),
+        "doc_type": chunk.metadata.get("source", "").split(".")[-1],
+        "indexed_at": datetime.now().isoformat(),
+        "chunk_size": len(chunk.page_content),
     })
-```
 
-## 3. Embedding + Indexação
+# Verify metadata
+print(chunks[0].metadata)
+```
 
+### Step 5: Generate Embeddings and Store in pgvector
 ```python
 from langchain_openai import OpenAIEmbeddings
 from langchain_postgres import PGVector
@@ -61,29 +103,28 @@ vectorstore = PGVector(
     embedding_function=embeddings,
 )
 vectorstore.add_documents(chunks)
+print(f"Indexed {len(chunks)} chunks in pgvector")
 ```
 
-## 4. Retrieval Híbrido
-
+### Step 6: Build Hybrid Retriever (Semantic + BM25)
 ```python
 from langchain.retrievers import EnsembleRetriever
 from langchain_community.retrievers import BM25Retriever
 
-# Semântico
+# Semantic retriever
 semantic_retriever = vectorstore.as_retriever(search_kwargs={"k": 20})
 
-# Keyword
+# Keyword retriever (BM25)
 bm25_retriever = BM25Retriever.from_documents(chunks, k=20)
 
-# Ensemble com RRF
+# Combine with Reciprocal Rank Fusion
 hybrid_retriever = EnsembleRetriever(
     retrievers=[semantic_retriever, bm25_retriever],
-    weights=[0.6, 0.4]
+    weights=[0.6, 0.4]  # favor semantic
 )
 ```
 
-## 5. Re-ranking
-
+### Step 7: Add Re-Ranking
 ```python
 from langchain.retrievers import ContextualCompressionRetriever
 from langchain_cohere import CohereRerank
@@ -95,34 +136,81 @@ final_retriever = ContextualCompressionRetriever(
 )
 ```
 
-## 6. Query Chain
-
+### Step 8: Build Query Chain
 ```python
+from langchain_openai import ChatOpenAI
 from langchain_core.prompts import ChatPromptTemplate
 from langchain_core.output_parsers import StrOutputParser
+from langchain_core.runnables import RunnablePassthrough
+
+llm = ChatOpenAI(model="gpt-4o", temperature=0)
 
 prompt = ChatPromptTemplate.from_template("""
-Responda a pergunta baseado apenas no contexto fornecido.
-Se a resposta não estiver no contexto, diga "Não encontrei essa informação".
+Answer the question based only on the provided context.
+If the answer is not in the context, say "I could not find that information."
 
-Contexto: {context}
-Pergunta: {question}
+Context: {context}
+Question: {question}
 """)
 
+def format_docs(docs):
+    return "\n\n".join(doc.page_content for doc in docs)
+
 chain = (
-    {"context": final_retriever, "question": RunnablePassthrough()}
+    {"context": final_retriever | format_docs, "question": RunnablePassthrough()}
     | prompt
     | llm
     | StrOutputParser()
 )
 
-result = chain.invoke("Qual skill usar para criar componentes React?")
+# Test the chain
+result = chain.invoke("What is the recommended chunk size for markdown documents?")
+print(result)
 ```
 
-## Checklist de qualidade
+### Step 9: Verify Retrieval Quality
+```python
+# Test retrieval with known questions
+test_queries = [
+    "How do I set up authentication?",
+    "What database should I use?",
+    "How do I deploy with Docker?",
+]
+
+for query in test_queries:
+    docs = final_retriever.invoke(query)
+    print(f"\nQuery: {query}")
+    print(f"Top result source: {docs[0].metadata.get('source', 'unknown')}")
+    print(f"Top result preview: {docs[0].page_content[:200]}...")
+```
 
-- [ ] Chunks testados com perguntas reais
-- [ ] Metadados completos em todos os chunks
-- [ ] Retrieval quality medido com golden dataset
-- [ ] Re-ranking ativo para refinar top-k
-- [ ] Fallback para quando retrieval não encontra nada
+## Resources
+- `references/chunking-strategies.md` - Guide to chunk sizes, overlap, and separators
+- `references/embedding-models.md` - Comparison of embedding model options
+
+## Examples
+### Example 1: Build Q&A Over Documentation
+User asks: "Set up a RAG system to answer questions about our API docs"
+Response approach:
+1. Load docs from the docs/ directory with DirectoryLoader
+2. Split with RecursiveCharacterTextSplitter (1000 chars, 200 overlap)
+3. Embed with OpenAI text-embedding-3-large and store in pgvector
+4. Build hybrid retriever with BM25 fallback
+5. Create query chain with GPT-4o
+6. Test with sample questions and verify source attribution
+
+### Example 2: Add Semantic Search to Existing App
+User asks: "Add search to our knowledge base"
+Response approach:
+1. Ingest existing knowledge base documents
+2. Index in pgvector with embeddings
+3. Expose a retriever endpoint that returns top-5 results
+4. Return results with source metadata and relevance scores
+
+## Notes
+- Chunk size 800-1200 works well for most text; use 400-600 for code
+- Always include chunk_overlap (15-20% of chunk_size) to preserve context across boundaries
+- Use hybrid retrieval (semantic + BM25) for better recall than either alone
+- Re-ranking is critical for precision -- always add it for production systems
+- Test retrieval quality with a golden dataset of question-answer pairs before deploying
+- Monitor retrieval metrics: precision@k, recall@k, MRR

package/templates/bundle-data-pipeline/skills/rag-pipeline/references/chunking-strategies.md

@@ -0,0 +1,51 @@
+# Chunking Strategies
+
+## Recommended Defaults
+
+| Content Type | chunk_size | chunk_overlap | Separators |
+|---|---|---|---|
+| Prose / documentation | 1000 | 200 | `["\n## ", "\n### ", "\n\n", "\n", ". "]` |
+| Code files | 500 | 50 | `["\nclass ", "\ndef ", "\n\n", "\n"]` |
+| Legal / contracts | 1500 | 300 | `["\n\n", "\n", ". "]` |
+| Chat logs / Q&A | 500 | 100 | `["\n\n", "\n"]` |
+
+## RecursiveCharacterTextSplitter
+```python
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+# For markdown docs
+splitter = RecursiveCharacterTextSplitter(
+    chunk_size=1000,
+    chunk_overlap=200,
+    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "]
+)
+```
+
+## Code Splitting
+```python
+from langchain.text_splitter import Language, RecursiveCharacterTextSplitter
+
+python_splitter = RecursiveCharacterTextSplitter.from_language(
+    language=Language.PYTHON, chunk_size=500, chunk_overlap=50
+)
+
+ts_splitter = RecursiveCharacterTextSplitter.from_language(
+    language=Language.TS, chunk_size=500, chunk_overlap=50
+)
+```
+
+## Validation
+After chunking, verify:
+1. No chunks are empty or whitespace-only
+2. Average chunk size is close to target
+3. Metadata is preserved on all chunks
+4. Key information is not split across chunk boundaries (spot-check)
+
+```python
+sizes = [len(c.page_content) for c in chunks]
+print(f"Chunks: {len(chunks)}")
+print(f"Avg size: {sum(sizes)/len(sizes):.0f}")
+print(f"Min: {min(sizes)}, Max: {max(sizes)}")
+empty = [c for c in chunks if len(c.page_content.strip()) == 0]
+print(f"Empty chunks: {len(empty)}")
+```

package/templates/bundle-data-pipeline/skills/rag-pipeline/references/embedding-models.md

@@ -0,0 +1,49 @@
+# Embedding Models Comparison
+
+## OpenAI Models
+
+| Model | Dimensions | Max Tokens | Best For |
+|---|---|---|---|
+| text-embedding-3-large | 3072 (or custom) | 8191 | Highest quality, production |
+| text-embedding-3-small | 1536 | 8191 | Cost-effective, good quality |
+| text-embedding-ada-002 | 1536 | 8191 | Legacy, still widely used |
+
+## Usage with LangChain
+```python
+from langchain_openai import OpenAIEmbeddings
+
+# Default (text-embedding-3-large with reduced dimensions)
+embeddings = OpenAIEmbeddings(model="text-embedding-3-large", dimensions=1536)
+
+# Full dimensions for maximum quality
+embeddings = OpenAIEmbeddings(model="text-embedding-3-large", dimensions=3072)
+
+# Cost-effective option
+embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
+```
+
+## Open Source Alternatives
+```python
+from langchain_community.embeddings import HuggingFaceEmbeddings
+
+# All-MiniLM (fast, lightweight)
+embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
+
+# BGE (high quality, multilingual)
+embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-large-en-v1.5")
+```
+
+## Dimensionality and pgvector
+When creating the pgvector extension and table:
+```sql
+CREATE EXTENSION IF NOT EXISTS vector;
+
+-- Match dimensions to your embedding model
+CREATE TABLE embeddings (
+    id SERIAL PRIMARY KEY,
+    content TEXT,
+    embedding vector(1536)  -- adjust to match model dimensions
+);
+
+CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops);
+```

package/templates/bundle-frontend-spa/skills/authentication/SKILL.md

@@ -1,17 +1,43 @@
 ---
 name: authentication
-description: Implementar autenticação JWT com login, refresh token e proteção de rotas no frontend e backend. Use quando precisar implementar login, auth, JWT, ou proteção de endpoints.
+description: Implement JWT authentication with login, refresh tokens, route protection, and Axios interceptors for both frontend and backend. Use when you need to add login, JWT auth, protected routes, or token refresh to an application.
+version: 1.0.0
+author: Maestro
 ---
 
 # Authentication
 
-## Backend — JWT com FastAPI
+Implement complete JWT authentication flow covering backend token generation, frontend auth state, Axios interceptors, and protected routes.
 
+## When to Use
+- User needs to implement login/logout functionality
+- User wants to protect routes requiring authentication
+- User needs JWT token generation and validation (backend)
+- User wants to add automatic token refresh
+- User needs to set up Axios interceptors for auth headers
+
+## Available Operations
+1. Create JWT token generation and validation (FastAPI backend)
+2. Build login/logout flow with Zustand auth store
+3. Set up Axios interceptors for automatic Bearer token headers
+4. Implement protected route wrappers
+5. Add refresh token rotation
+
+## Multi-Step Workflow
+
+### Step 1: Install Backend Dependencies
+```bash
+pip install fastapi python-jose[cryptography] passlib[bcrypt] python-multipart
+```
+
+### Step 2: Create JWT Token Utilities (Backend)
 ```python
+# src/auth/jwt.py
+import os
+from datetime import datetime, timedelta
+from jose import jwt, JWTError
 from fastapi import Depends, HTTPException, status
 from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
-from jose import jwt, JWTError
-from datetime import datetime, timedelta
 
 security = HTTPBearer()
 SECRET_KEY = os.environ["JWT_SECRET"]
@@ -23,7 +49,7 @@ def create_access_token(user_id: str) -> str:
     payload = {
         "sub": user_id,
         "exp": datetime.utcnow() + ACCESS_TOKEN_EXPIRE,
-        "type": "access"
+        "type": "access",
     }
     return jwt.encode(payload, SECRET_KEY, algorithm=ALGORITHM)
 
@@ -31,11 +57,13 @@ def create_refresh_token(user_id: str) -> str:
     payload = {
         "sub": user_id,
         "exp": datetime.utcnow() + REFRESH_TOKEN_EXPIRE,
-        "type": "refresh"
+        "type": "refresh",
     }
     return jwt.encode(payload, SECRET_KEY, algorithm=ALGORITHM)
 
-async def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(security)) -> User:
+async def get_current_user(
+    credentials: HTTPAuthorizationCredentials = Depends(security),
+) -> User:
     try:
         payload = jwt.decode(credentials.credentials, SECRET_KEY, algorithms=[ALGORITHM])
         if payload.get("type") != "access":
@@ -46,22 +74,68 @@ async def get_current_user(credentials: HTTPAuthorizationCredentials = Depends(s
         return user
     except JWTError:
         raise HTTPException(status_code=401, detail="Invalid token")
+```
+
+### Step 3: Create Auth Endpoints (Backend)
+```python
+# src/auth/routes.py
+from fastapi import APIRouter, HTTPException
+
+router = APIRouter(prefix="/auth", tags=["auth"])
+
+@router.post("/login")
+async def login(data: LoginRequest):
+    user = await user_repo.find_by_email(data.email)
+    if not user or not verify_password(data.password, user.hashed_password):
+        raise HTTPException(status_code=401, detail="Invalid credentials")
+    return {
+        "access_token": create_access_token(str(user.id)),
+        "refresh_token": create_refresh_token(str(user.id)),
+        "user": UserResponse.from_entity(user),
+    }
+
+@router.post("/refresh")
+async def refresh(data: RefreshRequest):
+    try:
+        payload = jwt.decode(data.refresh_token, SECRET_KEY, algorithms=[ALGORITHM])
+        if payload.get("type") != "refresh":
+            raise HTTPException(status_code=401, detail="Invalid token type")
+        return {"access_token": create_access_token(payload["sub"])}
+    except JWTError:
+        raise HTTPException(status_code=401, detail="Invalid refresh token")
 
-# Endpoint protegido
 @router.get("/me")
 async def get_me(user: User = Depends(get_current_user)):
     return UserResponse.from_entity(user)
 ```
 
-## Frontend — Auth Context
+Test the backend:
+```bash
+# Start the API server
+uvicorn src.main:app --reload --port 8000
 
+# Test login
+curl -X POST http://localhost:8000/auth/login -H "Content-Type: application/json" -d '{"email": "admin@example.com", "password": "admin123"}'
+```
+
+### Step 4: Install Frontend Dependencies
+```bash
+npm install axios zustand
+```
+
+### Step 5: Create Auth Store (Frontend)
 ```tsx
+// src/stores/useAuthStore.ts
+import { create } from 'zustand';
+import { authApi } from '@/services/authApi';
+
 interface AuthState {
   token: string | null;
   user: User | null;
+  isAuthenticated: boolean;
   login: (email: string, password: string) => Promise<void>;
   logout: () => void;
-  isAuthenticated: boolean;
+  setToken: (token: string) => void;
 }
 
 export const useAuthStore = create<AuthState>((set) => ({
@@ -70,21 +144,130 @@ export const useAuthStore = create<AuthState>((set) => ({
   isAuthenticated: !!localStorage.getItem('token'),
 
   login: async (email, password) => {
-    const { access_token, user } = await authApi.login(email, password);
+    const { access_token, refresh_token, user } = await authApi.login(email, password);
     localStorage.setItem('token', access_token);
+    localStorage.setItem('refresh_token', refresh_token);
     set({ token: access_token, user, isAuthenticated: true });
   },
 
   logout: () => {
     localStorage.removeItem('token');
+    localStorage.removeItem('refresh_token');
     set({ token: null, user: null, isAuthenticated: false });
   },
+
+  setToken: (token) => {
+    localStorage.setItem('token', token);
+    set({ token });
+  },
 }));
+```
+
+### Step 6: Set Up Axios Interceptors
+```tsx
+// src/lib/api.ts
+import axios from 'axios';
+import { useAuthStore } from '@/stores/useAuthStore';
 
-// Axios interceptor
+export const api = axios.create({
+  baseURL: import.meta.env.VITE_API_URL || 'http://localhost:8000/api/v1',
+  timeout: 10000,
+});
+
+// Add auth header to every request
 api.interceptors.request.use((config) => {
   const token = localStorage.getItem('token');
-  if (token) config.headers.Authorization = `Bearer ${token}`;
+  if (token) {
+    config.headers.Authorization = `Bearer ${token}`;
+  }
   return config;
 });
+
+// Handle 401 responses: try refresh, then logout
+api.interceptors.response.use(
+  (response) => response.data,
+  async (error) => {
+    if (error.response?.status === 401) {
+      const refreshToken = localStorage.getItem('refresh_token');
+      if (refreshToken && !error.config._retry) {
+        error.config._retry = true;
+        try {
+          const { access_token } = await authApi.refresh(refreshToken);
+          useAuthStore.getState().setToken(access_token);
+          error.config.headers.Authorization = `Bearer ${access_token}`;
+          return api(error.config);
+        } catch {
+          useAuthStore.getState().logout();
+          window.location.href = '/login';
+        }
+      } else {
+        useAuthStore.getState().logout();
+        window.location.href = '/login';
+      }
+    }
+    return Promise.reject(error.response?.data || error);
+  }
+);
+```
+
+### Step 7: Create Protected Route Wrapper
+```tsx
+// src/components/ProtectedRoute.tsx
+import { Navigate } from 'react-router-dom';
+import { useAuthStore } from '@/stores/useAuthStore';
+
+export function ProtectedRoute({ children }: { children: React.ReactNode }) {
+  const isAuthenticated = useAuthStore((s) => s.isAuthenticated);
+
+  if (!isAuthenticated) {
+    return <Navigate to="/login" replace />;
+  }
+
+  return <>{children}</>;
+}
+
+// Usage in router:
+// <Route path="/dashboard" element={<ProtectedRoute><Dashboard /></ProtectedRoute>} />
 ```
+
+### Step 8: Verify the Auth Flow
+```bash
+npm run dev
+# 1. Navigate to /login
+# 2. Enter credentials and submit
+# 3. Verify redirect to /dashboard
+# 4. Verify protected routes redirect to /login when not authenticated
+# 5. Verify token is in localStorage
+# 6. Verify API calls include Authorization header (check Network tab)
+```
+
+## Resources
+- `references/jwt-security.md` - JWT security best practices and common vulnerabilities
+
+## Examples
+### Example 1: Add Login to an Existing App
+User asks: "Add authentication to our React app with a login page"
+Response approach:
+1. Create backend auth endpoints (login, refresh, me)
+2. Create useAuthStore with Zustand
+3. Set up Axios interceptors for token management
+4. Create LoginPage component with form
+5. Wrap routes in ProtectedRoute
+6. Test the full flow end-to-end
+
+### Example 2: Fix Token Expiry Issues
+User asks: "Users keep getting logged out, how do I add token refresh?"
+Response approach:
+1. Add refresh token endpoint to backend
+2. Store refresh token in localStorage alongside access token
+3. Add 401 interceptor that tries refresh before logging out
+4. Set retry flag to prevent infinite refresh loops
+5. Test by setting short access token expiry and verifying refresh works
+
+## Notes
+- Never store tokens in cookies without httpOnly + secure + sameSite flags
+- Set a reasonable access token expiry (1 hour) and longer refresh token (7 days)
+- Always validate token type ("access" vs "refresh") server-side
+- Use environment variables for JWT_SECRET, never hardcode
+- Implement rate limiting on login endpoints to prevent brute force
+- Clear all stored tokens on logout

package/templates/bundle-frontend-spa/skills/authentication/references/jwt-security.md

@@ -0,0 +1,41 @@
+# JWT Security Best Practices
+
+## Token Storage
+| Method | Security | XSS Risk | CSRF Risk |
+|---|---|---|---|
+| localStorage | Low | High | None |
+| httpOnly Cookie | High | None | Medium |
+| In-memory (variable) | Highest | None | None |
+
+Recommendation: Use httpOnly cookies for production, localStorage for development/SPAs.
+
+## Token Expiry
+| Token Type | Recommended Expiry |
+|---|---|
+| Access Token | 15 min - 1 hour |
+| Refresh Token | 7 - 30 days |
+
+## Security Checklist
+- [ ] Use strong secret key (256+ bits): `openssl rand -hex 32`
+- [ ] Set algorithm explicitly (HS256 or RS256), never "none"
+- [ ] Validate token type (access vs refresh) on every endpoint
+- [ ] Implement refresh token rotation (new refresh token on each use)
+- [ ] Rate limit login endpoint (5 attempts per minute)
+- [ ] Log failed login attempts
+- [ ] Clear all tokens on logout (both client and server)
+- [ ] Use HTTPS in production
+
+## Common Vulnerabilities
+1. **Algorithm confusion**: Always specify algorithm in `jwt.decode()`, never accept from token
+2. **Missing expiry check**: Always include `exp` claim
+3. **Token reuse after logout**: Maintain a blocklist or use short-lived tokens
+4. **Refresh without rotation**: Rotate refresh tokens to limit damage from stolen tokens
+
+## Generating a Secure Secret
+```bash
+# Generate a 256-bit hex secret
+openssl rand -hex 32
+
+# Or with Python
+python -c "import secrets; print(secrets.token_hex(32))"
+```