@synsci/cli-darwin-x64 1.1.49

package/bin/skills/llamaindex/references/data_connectors.md

@@ -0,0 +1,108 @@
+# LlamaIndex Data Connectors Guide
+
+300+ data connectors via LlamaHub.
+
+## Built-in loaders
+
+### SimpleDirectoryReader
+
+```python
+from llama_index.core import SimpleDirectoryReader
+
+# Load all files
+documents = SimpleDirectoryReader("./data").load_data()
+
+# Filter by extension
+documents = SimpleDirectoryReader(
+    "./data",
+    required_exts=[".pdf", ".docx", ".txt"]
+).load_data()
+
+# Recursive
+documents = SimpleDirectoryReader("./data", recursive=True).load_data()
+```
+
+### Web pages
+
+```python
+from llama_index.readers.web import SimpleWebPageReader, BeautifulSoupWebReader
+
+# Simple loader
+reader = SimpleWebPageReader()
+documents = reader.load_data(["https://example.com"])
+
+# Advanced (BeautifulSoup)
+reader = BeautifulSoupWebReader()
+documents = reader.load_data(urls=[
+    "https://docs.python.org",
+    "https://numpy.org"
+])
+```
+
+### PDF
+
+```python
+from llama_index.readers.file import PDFReader
+
+reader = PDFReader()
+documents = reader.load_data("paper.pdf")
+```
+
+### GitHub
+
+```python
+from llama_index.readers.github import GithubRepositoryReader
+
+reader = GithubRepositoryReader(
+    owner="facebook",
+    repo="react",
+    filter_file_extensions=[".js", ".jsx"],
+    verbose=True
+)
+
+documents = reader.load_data(branch="main")
+```
+
+## LlamaHub connectors
+
+Visit https://llamahub.ai for 300+ connectors:
+- Notion, Google Docs, Confluence
+- Slack, Discord, Twitter
+- PostgreSQL, MongoDB, MySQL
+- S3, GCS, Azure Blob
+- Stripe, Shopify, Salesforce
+
+### Install from LlamaHub
+
+```bash
+pip install llama-index-readers-notion
+```
+
+```python
+from llama_index.readers.notion import NotionPageReader
+
+reader = NotionPageReader(integration_token="your-token")
+documents = reader.load_data(page_ids=["page-id"])
+```
+
+## Custom loader
+
+```python
+from llama_index.core.readers.base import BaseReader
+from llama_index.core import Document
+
+class CustomReader(BaseReader):
+    def load_data(self, file_path: str):
+        # Your custom loading logic
+        with open(file_path) as f:
+            text = f.read()
+        return [Document(text=text, metadata={"source": file_path})]
+
+reader = CustomReader()
+documents = reader.load_data("data.txt")
+```
+
+## Resources
+
+- **LlamaHub**: https://llamahub.ai
+- **Data Connectors Docs**: https://developers.llamaindex.ai/python/framework/modules/data_connectors/

package/bin/skills/llamaindex/references/query_engines.md

@@ -0,0 +1,406 @@
+# LlamaIndex Query Engines Guide
+
+Complete guide to query engines, modes, and customization.
+
+## What are query engines?
+
+Query engines power the retrieval and response generation in LlamaIndex:
+1. Retrieve relevant chunks from index
+2. Generate response using LLM + context
+3. Return answer (optionally with sources)
+
+## Basic query engine
+
+```python
+from llama_index.core import VectorStoreIndex
+
+index = VectorStoreIndex.from_documents(documents)
+
+# Default query engine
+query_engine = index.as_query_engine()
+response = query_engine.query("What is the main topic?")
+print(response)
+```
+
+## Response modes
+
+### 1. Compact (default) - Best for most cases
+
+```python
+query_engine = index.as_query_engine(
+    response_mode="compact"
+)
+
+# Combines chunks that fit in context window
+response = query_engine.query("Explain quantum computing")
+```
+
+### 2. Tree summarize - Hierarchical summarization
+
+```python
+query_engine = index.as_query_engine(
+    response_mode="tree_summarize"
+)
+
+# Builds summary tree from chunks
+# Best for: Summarization tasks, many retrieved chunks
+response = query_engine.query("Summarize all the key findings")
+```
+
+### 3. Simple summarize - Concatenate and summarize
+
+```python
+query_engine = index.as_query_engine(
+    response_mode="simple_summarize"
+)
+
+# Concatenates all chunks, then summarizes
+# Fast but may lose context if too many chunks
+```
+
+### 4. Refine - Iterative refinement
+
+```python
+query_engine = index.as_query_engine(
+    response_mode="refine"
+)
+
+# Refines answer iteratively across chunks
+# Most thorough, slowest
+# Best for: Complex questions requiring synthesis
+```
+
+### 5. No text - Return nodes only
+
+```python
+query_engine = index.as_query_engine(
+    response_mode="no_text"
+)
+
+# Returns retrieved nodes without LLM response
+# Useful for: Debugging retrieval, custom processing
+response = query_engine.query("machine learning")
+for node in response.source_nodes:
+    print(node.text)
+```
+
+## Configuration options
+
+### Similarity top-k
+
+```python
+# Return top 3 most similar chunks
+query_engine = index.as_query_engine(
+    similarity_top_k=3  # Default: 2
+)
+```
+
+### Streaming
+
+```python
+# Stream response tokens
+query_engine = index.as_query_engine(streaming=True)
+
+response = query_engine.query("Explain neural networks")
+for text in response.response_gen:
+    print(text, end="", flush=True)
+```
+
+### Verbose mode
+
+```python
+# Show retrieval and generation process
+query_engine = index.as_query_engine(verbose=True)
+
+response = query_engine.query("What is Python?")
+# Prints: Retrieved chunks, prompts, LLM calls
+```
+
+## Custom prompts
+
+### Text QA template
+
+```python
+from llama_index.core import PromptTemplate
+
+qa_prompt = PromptTemplate(
+    "Context information is below.\n"
+    "---------------------\n"
+    "{context_str}\n"
+    "---------------------\n"
+    "Given the context, answer: {query_str}\n"
+    "If the context doesn't contain the answer, say 'I don't know'.\n"
+    "Answer: "
+)
+
+query_engine = index.as_query_engine(text_qa_template=qa_prompt)
+```
+
+### Refine template
+
+```python
+refine_prompt = PromptTemplate(
+    "The original query is: {query_str}\n"
+    "We have an existing answer: {existing_answer}\n"
+    "We have new context: {context_msg}\n"
+    "Refine the answer based on new context. "
+    "If context isn't useful, return original answer.\n"
+    "Refined Answer: "
+)
+
+query_engine = index.as_query_engine(
+    response_mode="refine",
+    refine_template=refine_prompt
+)
+```
+
+## Node postprocessors
+
+### Metadata filtering
+
+```python
+from llama_index.core.postprocessor import MetadataReplacementPostProcessor
+
+postprocessor = MetadataReplacementPostProcessor(
+    target_metadata_key="window"  # Replace node content with window
+)
+
+query_engine = index.as_query_engine(
+    node_postprocessors=[postprocessor]
+)
+```
+
+### Similarity cutoff
+
+```python
+from llama_index.core.postprocessor import SimilarityPostprocessor
+
+# Filter nodes below similarity threshold
+postprocessor = SimilarityPostprocessor(similarity_cutoff=0.7)
+
+query_engine = index.as_query_engine(
+    node_postprocessors=[postprocessor]
+)
+```
+
+### Reranking
+
+```python
+from llama_index.core.postprocessor import SentenceTransformerRerank
+
+# Rerank retrieved nodes
+reranker = SentenceTransformerRerank(
+    model="cross-encoder/ms-marco-MiniLM-L-2-v2",
+    top_n=3
+)
+
+query_engine = index.as_query_engine(
+    node_postprocessors=[reranker],
+    similarity_top_k=10  # Retrieve 10, rerank to 3
+)
+```
+
+## Advanced query engines
+
+### Sub-question query engine
+
+```python
+from llama_index.core.query_engine import SubQuestionQueryEngine
+from llama_index.core.tools import QueryEngineTool
+
+# Multiple indices for different topics
+python_index = VectorStoreIndex.from_documents(python_docs)
+numpy_index = VectorStoreIndex.from_documents(numpy_docs)
+
+# Create tools
+python_tool = QueryEngineTool.from_defaults(
+    query_engine=python_index.as_query_engine(),
+    description="Useful for Python programming questions"
+)
+numpy_tool = QueryEngineTool.from_defaults(
+    query_engine=numpy_index.as_query_engine(),
+    description="Useful for NumPy array questions"
+)
+
+# Sub-question engine decomposes complex queries
+query_engine = SubQuestionQueryEngine.from_defaults(
+    query_engine_tools=[python_tool, numpy_tool]
+)
+
+# "How do I create numpy arrays in Python?" becomes:
+# 1. Query numpy_tool about array creation
+# 2. Query python_tool about syntax
+# 3. Synthesize answers
+response = query_engine.query("How do I create numpy arrays in Python?")
+```
+
+### Router query engine
+
+```python
+from llama_index.core.query_engine import RouterQueryEngine
+from llama_index.core.selectors import LLMSingleSelector
+
+# Route to appropriate index based on query
+selector = LLMSingleSelector.from_defaults()
+
+query_engine = RouterQueryEngine(
+    selector=selector,
+    query_engine_tools=[python_tool, numpy_tool]
+)
+
+# Automatically routes to correct index
+response = query_engine.query("What is Python?")  # Routes to python_tool
+response = query_engine.query("NumPy broadcasting?")  # Routes to numpy_tool
+```
+
+### Transform query engine
+
+```python
+from llama_index.core.query_engine import TransformQueryEngine
+from llama_index.core.query_transforms import HyDEQueryTransform
+
+# HyDE: Generate hypothetical document before retrieval
+hyde_transform = HyDEQueryTransform(include_original=True)
+
+query_engine = TransformQueryEngine(
+    query_engine=base_query_engine,
+    query_transform=hyde_transform
+)
+
+# Improves retrieval quality
+response = query_engine.query("What are the benefits of Python?")
+```
+
+## Chat engine (conversational)
+
+### Basic chat engine
+
+```python
+# Chat engine with memory
+chat_engine = index.as_chat_engine(
+    chat_mode="condense_plus_context"
+)
+
+# Multi-turn conversation
+response1 = chat_engine.chat("What is Python?")
+response2 = chat_engine.chat("What are its main features?")  # Remembers context
+response3 = chat_engine.chat("Can you give examples?")
+```
+
+### Chat modes
+
+```python
+# 1. condense_plus_context (recommended)
+chat_engine = index.as_chat_engine(chat_mode="condense_plus_context")
+# Condenses chat history + retrieves relevant context
+
+# 2. context - Simple RAG
+chat_engine = index.as_chat_engine(chat_mode="context")
+# Retrieves context for each query
+
+# 3. react - Agent-based
+chat_engine = index.as_chat_engine(chat_mode="react")
+# Uses ReAct agent pattern with tools
+
+# 4. best - Automatically selects best mode
+chat_engine = index.as_chat_engine(chat_mode="best")
+```
+
+### Reset conversation
+
+```python
+# Clear chat history
+chat_engine.reset()
+
+# Start new conversation
+response = chat_engine.chat("New topic: what is machine learning?")
+```
+
+## Structured output
+
+### Pydantic models
+
+```python
+from pydantic import BaseModel
+from llama_index.core.output_parsers import PydanticOutputParser
+
+class Summary(BaseModel):
+    title: str
+    main_points: list[str]
+    category: str
+
+output_parser = PydanticOutputParser(output_cls=Summary)
+
+query_engine = index.as_query_engine(
+    output_parser=output_parser
+)
+
+response = query_engine.query("Summarize the document")
+# response is a Pydantic model
+print(response.title, response.main_points)
+```
+
+## Source tracking
+
+### Get source nodes
+
+```python
+query_engine = index.as_query_engine()
+
+response = query_engine.query("What is Python?")
+
+# Access source nodes
+for node in response.source_nodes:
+    print(f"Text: {node.text}")
+    print(f"Score: {node.score}")
+    print(f"Metadata: {node.metadata}")
+```
+
+## Best practices
+
+1. **Use compact mode for most cases** - Good balance
+2. **Set similarity_top_k appropriately** - 2-5 usually optimal
+3. **Enable streaming for long responses** - Better UX
+4. **Add postprocessors for quality** - Reranking improves results
+5. **Use chat engine for conversations** - Built-in memory
+6. **Track source nodes** - Cite sources to users
+7. **Custom prompts for domain** - Better responses
+8. **Test different response modes** - Pick best for use case
+9. **Monitor token usage** - Retrieval + generation costs
+10. **Cache query engines** - Don't recreate each time
+
+## Performance tips
+
+### Caching
+
+```python
+from llama_index.core.storage.chat_store import SimpleChatStore
+
+# Cache chat history
+chat_store = SimpleChatStore()
+chat_engine = index.as_chat_engine(
+    chat_mode="condense_plus_context",
+    chat_store=chat_store
+)
+```
+
+### Async queries
+
+```python
+import asyncio
+
+# Async query for concurrent requests
+response = await query_engine.aquery("What is Python?")
+
+# Multiple concurrent queries
+responses = await asyncio.gather(
+    query_engine.aquery("What is Python?"),
+    query_engine.aquery("What is Java?")
+)
+```
+
+## Resources
+
+- **Query Engines Docs**: https://developers.llamaindex.ai/python/framework/modules/querying/
+- **Response Modes**: https://developers.llamaindex.ai/python/framework/modules/querying/response_modes/
+- **Chat Engines**: https://developers.llamaindex.ai/python/framework/modules/chat/