opencode-skills-collection 3.1.9 → 3.1.11

package/bundled-skills/weaviate-cookbooks/references/basic_rag.md

@@ -0,0 +1,219 @@
+# Basic RAG Cookbook
+
+Build basic RAG functionality with Weaviate.
+For advanced strategies, [see here](./advanced_rag.md).
+
+
+Docs to reference if needed:
+- Search patterns and basics in Weaviate: https://docs.weaviate.io/weaviate/search/basics
+- Filters in Weaviate: https://docs.weaviate.io/weaviate/search/filters
+- Vector search: https://docs.weaviate.io/weaviate/search/similarity
+- Keyword search: https://docs.weaviate.io/weaviate/search/bm25
+- Hybrid search: https://docs.weaviate.io/weaviate/search/hybrid
+- Image search: https://docs.weaviate.io/weaviate/search/image
+
+## Core Rules
+
+- Use a virtual environment via `venv`
+- Use `uv` for Python project/dependency management.
+- Do not manually author `pyproject.toml` or `uv.lock`; let `uv` generate/update them.
+- Use this install set: `uv add weaviate-client python-dotenv dspy`
+- Customise this cookbook to the users specification, ask them for details if not given. 
+
+Assume the user has data already to be used, do not create data unless asked to.
+
+## Env Rules
+
+Mandatory:
+- `WEAVIATE_URL`
+- `WEAVIATE_API_KEY`
+
+External provider keys:
+- Fill only keys actually used by the target Weaviate collection setup.
+
+
+## Weaviate Client
+
+```python
+import os
+from weaviate import connect_to_weaviate_cloud
+
+client = connect_to_weaviate_cloud(
+    cluster_url=os.getenv("WEAVIATE_URL", ""),
+    auth_credentials=os.getenv("WEAVIATE_API_KEY", ""),
+    headers={
+        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
+    },
+)
+```
+
+If the user's collections require vectorizer provider keys, set the matching keys listed in `environment_requirements.md`.
+
+Clients must be closed after completion. Wrap in `try/finally` blocks with `client.close()` (and `client.connect()` to reconnect if needed).
+
+
+## Multi-tenancy
+
+Multi-tenancy should be checked via 
+
+```python
+config = await collection.config.get()
+config.multi_tenancy_config.enabled # bool
+```
+
+e.g.
+
+```python
+
+base_collection = client.collections.use(collection_name)
+
+config = collection.config.get()
+if config.multi_tenancy_config.enabled:
+    collection = base_collection.with_tenant("<tenant_name>")
+else:
+    collection = base_collection
+```
+
+Tenant names can be obtained via
+```python
+all_tenants = list(collection.tenants.get().keys())
+```
+
+## Basic Retrieval
+
+Use collections via
+
+```python
+collection = client.collections.use("<collection_name>")
+```
+
+Weaviate can use vector, keyword or hybrid search.
+
+```python
+collection.query.near_text # semantic (text)
+collection.query.bm25 # keyword
+collection.query.hybrid # blend of keyword and semantic
+```
+
+It can also do image search
+
+```python
+collection.query.near_image(
+    near_image = ... # base 64 representation of image or Path object to image
+)
+```
+
+## Key Code Blocks
+
+RAG should have 4 pieces of core functionality:
+
+1. Pre-retrieval 
+2. Retrieval
+3. Post-retrieval
+4. Generation
+
+These should all be separate functions and combined into a single function, leaving scope for later editing or for the user themselves to modify it, to keep it understandable.
+
+## Pre-retrieval
+
+Transform the user question into a vector-database style (list of) query(ies). Basic RAG will provide no extra query transformations.
+
+```python
+def query_transformation(query: str) -> list[str]:
+    return [query]
+```
+
+## Retrieval
+
+```python
+def retrieve(
+    query: str, 
+    limit: int = 10,  # optional
+    filters = [] # optional
+    # additional arguments if required can go here and passed down to the search strategy
+) -> list[dict]:
+    
+    # import client logic here
+
+    collection = client.collections.use("<collection_name>")
+
+    response = collection.query.near_text( # or hybrid, bm25, near_image
+        query=query,
+        limit=limit,
+        filters=filters if filters else None
+    )
+    
+    return [
+        {
+            **obj.properties,
+            "uuid": obj.uuid
+        } 
+        for obj in response.objects
+    ]
+```
+
+## Post-Retrieval
+
+Modify the output of `retrieve`. Basic RAG will provide no extra post-processing. But you can consider adding uniqueness checks, formatting to remove properties, or more.
+
+```python
+def process_retrieval_results(objects: list[dict]) -> list[dict]:
+    return objects
+```
+
+
+## Generation
+
+This step depends on your LLM framework, [see below](#user-specific-customisations). Using DSPy:
+
+```python
+import dspy
+def generate(query: str, context: list[dict]) -> str:
+    lm = dspy.LM("<model_name>") # e.g. gpt-5.2, gpt-5-mini, claude-sonnet-4-5, etc.
+    answer = dspy.Predict("context, query -> answer") # inputs: context, query. outputs: answer
+    pred = answer(context=context, query=query, lm=lm)
+    return pred.answer # answer is then an attribute of pred
+```
+
+## User-specific Customisations
+
+If not specified ask the user about these points before implementing their respective strategies:
+
+**LLM Framework**
+
+You can use DSPy (works with all LiteLLM providers) or LiteLLM itself.
+
+- DSPy: https://dspy.ai/learn/programming/language_models/
+- LiteLLM: https://docs.litellm.ai/docs/
+
+Alternatively, users can use a single model provider. What model provider will they use?
+
+- OpenAI (https://platform.openai.com/docs/libraries) 
+- Anthropic (https://platform.claude.com/docs/) 
+- Google GenAI (https://ai.google.dev/gemini-api/docs/libraries)
+- Other (such as locally hosted models), use best judgement
+
+These may require additional installs.
+
+**Collections**
+
+Do collections already exist and what are they called? Does the user want to query multiple collections or just a single one? Does it need to be customisable?
+
+What format is the data, images or text or something else? What vectoriser is the collection set up as? What API keys are needed?
+
+**Search strategy**
+
+Does the user want semantic, keyword or hybrid search?
+
+Hybrid search has an `alpha` parameter, controlling tradeoff between keyword and semantic weights. `alpha=1` is pure semantic, `alpha=0` is pure keyword. 
+
+
+## Troubleshooting
+
+- Weaviate startup host errors: ensure `WEAVIATE_URL` is full `https://...` URL.
+- For any other issues, refer to the official library/package documentation and use web search extensively for troubleshooting.
+
+## Done Criteria
+
+- Create test scripts to check each function works independently with test data. Tear down tests after completion, or create a proper test suite with pytest (requires install)
+- User has completed specification of the app.

package/bundled-skills/weaviate-cookbooks/references/data_explorer.md

@@ -0,0 +1,336 @@
+# Build Data Explorer App
+
+## Overview
+
+Build a full-stack Data Explorer App for Weaviate Collections with FastAPI.
+
+Read first:
+- Search patterns and basics in Weaviate: https://docs.weaviate.io/weaviate/search/basics
+- Filters in Weaviate: https://docs.weaviate.io/weaviate/search/filters
+
+## Instructions
+
+### Core Rules
+
+- Use a virtual environment via `venv`
+- Use `uv` for Python project/dependency management.
+- Do not manually author `pyproject.toml` or `uv.lock`; let `uv` generate/update them.
+- Use this backend install set:
+  - `uv add fastapi 'uvicorn[standard]' weaviate-client pydantic-settings python-dotenv`
+- Depending on user request: consider combining this app with the [Query Agent Chatbot](./query_agent_chatbot.md).
+  - If the user explicitly only wants a data viewer/explorer, create this app independently
+  - If the user wants a fully featured chat and data explorer, combine the apps
+  - If no explicit instructions are given, ask the user their preference before continuing
+  - See the [Next Steps](#next-steps) section for more details
+
+### Fast Setup Commands
+
+Project bootstrap:
+
+```bash
+uv init data_explorer
+cd data_explorer
+uv venv
+uv add fastapi 'uvicorn[standard]' weaviate-client pydantic-settings python-dotenv
+```
+
+### Workflow Contract
+
+1. Build backend and frontend in one pass.
+2. Create `.env` from the canonical template in `environment_requirements.md`, then add app-specific fields (for example, `CORS_ORIGINS`).
+3. Before asking user to fill env, do non-secret local sanity checks that do not require real credentials (imports/compile/startup-shape checks).
+4. Ask user to fill real env values:
+   - Mandatory: `WEAVIATE_URL`, `WEAVIATE_API_KEY`
+   - Optional: only provider keys required by their collection setup
+5. After the user confirms, verify backend starts without errors and provide exact commands to run in the terminal.
+
+Do not ask avoidable questions that you can resolve from context.
+
+### Directory Structure
+
+Use a modular layout like:
+
+```text
+data_explorer/
+  backend/
+    app/
+      main.py
+      config.py
+      lifespan.py
+      dependencies.py
+      routers/
+      services/
+      models/
+    .env  # local file, never committed
+```
+
+Keep these boundaries:
+
+- routers: HTTP only
+- services: business/query-agent logic
+- models: request/response schemas
+- config/lifespan: wiring and startup/shutdown
+
+### Backend Requirements
+
+- FastAPI async app with lifespan.
+- Async Weaviate client initialized in lifespan and closed on shutdown.
+- Ensure no async blocking operations.
+- Not a full CRUD implementation - this is only for viewing data in a Weaviate collection.
+- Endpoints for:
+  - `GET /health`
+  - `GET /env_check`: returns what API keys are missing (if any) for verification on app start
+  - `GET /collections`: return available collections
+  - `GET /data/{collection_name}?xx=xx&yy=yy`: return data with optional arguments (more later), and pagination
+- Pydantic settings should read from process environment; local `.env` loading is optional for local development.
+- Conversation history mapping to Weaviate chat message format.
+
+### Env Rules
+
+Mandatory:
+- `WEAVIATE_URL`
+- `WEAVIATE_API_KEY`
+
+External provider keys:
+- Include every provider key needed by the target collections.
+- Leave unused provider keys empty/commented.
+
+CORS:
+
+- Default `CORS_ORIGINS` should include:
+  - `http://localhost:3000`
+  - `http://127.0.0.1:3000`
+  - `http://localhost:5173`
+  - `http://127.0.0.1:5173`
+
+### FastAPI standards
+
+1. Do not use hardcoded status values, use `status` from FastAPI, for example:
+
+```python
+from fastapi import status
+status.HTTP_200_OK # code 200
+status.HTTP_404_NOT_FOUND # code 404
+# and more
+```
+
+2. Use a Pydantic `BaseModel` for the `request` and `response_model` in all endpoints that require it. Ensure schema validation to mitigate user-error on the API.
+
+3. Use path parameters and query parameters for GET endpoints instead of payloads, for example:
+
+```python
+@app.get("/items/{item_id}")
+async def read_item(item_id: str):
+    return {"item_id": item_id}
+```
+
+```python
+@app.get("/items/")
+async def read_item(skip: int = 0, limit: int = 10):
+    return fake_items_db[skip : skip + limit]
+```
+
+4. Implement best practices for error-handling, do early returns and provide the correct status codes when necessary.
+
+5. Use proper logging for API usage, not simple print statements.
+
+### FastAPI endpoints
+
+Basic structure of endpoints. Customise according to user preference or suitability. Do not follow exactly, this is a guideline only.
+
+Ensure you also set up standard FastAPI procedures, such as global error handling, logging, dependencies. Set up an async client manager that connects on startup (via lifespan) and closes gracefully on app exit, use a dependency injection to add the client to the relevant endpoints.
+
+#### GET /health
+
+This is a standard health check. For example:
+
+```python
+from pydantic import BaseModel
+
+class HealthResponse(BaseModel):
+    status: str
+
+@app.get("/health", tags=["health"], response_model=HealthResponse)
+async def health_check() -> HealthResponse:
+    logger.info("Health check requested")
+    return HealthResponse(status="healthy")
+```
+
+#### GET /env_check
+
+Check what environment variables the backend has access to, used to verify the user's Weaviate configuration is correct. For example:
+
+```python
+import os
+from pydantic import BaseModel
+
+class EnvCheckResponse(BaseModel):
+    weaviate_url: bool
+    weaviate_api_key: bool
+
+@app.get("/env_check", tags=["health"])
+async def env_check() -> EnvCheckResponse:
+    logger.info("Environment check requested")
+    return EnvCheckResponse(
+        weaviate_url = os.getenv("WEAVIATE_URL") is not None,
+        weaviate_api_key = os.getenv("WEAVIATE_API_KEY") is not None,
+    )
+```
+
+### GET /collections
+
+Check what collections are available. For example:
+
+```python
+from pydantic import BaseModel
+from weaviate.client import WeaviateAsyncClient
+
+class CollectionsResponse(BaseModel):
+    collections: list[str]
+
+@app.get("/collections", tags=["collections"])
+async def collections() -> CollectionsResponse:
+
+    # include client management to import async client here
+
+    logger.info("Collections requested")
+    collections = await client.collections.list_all()
+    return CollectionsResponse(
+        collections = list(collections.keys())
+    )
+```
+
+Tip: consider expanding this endpoint to include collection descriptions and configs. `await client.collections.list_all()` returns `dict[str, _CollectionConfigSimple]` where `_CollectionConfigSimple` contains attributes:
+
+- `description`: `str`
+- `properties`: `list[Property]` where `Property` has `.name`, `.description` and `.data_type` (accessed via `.data_type[:]` to get name of data type as string)
+- `vector_config`: `dict[str, _NamedVectorConfig]` where `_NamedVectorConfig` has attribute `.vectorizer.vectorizer` (not a typo) which can be accessed via `.vectorizer.vectorizer[:]` to get the name of the vectoriser as a string.
+
+Multi-tenancy should be checked via
+
+```python
+config = await collection.config.get()
+config.multi_tenancy_config.enabled # bool
+```
+
+This is not available in the `_CollectionConfigSimple`, it must be fetched from `collection.config.get()`.
+
+#### GET /data/{collection_name}
+
+Retrieve data from a collection, using pagination, sorting and filters.
+
+```python
+from weaviate.collections import CollectionAsync
+from fastapi import Query
+from pydantic import BaseModel
+from typing import Any
+
+async def get_collection_data_types(collection: CollectionAsync) -> dict[str, str]:
+    config = await collection.config.get()
+    properties = config.properties
+    return {prop.name: prop.data_type[:] for prop in properties}
+
+class GetDataResponse(BaseModel):
+    data_types: dict[str, str]
+    items: list[dict[str, Any]]
+
+@router.post("/data/{collection_name}")
+async def get_data(
+    collection_name: str,
+    page_size: int = Query(default=10, ge=1, le=100),
+    page_number: int = Query(default=1, ge=1),
+    query: str = Query(default=""),
+    sort_on: str = Query(default=None),
+    ascending: bool = Query(default=True),
+) -> GetDataResponse:
+
+    # include client management to import async client here
+
+    collection = await client.collections.use(collection_name)
+    data_types = await async_get_collection_data_types(collection)
+
+    if query != "":
+        response = await collection.query.bm25(
+            query=query,
+            limit=page_size,
+            offset=page_size * (page_number - 1),
+        )
+    elif sort_on is not None:
+        response = await collection.query.fetch_objects(
+            sort=Sort.by_property(name=sort_on, ascending=ascending),
+            limit=page_size,
+            offset=page_size * (page_number - 1),
+        )
+    else:
+        response = await collection.query.fetch_objects(
+            limit=page_size,
+            offset=page_size * (page_number - 1),
+        )
+
+    return GetDataResponse(data_types = data_types, items = [obj.properties for obj in response.objects])
+```
+
+Tip: some collections can have multi-tenancy.
+Consider adding the tenant as an optional query parameter to `get_data`, e.g.
+
+```python
+async def get_data(
+    ... # existing args
+    tenant: str | None = Query(default=None)
+):
+    base_collection = await client.collections.use(collection_name)
+    data_types = await async_get_collection_data_types(collection)
+
+    config = await collection.config.get()
+    if config.multi_tenancy_config.enabled and tenant and tenant.strip():
+        collection = base_collection.with_tenant(tenant)
+    else:
+        collection = base_collection
+
+    # ...existing code
+```
+
+### Post-Env Hand-Holding (Required)
+
+After user says required env values are set, provide the terminal commands to run the backend:
+
+```bash
+cd data_explorer/backend
+uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
+```
+
+Then:
+
+- Ask user to start terminal.
+- Run smoke tests yourself against running services.
+- Report pass/fail in plain language and fix blockers.
+
+Do not offload detailed testing steps to the user unless they explicitly ask.
+
+## Troubleshooting
+
+- Weaviate startup host errors: ensure `WEAVIATE_URL` is full `https://...` URL.
+- For any other issues, refer to the official library/package documentation using web search.
+
+## Done Criteria
+
+- Backend healthy.
+- All endpoints work.
+- User can run server in terminal with provided commands.
+
+## Next Steps
+
+This application is currently a data explorer backend. You may optionally offer to integrate it with the [Query Agent Chatbot](./query_agent_chatbot.md) based on user preference.
+
+If the user chooses to combine these two applications, implement the integration as follows:
+
+- Create or use a directory `/routes` which separate functions for query agent chat and data exploration. Import the routers in the `main.py` file
+- If a frontend is requested, the frontend should have multiple pages/tabs depending on design choices so that data exploration and chat is separated
+- Consider crossovers between functionalities, e.g. a chat button from the data viewer/collection viewer which takes the user to chat with that collection selected.
+- Run quick tests to ensure the integration is seamless and the user can use both the chatbot and data explorer without any issues.
+
+### Frontend
+
+When the user explicitly asks for a frontend, use this reference as guideline:
+
+- [Frontend Interface](frontend_interface.md): Build a Next.js frontend to interact with the Weaviate backend.

package/bundled-skills/weaviate-cookbooks/references/environment_requirements.md

@@ -0,0 +1,78 @@
+# Environment Requirements for Weaviate
+
+Use this reference when building apps that connect to Weaviate and require external inference provider keys.
+
+## Required Weaviate Auth
+
+- `WEAVIATE_URL`
+- `WEAVIATE_API_KEY`
+
+## External Provider Env Vars and Headers
+
+| Provider | Environment Variable(s) | Header(s) sent to Weaviate |
+|----------|--------------------------|-----------------------------|
+| Anthropic | `ANTHROPIC_API_KEY` | `X-Anthropic-Api-Key` |
+| Anyscale | `ANYSCALE_API_KEY` | `X-Anyscale-Api-Key` |
+| AWS | `AWS_ACCESS_KEY`, `AWS_SECRET_KEY` | `X-Aws-Access-Key`, `X-Aws-Secret-Key` |
+| Cohere | `COHERE_API_KEY` | `X-Cohere-Api-Key` |
+| Databricks | `DATABRICKS_TOKEN` | `X-Databricks-Token` |
+| Friendli | `FRIENDLI_TOKEN` | `X-Friendli-Api-Key` |
+| Google Vertex AI | `VERTEX_API_KEY` | `X-Goog-Vertex-Api-Key` |
+| Google AI Studio | `STUDIO_API_KEY` | `X-Goog-Studio-Api-Key` |
+| HuggingFace | `HUGGINGFACE_API_KEY` | `X-HuggingFace-Api-Key` |
+| Jina AI | `JINAAI_API_KEY` | `X-JinaAI-Api-Key` |
+| Mistral | `MISTRAL_API_KEY` | `X-Mistral-Api-Key` |
+| NVIDIA | `NVIDIA_API_KEY` | `X-Nvidia-Api-Key` |
+| OpenAI | `OPENAI_API_KEY` | `X-OpenAI-Api-Key` |
+| Azure OpenAI | `AZURE_API_KEY` | `X-Azure-Api-Key` |
+| Voyage AI | `VOYAGE_API_KEY` | `X-Voyage-Api-Key` |
+| xAI | `XAI_API_KEY` | `X-Xai-Api-Key` |
+
+## Usage Notes
+
+- Set only the provider keys your collection configuration actually uses.
+- If multiple providers are configured, include all corresponding headers.
+
+## Canonical `.env` Template
+
+Use this template in all cookbook apps. Then ask the user to fill only the values their app actually needs.
+
+`WEAVIATE_URL` and `WEAVIATE_API_KEY` are mandatory for Weaviate-connected apps.
+
+```dotenv
+# Required for Weaviate cookbook apps (must be filled by user)
+WEAVIATE_URL=
+WEAVIATE_API_KEY=
+
+# Common app-level settings (uncomment when needed by the selected cookbook)
+# COLLECTIONS=
+# CORS_ORIGINS=http://localhost:3000,http://127.0.0.1:3000,http://localhost:5173,http://127.0.0.1:5173
+
+# External provider keys (uncomment only what the target collection uses)
+# ANTHROPIC_API_KEY=
+# ANYSCALE_API_KEY=
+# AWS_ACCESS_KEY=
+# AWS_SECRET_KEY=
+# AZURE_API_KEY=
+# COHERE_API_KEY=
+# DATABRICKS_TOKEN=
+# FRIENDLI_TOKEN=
+# HUGGINGFACE_API_KEY=
+# JINAAI_API_KEY=
+# MISTRAL_API_KEY=
+# NVIDIA_API_KEY=
+# OPENAI_API_KEY=
+# STUDIO_API_KEY=
+# VERTEX_API_KEY=
+# VOYAGE_API_KEY=
+# XAI_API_KEY=
+```
+
+## User Fill Guidance (Required)
+
+1. Create a local `.env` file from this template.
+2. Always ask the user to fill:
+   - `WEAVIATE_URL`
+   - `WEAVIATE_API_KEY`
+3. Ask them to uncomment and fill only the provider keys their Weaviate collections require.
+4. Keep `.env` local only and gitignored.