schift-cli 0.1.0__py3-none-any.whl

schift_cli/data/schift-best-practices/references/embed-task-types.md

@@ -0,0 +1,135 @@
+---
+title: Specify task_type for optimal embedding quality
+impact: HIGH
+impactDescription: Using the default task_type for every use case can silently degrade retrieval accuracy by 10-30%. Schift optimizes embeddings differently depending on whether the vector will be used for search, classification, clustering, or other tasks.
+tags: [embedding, task-type, retrieval, classification, quality]
+---
+
+## Specify task_type for optimal embedding quality
+
+Schift supports seven task types that control how the embedding model is fine-tuned for your specific use case. The default (`retrieval`) is a safe starting point, but explicitly setting the correct task type produces better embeddings and downstream results.
+
+| task_type | When to use |
+|---|---|
+| `retrieval` | Asymmetric search — query against document corpus |
+| `similarity` | Symmetric similarity — comparing two pieces of text |
+| `classification` | Labeling or categorizing text |
+| `clustering` | Grouping documents by topic |
+| `QA` | Question-answering — question against answer pairs |
+| `fact_verification` | Claim against evidence matching |
+| `code_retrieval` | Natural-language query against code snippets |
+
+**Incorrect** — using default `task_type` for every case reduces embedding quality:
+
+```python
+# Python - DON'T do this
+from schift import Schift
+
+client = Schift(api_key="sch_...")
+
+# Indexing documents for search — no task_type set
+doc_embedding = client.embed("The Eiffel Tower is in Paris.")
+
+# Classifying customer feedback — no task_type set
+label_embedding = client.embed("This product is terrible!")
+
+# Searching code — no task_type set
+code_query = client.embed("function that sorts a list")
+```
+
+```typescript
+// TypeScript - DON'T do this
+import { Schift } from '@schift-io/sdk';
+
+const client = new Schift({ apiKey: 'sch_...' });
+
+// All three use cases use the same default — suboptimal for each
+const docEmbedding  = await client.embed('The Eiffel Tower is in Paris.');
+const labelEmbedding = await client.embed('This product is terrible!');
+const codeQuery      = await client.embed('function that sorts a list');
+```
+
+**Correct** — match `task_type` to your use case:
+
+```python
+# Python - DO this
+from schift import Schift
+
+client = Schift(api_key="sch_...")
+
+# Asymmetric document search: index documents as "retrieval"
+doc_embedding = client.embed(
+    "The Eiffel Tower is in Paris.",
+    task_type="retrieval",
+)
+
+# Classifying customer feedback
+label_embedding = client.embed(
+    "This product is terrible!",
+    task_type="classification",
+)
+
+# Natural-language query against a code repository
+code_query = client.embed(
+    "function that sorts a list",
+    task_type="code_retrieval",
+)
+
+# QA: embed question against pre-embedded answers
+question_vec = client.embed(
+    "What year was the Eiffel Tower built?",
+    task_type="QA",
+)
+
+# Fact verification: claim against supporting evidence
+claim_vec = client.embed(
+    "The Eiffel Tower was built in 1889.",
+    task_type="fact_verification",
+)
+
+# Batch embed with task_type
+docs = ["Doc A content...", "Doc B content...", "Doc C content..."]
+results = client.embed_batch(docs, task_type="retrieval")
+```
+
+```typescript
+// TypeScript - DO this
+import { Schift } from '@schift-io/sdk';
+
+const client = new Schift({ apiKey: 'sch_...' });
+
+// Asymmetric document search
+const docEmbedding = await client.embed(
+  'The Eiffel Tower is in Paris.',
+  { taskType: 'retrieval' },
+);
+
+// Classification
+const labelEmbedding = await client.embed(
+  'This product is terrible!',
+  { taskType: 'classification' },
+);
+
+// Code search
+const codeQuery = await client.embed(
+  'function that sorts a list',
+  { taskType: 'code_retrieval' },
+);
+
+// QA
+const questionVec = await client.embed(
+  'What year was the Eiffel Tower built?',
+  { taskType: 'QA' },
+);
+
+// Batch with task type
+const docs = ['Doc A content...', 'Doc B content...', 'Doc C content...'];
+const results = await client.embedBatch(docs, { taskType: 'retrieval' });
+```
+
+Use the same `task_type` when indexing documents and when embedding queries — mixing task types in the same collection degrades search quality. For collections that serve multiple use cases, create separate collections per task type.
+
+## Reference
+
+- https://docs.schift.io/api/embed#task-type
+- https://docs.schift.io/concepts/task-types

schift_cli/data/schift-best-practices/references/rag-chunking.md

@@ -0,0 +1,173 @@
+---
+title: Let Schift Handle Chunking via Bucket Upload
+impact: HIGH
+impactDescription: Custom chunking loses document structure (headings, tables, lists), produces uneven chunk quality, and requires maintaining your own chunking logic. Schift's bucket upload handles OCR, structure-aware chunking, and embedding automatically.
+tags:
+  - rag
+  - chunking
+  - bucket
+  - document-structure
+  - preprocessing
+---
+
+## Let Schift Handle Chunking via Bucket Upload
+
+Chunking is one of the most impactful and underestimated decisions in a RAG pipeline. Naively splitting text by character count or regex destroys semantic units: headings get separated from their content, table rows become orphaned fragments, and list items lose their parent context. The result is chunks that embed poorly and retrieve incorrectly.
+
+Schift's bucket upload pipeline runs document parsing, OCR, and structure-aware chunking on your behalf. It understands heading hierarchies, tables, code blocks, and lists — producing semantically coherent chunks that embed and retrieve at higher quality without any custom code.
+
+### Incorrect
+
+Custom regex or fixed-size chunking that destroys document structure:
+
+```python
+# Python — naive fixed-size chunking, loses all structure
+import re
+from schift import Schift
+
+client = Schift(api_key="sch_...")
+
+with open("technical_report.pdf", "rb") as f:
+    raw_text = extract_text_from_pdf(f)  # flat string, all structure lost
+
+# Split every 500 characters regardless of sentence or section boundaries
+chunks = [raw_text[i:i+500] for i in range(0, len(raw_text), 500)]
+
+# Problems:
+# - Heading "## Performance Results" ends up in a different chunk than the table below it
+# - Table rows split mid-row: "| 98.2% | la" / "tency 12ms |"
+# - Code blocks broken at arbitrary positions
+# - No metadata: no page number, no section, no source
+
+for i, chunk in enumerate(chunks):
+    client.embed_and_insert(collection_id, chunk, metadata={"index": i})
+```
+
+```typescript
+// TypeScript — naive regex split, loses paragraph context
+import { Schift } from '@schift-io/sdk';
+
+const client = new Schift({ apiKey: 'sch_...' });
+
+const rawText = await extractText('technical_report.pdf');  // flat string
+
+// Split on double newlines — misses tables, code blocks, lists
+const chunks = rawText.split(/\n\n+/).filter(c => c.trim().length > 0);
+
+// No overlap, no metadata, no awareness of document hierarchy
+for (const chunk of chunks) {
+  await client.embedAndInsert(collectionId, chunk);
+}
+```
+
+This approach drops all structural signals that models rely on to understand context. A chunk reading `"98.2%  12ms  4.1GB"` is meaningless without the table header that provides column names.
+
+### Correct
+
+Upload directly to a Schift bucket and let the pipeline handle chunking:
+
+```python
+# Python — bucket upload: structure-aware chunking, no manual work
+from schift import Schift
+
+client = Schift(api_key="sch_...")
+
+# Upload the raw file — Schift handles OCR, parsing, chunking, embedding
+with open("technical_report.pdf", "rb") as f:
+    result = client.bucket.upload(
+        bucket_id=bucket_id,
+        file=f,
+        filename="technical_report.pdf",
+        metadata={
+            "source": "internal-research",
+            "version": "2026-Q1",
+            "department": "engineering",
+        }
+    )
+
+print(result.chunk_count)   # e.g. 142 semantically coherent chunks
+print(result.document_id)   # reference for deletion or re-indexing
+
+# Schift preserves:
+# - Heading hierarchy (H1 > H2 > H3 as metadata fields)
+# - Table structure (header row context attached to each data row chunk)
+# - Code blocks (kept intact, tagged as code)
+# - List items (grouped under their parent heading)
+```
+
+```typescript
+// TypeScript — bucket upload: let Schift do the heavy lifting
+import { Schift } from '@schift-io/sdk';
+import { readFileSync } from 'fs';
+
+const client = new Schift({ apiKey: 'sch_...' });
+
+const fileBuffer = readFileSync('technical_report.pdf');
+
+const result = await client.bucket.upload(bucketId, fileBuffer, {
+  filename: 'technical_report.pdf',
+  metadata: {
+    source: 'internal-research',
+    version: '2026-Q1',
+    department: 'engineering',
+  },
+});
+
+console.log(result.chunkCount);   // semantically coherent chunks
+console.log(result.documentId);   // for future reference or deletion
+```
+
+If you must chunk manually (e.g., streaming data with no file format), follow these rules:
+
+```python
+# Python — manual chunking fallback: overlap + boundaries + metadata
+def chunk_text_safely(text: str, source: str, page: int = 0) -> list[dict]:
+    paragraphs = text.split("\n\n")
+    chunks = []
+    window = []
+    window_size = 0
+    target_size = 400   # tokens (approximate)
+    overlap_size = 60   # 10–20% overlap to preserve cross-boundary context
+
+    for para in paragraphs:
+        words = para.split()
+        window.extend(words)
+        window_size += len(words)
+
+        if window_size >= target_size:
+            chunk_text = " ".join(window)
+            chunks.append({
+                "text": chunk_text,
+                "metadata": {
+                    "source": source,
+                    "page": page,
+                    "char_count": len(chunk_text),
+                }
+            })
+            # Retain overlap to preserve context across chunk boundary
+            window = window[-overlap_size:]
+            window_size = len(window)
+
+    if window:  # flush remainder
+        chunks.append({
+            "text": " ".join(window),
+            "metadata": {"source": source, "page": page}
+        })
+
+    return chunks
+```
+
+**Manual chunking checklist (if unavoidable):**
+
+| Rule | Why |
+|------|-----|
+| 10–20% overlap between chunks | Prevents context loss at boundaries |
+| Respect paragraph/sentence boundaries | Keeps semantic units intact |
+| Include source, page, section in metadata | Enables filtered search and attribution |
+| Target 300–500 tokens per chunk | Balances precision and recall |
+| Never split mid-sentence | Broken sentences embed poorly |
+
+## Reference
+
+- https://docs.schift.io/buckets/upload
+- https://docs.schift.io/buckets/chunking

schift_cli/data/schift-best-practices/references/rag-workflow-builder.md

@@ -0,0 +1,205 @@
+---
+title: Use WorkflowBuilder for Reproducible RAG Pipelines
+impact: HIGH
+impactDescription: Ad-hoc inline RAG code is hard to test, version, or share. WorkflowBuilder defines RAG as a versioned DAG — reproducible across environments, observable via run history, and shareable as templates.
+tags:
+  - rag
+  - workflow
+  - dag
+  - workflowbuilder
+  - pipeline
+  - observability
+---
+
+## Use WorkflowBuilder for Reproducible RAG Pipelines
+
+Embedding retrieval + reranking + LLM generation is a multi-step pipeline. When this logic lives as scattered inline code, it becomes impossible to test individual steps in isolation, reproduce a specific run for debugging, or share the pipeline as a reusable template across projects.
+
+Schift's workflow engine models RAG pipelines as directed acyclic graphs (DAGs). Each step is a typed node (retrieve, rerank, generate, filter, transform). The `WorkflowBuilder` API in TypeScript (and its Python equivalent) lets you define, version, and register these pipelines as first-class resources. You run them by ID, and every run is logged — input, intermediate results, output — for debugging and evaluation.
+
+### Incorrect
+
+Inline search + LLM call logic scattered across the application:
+
+```typescript
+// TypeScript — ad-hoc RAG scattered across application code
+import { Schift } from '@schift-io/sdk';
+import OpenAI from 'openai';
+
+const client = new Schift({ apiKey: 'sch_...' });
+const openai = new OpenAI();
+
+// This logic is duplicated in routes/chat.ts, routes/search.ts, scripts/batch.ts
+// No versioning, no run history, impossible to A/B test
+async function answerQuestion(question: string) {
+  // Step 1: retrieve — hardcoded params, no way to tune per environment
+  const results = await client.search(collectionId, question, { topK: 5 });
+
+  // Step 2: no reranking step — order determined by vector score alone
+  const context = results.map(r => r.text).join('\n\n');
+
+  // Step 3: generate — prompt template hardcoded here, not versioned
+  const completion = await openai.chat.completions.create({
+    model: 'gpt-4o',
+    messages: [
+      { role: 'system', content: 'Answer based on context.' },
+      { role: 'user', content: `Context:\n${context}\n\nQuestion: ${question}` },
+    ],
+  });
+
+  return completion.choices[0].message.content;
+  // No logging of which chunks were retrieved, no run ID, no eval surface
+}
+```
+
+```python
+# Python — same problem: logic copy-pasted across scripts, no observability
+from schift import Schift
+from openai import OpenAI
+
+schift = Schift(api_key="sch_...")
+openai = OpenAI()
+
+def answer(question: str) -> str:
+    results = schift.search(collection_id, question, top_k=5)
+    context = "\n\n".join(r.text for r in results)
+    resp = openai.chat.completions.create(
+        model="gpt-4o",
+        messages=[
+            {"role": "system", "content": "Answer based on context."},
+            {"role": "user", "content": f"Context:\n{context}\n\nQ: {question}"},
+        ]
+    )
+    return resp.choices[0].message.content
+    # No run ID, no intermediate state, cannot replay or debug
+```
+
+When this breaks in production, you cannot inspect which chunks were retrieved, which reranking score caused a bad answer, or compare two versions of the pipeline.
+
+### Correct
+
+Define the pipeline once as a versioned workflow, run it by ID:
+
+```typescript
+// TypeScript — WorkflowBuilder: define once, run anywhere, fully observable
+import { Schift, WorkflowBuilder } from '@schift-io/sdk';
+
+const client = new Schift({ apiKey: 'sch_...' });
+
+// Define the DAG — do this once during setup (e.g., in a migration script)
+const graph = new WorkflowBuilder()
+  .addNode({
+    id: 'retrieve',
+    type: 'retrieve',
+    config: {
+      collectionId: 'col_docs_prod',
+      topK: 10,
+      scoreThreshold: 0.70,
+      taskType: 'qa',
+    },
+  })
+  .addNode({
+    id: 'rerank',
+    type: 'rerank',
+    config: {
+      model: 'schift-reranker-v1',
+      topN: 3,       // keep only the top 3 after reranking
+    },
+  })
+  .addNode({
+    id: 'generate',
+    type: 'generate',
+    config: {
+      model: 'gpt-4o',
+      systemPrompt: 'Answer the question using only the provided context. If unsure, say so.',
+      maxTokens: 512,
+    },
+  })
+  .addEdge('retrieve', 'rerank')   // retrieve → rerank
+  .addEdge('rerank', 'generate')   // rerank → generate
+  .build();
+
+// Register the workflow — returns a versioned workflow ID
+const workflow = await client.workflow.create('docs-qa-v1', graph);
+console.log(workflow.id);  // wf_abc123 — store this in your config
+
+// Run the workflow — every run gets a run ID
+const run = await client.workflow.run(workflow.id, {
+  query: 'What are the latency benchmarks for the v2 model?',
+});
+
+console.log(run.runId);          // run_xyz789 — for debugging
+console.log(run.output.answer);  // final generated answer
+console.log(run.steps.retrieve.chunks);  // intermediate: which chunks retrieved
+console.log(run.steps.rerank.chunks);    // intermediate: which chunks survived reranking
+```
+
+```python
+# Python — same workflow pattern
+from schift import Schift
+
+client = Schift(api_key="sch_...")
+
+# Build the graph as a dict (Python SDK uses graph spec format)
+graph = {
+    "nodes": [
+        {
+            "id": "retrieve",
+            "type": "retrieve",
+            "config": {
+                "collection_id": "col_docs_prod",
+                "top_k": 10,
+                "score_threshold": 0.70,
+                "task_type": "qa",
+            },
+        },
+        {
+            "id": "rerank",
+            "type": "rerank",
+            "config": {"model": "schift-reranker-v1", "top_n": 3},
+        },
+        {
+            "id": "generate",
+            "type": "generate",
+            "config": {
+                "model": "gpt-4o",
+                "system_prompt": "Answer using only the provided context.",
+                "max_tokens": 512,
+            },
+        },
+    ],
+    "edges": [
+        {"from": "retrieve", "to": "rerank"},
+        {"from": "rerank", "to": "generate"},
+    ],
+}
+
+# Register once
+workflow = client.workflow.create("docs-qa-v1", graph)
+
+# Run — returns structured run result with full step history
+run = client.workflow.run(workflow.id, {"query": "What are the latency benchmarks?"})
+
+print(run.run_id)               # for debugging
+print(run.output["answer"])     # final answer
+print(run.steps["retrieve"])    # which chunks were retrieved
+print(run.steps["rerank"])      # which chunks survived reranking
+```
+
+**Why WorkflowBuilder over inline code:**
+
+| Concern | Inline Code | WorkflowBuilder |
+|---------|-------------|-----------------|
+| Versioning | Manual, error-prone | Built-in (v1, v2, ...) |
+| Observability | None | Full run history with step-level traces |
+| Testing | Full integration test required | Each node testable in isolation |
+| Sharing | Copy-paste | Export/import workflow templates |
+| A/B testing | Requires code branching | Run two workflow IDs, compare run metrics |
+| Prompt updates | Redeploy required | Update workflow node config, no code change |
+
+Store the workflow ID in your application config (environment variable or database). If you need to update the pipeline — change `topK`, swap the reranker, update the system prompt — create a new workflow version and update the config, with no application code changes.
+
+## Reference
+
+- https://docs.schift.io/workflows/builder
+- https://docs.schift.io/workflows/run-history

schift_cli/data/schift-best-practices/references/sdk-async-patterns.md

@@ -0,0 +1,103 @@
+---
+title: SDK Async Patterns
+impact: MEDIUM
+impactDescription: Calling the synchronous Schift client inside an async web framework blocks the event loop, serializing all requests and eliminating the concurrency benefit of async frameworks like FastAPI or Express.
+tags: [sdk, async, fastapi, performance, python, typescript]
+---
+
+## Use the Async Client in Async Applications
+
+The Schift Python SDK ships two client classes: `Schift` (synchronous, thread-blocking) and `AsyncSchift` (async/await, non-blocking). The TypeScript SDK is async-only and always returns Promises.
+
+Using the synchronous Python client inside an `async def` handler does not raise an error but blocks the entire event loop for the duration of the network call. Under load, this effectively single-threads your server.
+
+### Incorrect
+
+```python
+# Python - synchronous client blocks the event loop inside an async handler
+import os
+from fastapi import FastAPI
+from schift import Schift  # synchronous client
+
+app = FastAPI()
+client = Schift()  # uses SCHIFT_API_KEY env var
+
+@app.get("/search")
+async def search(q: str, bucket_id: str):
+    # client.bucket.search is a blocking I/O call
+    # This stalls the event loop while waiting for the HTTP response
+    results = client.bucket.search(bucket_id, q)
+    return results
+```
+
+```typescript
+// TypeScript - no sync client exists, but forgetting await is a common mistake
+import { Schift } from '@schift-io/sdk';
+
+const client = new Schift();
+
+app.get('/search', (req, res) => {
+  // Missing await - returns a Promise, not the actual results
+  const results = client.bucket.search(req.query.bucketId, req.query.q);
+  res.json(results);  // serializes the Promise object, not the data
+});
+```
+
+### Correct
+
+```python
+# Python - use AsyncSchift with async def handlers
+import os
+from fastapi import FastAPI
+from schift import AsyncSchift  # async client
+
+app = FastAPI()
+client = AsyncSchift()  # reads SCHIFT_API_KEY automatically
+
+@app.get("/search")
+async def search(q: str, bucket_id: str):
+    results = await client.bucket.search(bucket_id, q)
+    return results
+
+# Parallel calls: use asyncio.gather to fan out multiple searches concurrently
+import asyncio
+
+@app.get("/multi-search")
+async def multi_search(q: str, bucket_ids: list[str]):
+    tasks = [client.bucket.search(bid, q) for bid in bucket_ids]
+    results = await asyncio.gather(*tasks)
+    return results
+```
+
+```typescript
+// TypeScript - always await; use Promise.all for concurrent calls
+import { Schift } from '@schift-io/sdk';
+import express from 'express';
+
+const app = express();
+const client = new Schift(); // reads SCHIFT_API_KEY automatically
+
+app.get('/search', async (req, res) => {
+  const results = await client.bucket.search(
+    req.query.bucketId as string,
+    req.query.q as string
+  );
+  res.json(results);
+});
+
+// Parallel calls
+app.get('/multi-search', async (req, res) => {
+  const bucketIds = (req.query.bucketIds as string).split(',');
+  const results = await Promise.all(
+    bucketIds.map(id => client.bucket.search(id, req.query.q as string))
+  );
+  res.json(results);
+});
+```
+
+For background jobs or scripts where blocking is acceptable, the synchronous `Schift` client is fine and has simpler usage. Reserve `AsyncSchift` for code running inside an event loop.
+
+## Reference
+
+- https://docs.schift.io/sdk/python#async
+- https://docs.schift.io/sdk/typescript

schift_cli/data/schift-best-practices/references/sdk-auth-patterns.md

@@ -0,0 +1,76 @@
+---
+title: SDK Authentication Patterns
+impact: CRITICAL
+impactDescription: Hardcoded API keys in source code are the most common cause of credential leaks. A leaked `sch_` key can drain your quota and expose your data before you notice.
+tags: [sdk, auth, security, api-key]
+---
+
+## Never Hardcode API Keys
+
+API keys must never appear as string literals in source code. Keys committed to version control or embedded in build artifacts are exposed to anyone with repository access, container image access, or build log access. Rotate immediately if a key is ever exposed.
+
+Schift API keys follow the format `sch_` followed by a random string. Treat them like passwords.
+
+### Incorrect
+
+```python
+# Python - key is visible in source and will leak via git history
+from schift import Schift
+
+client = Schift(api_key="sch_abc123xyz789...")  # NEVER do this
+```
+
+```typescript
+// TypeScript - same problem; bundlers may embed this in client-side code
+import { Schift } from '@schift-io/sdk';
+
+const client = new Schift({ apiKey: 'sch_abc123xyz789...' }); // NEVER do this
+```
+
+Both examples embed a literal key that will appear in `git log`, CI logs, Docker image layers, and any bundle output.
+
+### Correct
+
+```python
+# Python - read from environment variable explicitly
+import os
+from schift import Schift
+
+client = Schift(api_key=os.environ["SCHIFT_API_KEY"])
+
+# Or rely on the SDK's automatic env-var lookup (recommended for simplicity)
+# The SDK reads SCHIFT_API_KEY automatically when api_key is omitted
+client = Schift()
+```
+
+```typescript
+// TypeScript - read from process.env
+import { Schift } from '@schift-io/sdk';
+
+const client = new Schift({ apiKey: process.env.SCHIFT_API_KEY });
+
+// Or rely on SDK auto-detection (reads SCHIFT_API_KEY if apiKey is omitted)
+const client = new Schift();
+```
+
+Set the key in your environment:
+
+```bash
+# Local development (.envrc, never committed)
+export SCHIFT_API_KEY=sch_dev_...
+
+# CI/CD - set as a masked secret variable in your platform
+# Docker / Kubernetes - inject via --env-file or a Secret resource
+```
+
+Additional best practices:
+- Use **project-scoped keys** (not org-wide admin keys) so a leaked key has minimal blast radius
+- Create **separate keys** for `dev`, `staging`, and `prod` environments
+- **Rotate keys regularly** from the Schift dashboard; old keys can be revoked without downtime
+- Add `SCHIFT_API_KEY` to `.gitignore`-adjacent `.envrc` or use a secrets manager (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault)
+
+## Reference
+
+- https://docs.schift.io/authentication
+- https://docs.schift.io/sdk/python#authentication
+- https://docs.schift.io/sdk/typescript#authentication