spruceup-ai 0.1.0__tar.gz

spruceup_ai-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+dist/
+**/.env
+**/*.pyc
+**/.venv
+.pytest_cache/
+pyrightconfig.json
+__pycache__
+.claude
+poetry.lock
+uv.lock
+Notes.md
+tenacity_test.py
+spruceup_manifest.db

spruceup_ai-0.1.0/CLAUDE.md

@@ -0,0 +1,139 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Commands
+
+```bash
+# Install dependencies (requires Python 3.14)
+uv sync
+
+# Run the app (must be run from the directory containing spruceup_pipeline.py)
+uv run spruceup start
+
+# Run all tests
+uv run pytest
+
+# Run a single test file
+uv run pytest tests/test_sync_engine.py
+
+# Run a specific test
+uv run pytest tests/test_sync_engine.py::test_reconcile_new_file
+```
+
+Required env vars for the pipeline: `PG_CONNSTR`, and an embedder API key (e.g. `OPENAI_API_KEY`). Copy credentials into `.env`; `spruceup_pipeline.py` calls `dotenv.load_dotenv()` at import time.
+
+## Architecture
+
+SpruceUp is a document ingestion daemon. It watches source connectors for file changes, transforms documents into chunks, embeds them, and keeps a target vector store in sync.
+
+### Pipeline file (`spruceup_pipeline.py`)
+
+The user-authored entry point. The CLI (`spruceup start`) imports it dynamically from the CWD. It must define a `config` variable returned by `defineConfig()`:
+
+```python
+config = defineConfig(
+    sources=[LocalFilesSource(watched_dir="example/data_corpus")],
+    target=PgVectorTarget(connstr=..., table="data_chunks", schema=LectureChunk, vector_column="chunk_embedding"),
+    embedder=OpenAIEmbedder(api_key=..., model="text-embedding-3-small"),
+    transform=build_lecture_chunks,  # async fn(*, file_props: FileProps, embed) -> list[schema]
+    cache_files=False,  # optional; True caches raw file content in the manifest (default False)
+)
+```
+
+`defineConfig()` validates types eagerly at import time. `validate_pipeline()` in `cli.py` checks the contract exists before starting the event loop.
+
+### Runtime flow (`app.py`)
+
+On startup, `app.run(pipeline)` compares persisted fingerprints in the Manifest against the current config. Any mismatch triggers a **full reindex** (all files re-fetched, re-transformed, re-upserted) instead of incremental sync:
+1. Transform function body changed (source hash)
+2. Any `@memoize`-decorated function changed
+3. Embedding model changed
+4. Embedding dimensions changed
+5. Target identity changed — `target.identity()`, a credential-free string (host/db/table or index/collection)
+6. Schema changed — `hash_schema()` over field names+types and the designated `vector_column`
+
+Signals 3–4 additionally **flush the embedding cache** (`embeddings_invalidated`). Signals 4–6 are **structural** and additionally **drop + recreate** the target table/index before reingest (`ensure_table_exists(recreate=True)`) — chosen over in-place migration because reingest must re-embed everything anyway.
+
+On any mismatch, every file row is marked `needs_reindex` and the new fingerprints are persisted immediately (mark first, persist second — a crash in between just re-marks on the next start). A file stays `needs_reindex` until a sync **succeeds**; failures and restarts don't clear it. `SyncEngine.reconcile` pushes **all** chunks of a `needs_reindex` file instead of the diff (config changes don't alter chunk hashes, so the diff can't see them). `needs_reindex` files are re-enqueued at every startup and retried by the sync sweeper, so an interrupted reindex resumes where it left off.
+
+Then it launches three concurrent asyncio tasks:
+
+| Task | Role |
+|------|------|
+| `Monitor` | Runs all watchers; each watcher does a catch-up scan then enters a watch loop |
+| `Coordinator` | Dequeues `SyncTask` objects and processes them (up to 32 concurrent) |
+| `SyncSweeper` | Retries `failed` and `needs_reindex` files every 60 seconds |
+
+### File change lifecycle
+
+```
+Source watcher → DebounceQueue → Coordinator
+                                     ↓
+                              source.fetch() → SpruceFile
+                                     ↓
+                              transform(file_props, embed) → list[UserChunk]
+                                     ↓
+                              SyncEngine.reconcile() → chunk diff → target.sync()
+                                     ↓
+                              Manifest.set_sync_state("synced")
+```
+
+`DebounceQueue` (wraps `asyncio.Queue`) evicts any already-queued task for the same `file_id` when a newer task arrives, preventing redundant processing. **Tradeoff:** to evict the superseded task it reaches into `asyncio.Queue` internals (`_queue`, `_unfinished_tasks`), so it's coupled to the CPython queue implementation and could break on a stdlib change.
+
+### Manifest (`manifest.py`)
+
+A local SQLite database (`spruceup_manifest.db`) that is the source of truth for:
+- Registered data sources and their state (e.g. Google Drive page tokens)
+- File rows: content hash, raw content (only when `cache_files=True`), sync state (`needs_reindex` / `in_flight` / `synced` / `failed`)
+- Chunk rows: `(file_id, user_chunk_object_hash)` pairs for diffing
+- Memoize cache: `(file_id, fn_hash, args_hash) → result`
+- Embedding cache: `(file_id, chunk_text_hash) → embedding bytes`
+- Config state: `embedding_model`, `embedding_dimensions`, `target_identity`, `schema_fingerprint`
+
+Opened with `autocommit=True`; use `manifest.transaction()` only when multiple writes must be atomic.
+
+### Connector ABCs (`connectors/base.py`)
+
+All connectors implement one of three ABCs:
+
+- **`SourceConnector`** — `source_type`, `source_identifier`, `create_watcher()`, `fetch()`, `validate()`, `is_supported()`, `decode_content()`
+- **`TargetConnector`** — `vector_column`, `identity()`, `ensure_table_exists(recreate=False)`, `sync(upserts, deletes)`, `aclose()`
+- **`EmbedderConnector`** — `embed_batch(batch)`, `process_chunks(chunks)`, `aclose()`
+
+Available implementations:
+
+| Type | Implementations |
+|------|----------------|
+| Source | `LocalFilesSource`, `GoogleDriveSource` |
+| Target | `PgVectorTarget`, `PineconeTarget`, `WeaviateTarget` |
+| Embedder | `OpenAIEmbedder`, `CohereEmbedder`, `GeminiEmbedder`, `VoyageAIEmbedder` |
+
+`LocalFilesSource` and `LocalFileWatcher` exist for local testing. Production reasoning should be framed in terms of the connector ABCs.
+
+An embedder's `api_key` accepts a `str` or a `Callable[[], str]` (e.g. a secrets-manager fetch, resolved at client build). On a credential rejection the embedder raises `TokenExpiredError`; the base `embed_batch_retrying` then drops the cached client so the next retry rebuilds it with a re-resolved token. Static-string keys are left untouched (no point re-resolving). Auth-error detection is per-SDK (each `embed_batch` catches its provider's exception and normalizes to `TokenExpiredError`).
+
+### EmbeddingBatcher (`connectors/embedders/embedding_batcher.py`)
+
+Wraps any `EmbedderConnector`. Accumulates chunks from concurrent file transforms and flushes them as batched API calls (max 100ms wait or `max_batch_size` chunks, max 5 concurrent API calls). Also consults the Manifest embedding cache before calling the API — cache is scoped per `file_id` and keyed by `blake2b(chunk_text)`.
+
+Each accumulated chunk gets its own `asyncio.Future`; a per-call `asyncio.gather` over those futures reassembles a caller's embeddings in order and waits for completion, so the batcher carries no per-file slot bookkeeping (a flushed batch may mix chunks from several callers).
+
+### `@memoize` decorator (`memoize/decorator.py`)
+
+Caches async subfunctions in the Manifest, scoped per file. The decorated function must be `async` — decorating a sync function raises `TypeError` at import. Results are invalidated when the function body changes. Valid **only** when called from within the `transform` function — it reads the `contextvars` set by `Coordinator.upsert_file()` via `transform_scope`.
+
+```python
+@memoize(return_type=str)
+async def summarize(text: str) -> str: ...
+```
+
+Supported return types: `str`, `int`, `float`, `bool`, `list`, `dict`.
+
+### PgVectorTarget schema mapping
+
+`ensure_table_exists()` inspects the user dataclass with `typing.get_type_hints()` and maps Python types to Postgres types. The embedding column is named **explicitly** via the target's `vector_column=` (validated at construction to be a `list[float]` field) and becomes `vector(N)` (requires `pgvector` extension), where `N` is the embedder's `embedding_dimensions`. Any *other* `list[float]` field maps to a plain `DOUBLE PRECISION[]` array, not a vector. The `id` column is always `TEXT PRIMARY KEY`, set to `f"{file_id}:{chunk.user_chunk_object_hash.hex()}"` (keyed per file). Upserts use `ON CONFLICT (id) DO UPDATE` so re-embeds (e.g. after a model change) overwrite existing rows.
+
+### Google Drive source
+
+`GoogleDriveSource` takes a `watched_dir` (folder ID) and an `on_token_expired: Callable[[], str]` that returns a fresh OAuth access token. The `drive.readonly` scope covers all required API calls (list, download, export, changes). Startup validation rejects nested watched folders.

spruceup_ai-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 SpruceUp Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

spruceup_ai-0.1.0/PKG-INFO

@@ -0,0 +1,460 @@
+Metadata-Version: 2.4
+Name: spruceup-ai
+Version: 0.1.0
+Summary: A standalone system for making automated, incremental updates to a vector database.
+Project-URL: Repository, https://github.com/SpruceUp-ai/SpruceUp
+Author-email: Ekerin Agboola <ekerin.m.a@gmail.com>, Eric Cho <ekc7590@gmail.com>, AJ Fuhler <aart.fuhler@gmail.com>, Caleb Pickard <caleb.pickard@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 SpruceUp Contributors
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: <3.15,>=3.12
+Requires-Dist: cohere<8.0.0,>=7.0.0
+Requires-Dist: google-api-python-client<3.0.0,>=2.197.0
+Requires-Dist: google-auth<3.0.0,>=2.53.0
+Requires-Dist: google-genai<3.0.0,>=2.6.0
+Requires-Dist: openai<3.0.0,>=2.37.0
+Requires-Dist: pinecone>=3.0.0
+Requires-Dist: psycopg-pool<4.0.0,>=3.0.0
+Requires-Dist: psycopg<4.0.0,>=3.3.4
+Requires-Dist: tenacity<10.0.0,>=9.1.4
+Requires-Dist: voyageai<0.5.0,>=0.4.0
+Requires-Dist: watchfiles>=1.0.0
+Requires-Dist: weaviate-client<5.0.0,>=4.0.0
+Description-Content-Type: text/markdown
+
+## SpruceUp
+
+**SpruceUp** is a standalone system for making automated, incremental updates to a vector database.
+
+## Installation
+
+Add SpruceUp to your project (e.g., with `poetry`, `pip`, or `uv`):
+
+```bash
+poetry add spruceup-ai # OR
+pip install spruceup-ai # OR
+uv add spruceup-ai
+```
+
+---
+
+## Setup
+
+Create a file named `spruceup_pipeline.py` in your project directory. This is the user-authored entry point SpruceUp loads at startup. It must export a single `config` variable built with `defineConfig()`.
+
+```python
+# spruceup_pipeline.py
+import re
+import os
+from dataclasses import dataclass
+from spruceup import defineConfig, FileProps, LocalFilesSource, PgVectorTarget, OpenAIEmbedder
+
+@dataclass
+class ArticleChunk:
+    title: str
+    content: str
+    embedding: list[float]
+
+def split_into_paragraphs(text: str) -> list[str]:
+    return [p.strip() for p in re.split(r"\n{2,}", text) if p.strip()]
+
+async def transform(*, file_props: FileProps, embed) -> list[ArticleChunk]:
+    paragraphs = split_into_paragraphs(file_props.raw_content)
+    embeddings = await embed(paragraphs)
+    return [
+        ArticleChunk(title=file_props.display_name, content=para, embedding=emb)
+        for para, emb in zip(paragraphs, embeddings)
+    ]
+
+config = defineConfig(
+    sources=[LocalFilesSource(watched_dir="./articles")],
+    target=PgVectorTarget(
+        connstr=os.environ["PG_CONNSTR"],
+        table="article_chunks",
+        schema=ArticleChunk,
+        vector_column="embedding",
+    ),
+    embedder=OpenAIEmbedder(api_key=os.environ["OPENAI_API_KEY"]),
+    transform=transform,
+)
+```
+
+---
+
+## Running SpruceUp
+
+From the directory containing your `spruceup_pipeline.py` file:
+
+```bash
+spruceup start
+```
+
+SpruceUp will scan your sources, sync any files not yet in the manifest, then enter a watch loop for incremental updates.
+
+---
+
+## Imports
+
+Everything you need is importable from the top-level `spruceup` package:
+
+```python
+from spruceup import (
+    defineConfig,
+    FileProps,
+
+    # Sources
+    LocalFilesSource,
+    GoogleDriveSource,
+
+    # Targets
+    PgVectorTarget,
+    PineconeTarget,
+    WeaviateTarget,
+
+    # Embedders
+    OpenAIEmbedder,
+    CohereEmbedder,
+    GeminiEmbedder,
+    VoyageAIEmbedder,
+
+    # Utilities
+    memoize,
+)
+```
+
+---
+
+## `defineConfig()`
+
+```python
+config = defineConfig(
+    sources=[...],
+    target=...,
+    embedder=...,
+    transform=...,
+)
+```
+
+All parameters are keyword-only.
+
+| Parameter     | Type                    | Required | Default | Description                           |
+| ------------- | ----------------------- | -------- | ------- | ------------------------------------- |
+| `sources`     | `list[SourceConnector]` | Yes      | —       | At least one source connector         |
+| `target`      | `TargetConnector`       | Yes      | —       | Where synced chunks are written       |
+| `embedder`    | `EmbedderConnector`     | Yes      | —       | Generates embeddings for your chunks  |
+| `transform`   | `async callable`        | Yes      | —       | Converts a file into a list of chunks |
+| `cache_files` | `bool`                  | No       | `False` | Cache raw file bytes in the manifest  |
+
+---
+
+## The Transform Function
+
+The transform function is where you split, enrich, and embed your documents. SpruceUp calls it for every file that changes. This function **must** be async.
+
+```python
+async def transform(*, file_props: FileProps, embed) -> list[YourSchema]:
+    ...
+```
+
+### `FileProps`
+
+| Field          | Type           | Description                                                  |
+| -------------- | -------------- | ------------------------------------------------------------ |
+| `raw_content`  | `str \| bytes` | File content. Text formats are decoded as UTF-8; binary formats like PDF are passed through as raw `bytes`. |
+| `display_name` | `str`          | The filename                                                 |
+| `file_type`    | `str`          | File extension (e.g. `"txt"`, `"pdf"`)                       |
+
+### `embed`
+
+`embed` is an async callable that takes a list of strings and returns a list of embedding vectors:
+
+```python
+embeddings: list[list[float]] = await embed(["chunk one", "chunk two"])
+```
+
+### Chunk Schema
+
+Your transform returns a list of instances of a user-defined dataclass. SpruceUp uses this schema for diffing and for writing to the target store. Define it as a plain dataclass:
+
+```python
+@dataclass
+class MyChunk:
+    title: str
+    text: str
+    embedding: list[float]
+```
+
+All target connectors support `str`, `int`, `float`, `bool`, and `list[float]` as field types. Use `list[float]` for your embedding vector. You do not need to define an `id` field. SpruceUp generates one from each chunk's content hash.
+
+---
+
+## Source Connectors
+
+### `LocalFilesSource`
+
+Watches a local directory for file changes.
+
+```python
+LocalFilesSource(watched_dir="./data")
+```
+
+| Parameter     | Type  | Required | Default | Description                    |
+| ------------- | ----- | -------- | ------- | ------------------------------ |
+| `watched_dir` | `str` | Yes      | —       | Path to the directory to watch |
+
+---
+
+### `GoogleDriveSource`
+
+Watches a Google Drive folder for file changes. Requires the `drive.readonly` OAuth scope.
+
+```python
+GoogleDriveSource(
+    watched_dir="<folder-id>",
+    on_token_expired=get_access_token,
+    recursive=True,
+)
+```
+
+| Parameter          | Type                | Required | Default | Description                                                  |
+| ------------------ | ------------------- | -------- | ------- | ------------------------------------------------------------ |
+| `watched_dir`      | `str`               | Yes      | —       | Google Drive folder ID                                       |
+| `on_token_expired` | `Callable[[], str]` | Yes      | —       | Called when the access token expires; must return a fresh token string |
+| `recursive`        | `bool`              | No       | `True`  | Whether to watch subfolders                                  |
+
+The `on_token_expired` callback is invoked whenever the connector needs a new OAuth token. It should return a valid access token or raise an exception.
+
+---
+
+## Target Connectors
+
+### `PgVectorTarget`
+
+Syncs chunks to a PostgreSQL table using the `pgvector` extension.
+
+```python
+PgVectorTarget(
+    connstr="postgresql://user:pass@localhost/mydb",
+    table="my_chunks",
+    schema=MyChunk,
+    vector_column="embedding",
+)
+```
+
+| Parameter       | Type   | Required | Default | Description                                    |
+| --------------- | ------ | -------- | ------- | ---------------------------------------------- |
+| `connstr`       | `str`  | Yes      | —       | PostgreSQL connection string                   |
+| `table`         | `str`  | Yes      | —       | Table name                                     |
+| `schema`        | `type` | Yes      | —       | Your chunk dataclass                           |
+| `vector_column` | `str`  | Yes      | —       | Field name on your schema that holds the vector |
+
+SpruceUp creates the table and its columns automatically based on your schema's type hints. The `pgvector` extension must be installed on your database.
+
+---
+
+### `PineconeTarget`
+
+Syncs chunks to a Pinecone index.
+
+```python
+PineconeTarget(
+    api_key="pc-...",
+    index_name="my-index",
+    schema=MyChunk,
+    vector_column="embedding",
+    namespace="",
+    metric="cosine",
+    cloud="aws",
+    region="us-east-1",
+)
+```
+
+| Parameter       | Type          | Required | Default       | Description                                                 |
+| --------------- | ------------- | -------- | ------------- | ----------------------------------------------------------- |
+| `api_key`       | `str \| None` | Yes      | —             | Pinecone API key                                            |
+| `index_name`    | `str`         | Yes      | —             | Name of the Pinecone index                                  |
+| `schema`        | `type`        | Yes      | —             | Your chunk dataclass                                        |
+| `vector_column` | `str`         | Yes      | —             | Field name on your schema that holds the vector             |
+| `namespace`     | `str`         | No       | `""`          | Namespace within the index                                  |
+| `metric`        | `str`         | No       | `"cosine"`    | Distance metric (`"cosine"`, `"euclidean"`, `"dotproduct"`) |
+| `cloud`         | `str`         | No       | `"aws"`       | Cloud provider                                              |
+| `region`        | `str`         | No       | `"us-east-1"` | Cloud region                                                |
+
+---
+
+### `WeaviateTarget`
+
+Syncs chunks to a Weaviate collection.
+
+```python
+# Local instance
+WeaviateTarget(
+    collection_name="MyChunks",
+    schema=MyChunk,
+    vector_column="embedding",
+    url="http://localhost:8080",
+)
+
+# Weaviate Cloud
+WeaviateTarget(
+    collection_name="MyChunks",
+    schema=MyChunk,
+    vector_column="embedding",
+    cluster_url="https://my-cluster.weaviate.network",
+    api_key="wvp-...",
+)
+```
+
+| Parameter         | Type          | Required | Default                   | Description                               |
+| ----------------- | ------------- | -------- | ------------------------- | ----------------------------------------- |
+| `collection_name` | `str`         | Yes      | —                         | Weaviate collection name                  |
+| `schema`          | `type`        | Yes      | —                         | Your chunk dataclass                      |
+| `vector_column`   | `str`         | Yes      | —                         | Field name on your schema that holds the vector |
+| `url`             | `str`         | No       | `"http://localhost:8080"` | URL for a local Weaviate instance         |
+| `cluster_url`     | `str \| None` | No       | `None`                    | URL for a Weaviate Cloud cluster          |
+| `api_key`         | `str \| None` | No       | `None`                    | API key for Weaviate Cloud authentication |
+
+Use either `url` for a local instance or `cluster_url` + `api_key` for a cloud deployment.
+
+---
+
+## Embedder Connectors
+
+SpruceUp runs a health check at startup that embeds a test string and reads the actual output size from the API. The `embedding_dimensions` parameter is optional on all embedders. If omitted, the dimension is detected automatically. If provided, SpruceUp validates it matches what the API actually returns and raises an error if not.
+
+### `OpenAIEmbedder`
+
+```python
+OpenAIEmbedder(
+    api_key="sk-...",
+    model="text-embedding-3-small",
+    max_batch_size=150,
+    embedding_dimensions=None,
+)
+```
+
+| Parameter              | Type                        | Required | Default                    | Description                |
+| ---------------------- | --------------------------- | -------- | -------------------------- | -------------------------- |
+| `api_key`              | `str \| Callable[[], str]`  | Yes      | —                          | OpenAI API key, or a callable that returns one |
+| `model`                | `str`                       | No       | `"text-embedding-3-small"` | Embedding model            |
+| `max_batch_size`       | `int`                       | No       | `150`                      | Max texts per API call     |
+| `embedding_dimensions` | `int \| None`               | No       | `None`                     | Override output dimensions. If omitted, SpruceUp reads the actual dimension from the API at startup. |
+
+---
+
+### `CohereEmbedder`
+
+```python
+CohereEmbedder(
+    api_key="...",
+    model="embed-v4.0",
+    max_batch_size=96,
+    embedding_dimensions=None,
+)
+```
+
+| Parameter              | Type                        | Required | Default        | Description                |
+| ---------------------- | --------------------------- | -------- | -------------- | -------------------------- |
+| `api_key`              | `str \| Callable[[], str]`  | Yes      | —              | Cohere API key, or a callable that returns one |
+| `model`                | `str`                       | No       | `"embed-v4.0"` | Embedding model            |
+| `max_batch_size`       | `int`                       | No       | `96`           | Max texts per API call     |
+| `embedding_dimensions` | `int \| None`               | No       | `None`         | Override output dimensions. If omitted, SpruceUp reads the actual dimension from the API at startup. |
+
+When using an `embed-v4` model with a custom `embedding_dimensions`, the value must be one of `256`, `512`, `1024`, or `1536`.
+
+---
+
+### `GeminiEmbedder`
+
+```python
+GeminiEmbedder(
+    api_key="...",
+    model="gemini-embedding-001",
+    max_batch_size=100,
+)
+```
+
+| Parameter              | Type                        | Required | Default                  | Description                              |
+| ---------------------- | --------------------------- | -------- | ------------------------ | ---------------------------------------- |
+| `api_key`              | `str \| Callable[[], str]`  | Yes      | —                        | Google Generative AI API key, or a callable that returns one |
+| `model`                | `str`                       | No       | `"gemini-embedding-001"` | Embedding model                          |
+| `max_batch_size`       | `int`                       | No       | `100`                    | Max texts per API call (hard limit: 100) |
+| `embedding_dimensions` | `int \| None`               | No       | `None`                   | Override output dimensions. If omitted, SpruceUp reads the actual dimension from the API at startup. |
+
+---
+
+### `VoyageAIEmbedder`
+
+```python
+VoyageAIEmbedder(
+    api_key="...",
+    model="voyage-4-large",
+    max_batch_size=150,
+    embedding_dimensions=None,
+)
+```
+
+| Parameter              | Type                        | Required | Default            | Description                |
+| ---------------------- | --------------------------- | -------- | ------------------ | -------------------------- |
+| `api_key`              | `str \| Callable[[], str]`  | Yes      | —                  | Voyage AI API key, or a callable that returns one |
+| `model`                | `str`                       | No       | `"voyage-4-large"` | Embedding model            |
+| `max_batch_size`       | `int`                       | No       | `150`              | Max texts per API call     |
+| `embedding_dimensions` | `int \| None`               | No       | `None`             | Override output dimensions. If omitted, SpruceUp reads the actual dimension from the API at startup. |
+
+When using a `voyage-4` model with a custom `embedding_dimensions`, the value must be one of `256`, `512`, `1024`, or `2048`.
+
+---
+
+## `@memoize`
+
+The `memoize` decorator caches the results of expensive subfunctions inside your transform. Results are stored in the SpruceUp manifest (a local SQLite database), scoped per file and invalidated automatically when the decorated function's body changes.
+
+```python
+from spruceup import memoize
+import asyncio
+
+@memoize(return_type=str)
+async def summarize(text: str) -> str:
+    # expensive LLM call
+    ...
+
+async def transform(*, file_props: FileProps, embed) -> list[MyChunk]:
+    chunk_strs = split_into_chunks(file_props.raw_content)
+    # summarize each chunk concurrently; results are cached per file
+    summaries = await asyncio.gather(*[summarize(c) for c in chunk_strs])
+    embeddings = await embed(chunk_strs)
+    return [
+        MyChunk(content=c, summary=s, embedding=e)
+        for c, s, e in zip(chunk_strs, summaries, embeddings)
+    ]
+```
+
+| Parameter     | Type   | Required | Description                                                  |
+| ------------- | ------ | -------- | ------------------------------------------------------------ |
+| `return_type` | `type` | Yes      | Return type of the decorated function — used for serialization |
+
+Supported return types: `str`, `int`, `float`, `bool`, `list`, `dict`.
+
+`memoize` only works on `async` functions. Decorating a sync function raises a `TypeError`. It can only be used inside a transform function. Calling a memoized function outside of a transform context will raise a `RuntimeError`.