@fizzyflow/endless-vector 0.0.7 → 0.0.10

package/README.md

@@ -1,6 +1,6 @@
 # Endless Vector - JavaScript SDK
 
-JavaScript/TypeScript SDK for interacting with the Endless Vector smart contract on the Sui blockchain. Endless Vector provides a scalable, on-chain data structure storing `vector<vector<u8>>` that can grow beyond Sui object size limits.
+JavaScript/TypeScript SDK for the Endless Vector smart contract on [Sui](https://sui.io). Endless Vector is a scalable, append-only on-chain `vector<vector<u8>>` that grows beyond Sui object size limits by automatically splitting data into history segments. Items larger than ~120 KB are transparently stored as [Walrus](https://walrus.xyz) blobs. Optional [Seal](https://github.com/nicola/seal) encryption protects all stored data with AES-256-GCM.
 
 ## Installation
 
@@ -10,276 +10,346 @@ npm install @fizzyflow/endless-vector
 
 ## Quick Start
 
-### Creating a New Vector
-
 ```javascript
-import { SuiClient } from '@mysten/sui/client';
 import { EndlessVector } from '@fizzyflow/endless-vector';
 
-const client = new SuiClient({ url: 'https://fullnode.mainnet.sui.io:443' });
-
-// Create an empty vector
-const vector = await EndlessVector.create({
+// Create
+const ev = await EndlessVector.create({
     suiClient: client,
-    packageId: 'testnet',  // or 'mainnet' or '0xYOUR_PACKAGE_ID'
+    packageId: 'testnet', // or 'mainnet', or an explicit 0x... package ID
     signAndExecuteTransaction: async (tx) => {
         const result = await wallet.signAndExecuteTransaction({ transaction: tx });
         return result.digest;
-    }
+    },
 });
 
-// Or create with initial data
-const vectorWithData = await EndlessVector.create({
-    suiClient: client,
-    packageId: 'testnet',  // or 'mainnet' or '0xYOUR_PACKAGE_ID'
-    array: new Uint8Array([1, 2, 3]),  // [0] to append to EndlessVector
-    //array: [new Uint8Array([1, 2, 3]), new Uint8Array([5, 6, 7])],  // or [0] and [1] to append to EndlessVector
-    signAndExecuteTransaction: async (tx) => {
-        const result = await wallet.signAndExecuteTransaction({ transaction: tx });
-        return result.digest;
-    }
-});
+// Write
+await ev.push(new Uint8Array([1, 2, 3]));
+
+// Read
+const item = await ev.at(0); // Uint8Array
 ```
 
-### Reading an Existing Vector
+## Constructor
 
 ```javascript
-const vector = new EndlessVector({
-    suiClient: client,
-    id: '0xYOUR_VECTOR_OBJECT_ID'
-});
+const ev = new EndlessVector(params);
+```
 
-await vector.initialize();
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `suiClient` | SuiGrpcClient | yes | Sui gRPC client instance |
+| `id` | string | yes | On-chain object ID of the vector |
+| `packageId` | string | no | Move package ID, or `'testnet'`/`'mainnet'`. Enables writes |
+| `signAndExecuteTransaction` | function | no | Signs and submits a Transaction, returns its digest. Enables writes |
+| `walrusClient` | WalrusClient | no | `@mysten/walrus` client for blob read/write |
+| `publisherUrl` | string | no | Walrus publisher HTTP URL (fallback when no `walrusClient`) |
+| `aggregatorUrl` | string | no | Walrus aggregator HTTP URL (fallback when no `walrusClient`) |
+| `senderAddress` | string | no | Sender Sui address, required for Walrus blob writes |
+| `sealClient` | SealClient | no | `@mysten/seal` client for encryption/decryption |
+| `sessionKey` | SessionKey | no | Pre-built Seal SessionKey. Alternative to `signer` |
+| `signer` | Signer | no | Keypair or wallet signer used to auto-create a SessionKey when needed |
+| `sealTtlMin` | number | no | SessionKey TTL in minutes (default: 5) |
 
-console.log('Total items:', vector.length);
-console.log('Total size:', vector.binaryLength, 'bytes');
+Providing only `suiClient` + `id` gives a **read-only** instance. Add `packageId` + `signAndExecuteTransaction` for **writes**. Add Walrus params for **large items** (>120 KB). Add Seal params for **encryption**.
 
-// Read items
-const firstItem = await vector.at(0); // Uint8Array
+## EndlessVector.create(params)
+
+Creates a new on-chain vector.
+
+Accepts all constructor params above, plus:
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `array` | Uint8Array \| Uint8Array[] | no | Initial item(s) to push |
+| `gasCoin` | `{objectId, digest, version}` | no | Explicit gas coin for parallel creation |
+| `options.timeout` | number | no | Tx confirmation timeout, ms (default: 30000) |
+| `options.pollIntervalMs` | number | no | Tx poll interval, ms (default: 200) |
+
+When `sealClient` is provided, the vector is created with Seal encryption enabled. A random AES-256-GCM key is generated, Seal-wrapped scoped to the new vector's object ID, and stored on-chain. Any initial `array` items are encrypted before storage.
+
+```javascript
+const ev = await EndlessVector.create({
+    suiClient: client,
+    packageId: 'testnet',
+    sealClient,
+    signer: keypair,
+    signAndExecuteTransaction: sign,
+});
+// All push() / at() calls now encrypt/decrypt transparently
 ```
 
-## API Reference
+## Properties
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `id` | string | Object ID |
+| `isWritable` | boolean | Whether writes are enabled |
+| `length` | number | Total item count (append-only, never decreases) |
+| `binaryLength` | number | Total size of all items in bytes |
+| `sealEncryptedKey` | Uint8Array \| null | Seal-wrapped AES key, or null if unencrypted |
+| `seal` | EndlessVectorSeal | Seal companion (always present; active only when `sealClient` was provided) |
+| `walrus` | EndlessVectorWalrus | Walrus companion (always present; active only when Walrus params were provided) |
+| `historyItemsCount` | number | History segments in the current object |
+| `archiveItemsCount` | number | Total archive entries ever created |
+| `archivedAtLength` | number | `length` at the time of the last archive |
+| `archivedFromLength` | number | Items before this index have been burned |
+| `burnedArchiveCount` | number | Burned archive count |
+| `firstNotHistoryIndex` | number | First index stored in the current object |
 
-### Static Methods
+## Methods
 
-#### EndlessVector.create(params)
+### initialize()
 
-Creates a new EndlessVector on the blockchain.
+Loads metadata from chain. Most read methods call this internally.
+
+```javascript
+await ev.initialize();
+```
 
-**Parameters:**
-- `suiClient` (SuiClient) - Sui client instance for blockchain interactions
-- `packageId` (string) - 'testnet', 'mainnet', or ID of the Move package containing the EndlessVector module
-- `signAndExecuteTransaction` (function) - Function to sign and execute transactions
-- `array` (Uint8Array or Uint8Array[], optional) - Optional first vector<u8>(s) to push back to the new vector
-- `gasCoin` (Object, optional) - Gas coin object reference `{objectId: string, digest: string, version: string}` for transaction payment
-- `options` (Object, optional) - Additional options:
-  - `timeout` (number) - Transaction confirmation timeout in ms (default: 30000)
-  - `pollIntervalMs` (number) - Poll interval in ms (default: 1000)
+### reInitialize()
 
-**Returns:** Promise<EndlessVector>
+Marks the instance stale so the next operation re-fetches from chain.
 
-**Example:**
 ```javascript
-const vector = await EndlessVector.create({
-    suiClient: client,
-    packageId: 'testnet',  // or 'mainnet' or '0xPACKAGE_ID'
-    array: new Uint8Array([1, 2, 3]),
-    gasCoin: {
-        objectId: '0xGAS_COIN_ID',
-        digest: 'DIGEST',
-        version: 'VERSION'
-    },
-    signAndExecuteTransaction: async (tx) => {
-        const result = await wallet.signAndExecuteTransaction({ transaction: tx });
-        return result.digest;
-    }
-});
+ev.reInitialize();
 ```
 
-### Constructor
+### isEncrypted()
+
+Async. Returns `true` if the vector has a Seal encryption key on-chain. Calls `initialize()` internally.
 
 ```javascript
-const vector = new EndlessVector({
-    suiClient,                 // SuiClient instance (required for reading)
-    id,                        // Object ID of the EndlessVector (required)
-    packageId,                 // 'testnet', 'mainnet', or Package ID for write operations (optional)
-    signAndExecuteTransaction  // Function to sign/execute transactions (optional)
-});
+if (await ev.isEncrypted()) { /* ... */ }
 ```
 
-**Modes:**
-- **Read-only mode**: Provide only `suiClient` and `id`
-- **Writable mode**: Provide all parameters including `packageId` and `signAndExecuteTransaction`
+### push(arr, params?)
 
-### Properties
+Appends one or more `Uint8Array` items. Requires writable mode.
 
-- `id` (string) - Object ID of the EndlessVector
-- `isWritable` (boolean) - Whether the instance can perform write operations
-- `length` (number) - Total number of items in the vector
-- `binaryLength` (number) - Total binary size of all items in bytes
-- `historyItemsCount` (number) - Number of history segments
-- `archiveItemsCount` (number) - Number of archive segments
-- `archivedFromLength` (number) - Starting index after burned archives
-- `burnedArchiveCount` (number) - Number of archives that have been burned
-- `firstNotHistoryIndex` (number) - First index stored in current object (not in history)
+- Items up to ~120 KB are stored on-chain as `vector<u8>`.
+- Larger items are stored as Walrus blobs (requires Walrus params).
+- On encrypted vectors, every item is AES-256-GCM encrypted before storage (28 bytes overhead per item).
 
-### Methods
+```javascript
+await ev.push(new Uint8Array([1, 2, 3]));
+await ev.push([chunk1, chunk2, chunk3]); // multiple items
+```
 
-#### initialize()
+### getPushTransaction(arr, tx?)
 
-Loads the vector's metadata from the blockchain. Called automatically by most methods.
+Returns a `Transaction` without executing it. Useful for batching multiple pushes.
 
 ```javascript
-await vector.initialize();
+const tx = new Transaction();
+ev.getPushTransaction(data1, tx);
+ev.getPushTransaction(data2, tx);
+await signAndExecuteTransaction(tx);
 ```
 
-#### reInitialize()
+### at(index)
 
-Forces a reload of the vector's metadata, clearing caches.
+Reads the item at a zero-based index. On encrypted vectors, decrypts transparently.
 
 ```javascript
-await vector.reInitialize();
+const data = await ev.at(0); // Uint8Array
 ```
 
-#### push(arr, params)
+### concat(other, params?)
 
-Pushes a Uint8Array or few Uint8Array(Uint8Array[]) to the vector. Requires writable mode. Maximum size per push: ~120KB.
+Appends all items from another vector (or array of vectors) into this one. Sources are consumed (destroyed).
 
 ```javascript
-const data = new Uint8Array([1, 2, 3, 4, 5]);
-await vector.push(data);
+await ev.concat(otherVector);
+await ev.concat([v2, v3]);
+await ev.concat('0xOTHER_VECTOR_ID');
 ```
 
-**Parameters:**
-- `arr` (Uint8Array or Uint8Array[]) - Data to push
-- `params` (Object, optional) - Additional parameters
+**Restrictions:** cannot concat vectors that have archived items or that are Seal-encrypted.
 
-#### getPushTransaction(arr, tx)
+### getConcatTransaction(other, tx?)
 
-Creates a transaction for pushing data without executing it. Useful for batching multiple pushes.
+Returns a concat `Transaction` without executing.
 
-```javascript
-// Single push transaction
-const tx = vector.getPushTransaction(new Uint8Array([1, 2, 3]));
-await signAndExecuteTransaction(tx);
+### archive(params?)
 
-// Multiple pushes in one transaction
-const tx = new Transaction();
-vector.getPushTransaction(new Uint8Array([1, 2, 3]), tx);
-vector.getPushTransaction(new Uint8Array([4, 5, 6]), tx);
-await signAndExecuteTransaction(tx);
+Sweeps current history segments into a new archive entry, freeing capacity for future pushes.
+
+```javascript
+await ev.archive();
 ```
 
-**Parameters:**
-- `arr` (Uint8Array or Uint8Array[]) - Data to push
-- `tx` (Transaction, optional) - Existing transaction to append to
+### getArchiveTransaction(tx?)
 
-**Returns:** Transaction
+Returns an archive `Transaction` without executing.
 
-#### at(index)
+### burnArchive(params?)
 
-Retrieves an item at a specific index. Alias: `get(index)`
+Permanently deletes the oldest archive entry. Items in the burned range become unreadable.
 
 ```javascript
-const item = await vector.at(42);  // Returns Uint8Array
+await ev.burnArchive();
+// ev.archivedFromLength now advanced; at() throws for burned indices
 ```
 
-**Parameters:**
-- `index` (number) - Zero-based index
+### getBurnArchiveTransaction(tx?)
 
-**Returns:** Promise<Uint8Array>
+Returns a burn-archive `Transaction` without executing.
 
-#### concat(other)
+## Walrus Blob Storage
 
-Concatenates another EndlessVector (or array of vectors) into this one. The other vector(s) will be consumed.
+When Walrus params are configured, items larger than ~120 KB are automatically stored as Walrus blobs instead of on-chain `vector<u8>`. The SDK handles upload, certification, and read-back. On encrypted vectors, blobs contain ciphertext only.
 
 ```javascript
-// Concat single vector
-await vector1.concat(vector2);
-
-// Concat multiple vectors at once
-await vector1.concat([vector2, vector3, vector4]);
+const ev = new EndlessVector({
+    suiClient: client,
+    id: '0x...',
+    packageId: 'testnet',
+    walrusClient,                       // or aggregatorUrl + publisherUrl
+    senderAddress: wallet.address,
+    signAndExecuteTransaction: sign,
+});
 
-// Also accepts object IDs
-await vector1.concat('0xVECTOR2_ID');
-await vector1.concat(['0xVECTOR2_ID', '0xVECTOR3_ID']);
+await ev.push(largeFile); // >120 KB → stored as Walrus blob
+const data = await ev.at(0); // fetched from Walrus transparently
 ```
 
-**Parameters:**
-- `other` (string | EndlessVector | Array<string | EndlessVector>) - Vector(s) to concatenate
+### Blob lifetime: inspect & extend
 
-**Returns:** Promise<void>
+Walrus blobs are backed by storage that expires at an `end_epoch`. The Walrus companion (`ev.walrus`) can report when a vector's blobs expire and renew them all in a single transaction — covering blobs in current items, history segments, and non-burned archive segments. These methods require `walrusClient` (to resolve the Walrus `System` object and current storage price).
 
-**Note:** Cannot concat vectors that have archived items.
+#### minBlobEndEpoch()
 
-#### getConcatTransaction(other, tx)
+Async. Reads the soonest-expiring blob's `end_epoch` across the whole vector, on-chain via simulation (no gas, works read-only). Returns `null` if the vector holds no blobs.
 
-Creates a transaction for concatenation without executing it.
+```javascript
+const minEpoch = await ev.walrus.minBlobEndEpoch(); // number | null
+```
+
+#### extendBlobsCostToEpoch(targetEndEpoch)
+
+Async. Returns the exact WAL cost (in FROST, as a `bigint`) to bring every blob up to `targetEndEpoch`. Computed on-chain, so it accounts for every blob across all tiers — even ones not loaded in the SDK. `0n` when nothing needs extending.
 
 ```javascript
-const tx = vector1.getConcatTransaction(vector2);
-await signAndExecuteTransaction(tx);
+const cost = await ev.walrus.extendBlobsCostToEpoch(minEpoch + 10); // bigint (FROST)
+```
 
-// Or append to existing transaction
-const tx = new Transaction();
-vector1.getConcatTransaction([vector2, vector3], tx);
-await signAndExecuteTransaction(tx);
+#### extendBlobsToEpoch(targetEndEpoch, params?)
+
+Async. Extends every blob whose storage ends before `targetEndEpoch` up to that epoch, in one transaction. Blobs already valid through the target (and expired blobs, which Walrus cannot extend) are skipped on-chain. Resolves to the new minimum blob end epoch. Requires writable mode and `senderAddress`.
+
+The WAL payment is sourced automatically from the sender's balance for exactly the on-chain cost, and any leftover is returned to the sender. Pass `walCoin` to pay from a specific coin, or `cost` to skip the cost read.
+
+```javascript
+const minEpoch = await ev.walrus.minBlobEndEpoch();
+const newMin = await ev.walrus.extendBlobsToEpoch(minEpoch + 10);
+// newMin === minEpoch + 10
 ```
 
-**Parameters:**
-- `other` (string | EndlessVector | Array<string | EndlessVector>) - Vector(s) to concatenate
-- `tx` (Transaction, optional) - Existing transaction to append to
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `cost` | bigint | no | Precomputed cost in FROST; skips the on-chain cost read |
+| `walCoin` | TransactionObjectArgument | no | WAL coin to pay from; if omitted, one is sourced from the sender's balance |
+| `timeout` | number | no | Tx confirmation timeout, ms |
+| `pollIntervalMs` | number | no | Tx poll interval, ms |
+
+#### getExtendBlobsToEpochTransaction(targetEndEpoch, params?)
+
+Async. Returns the extend `Transaction` without executing, for batching. Accepts `cost`, `walCoin`, and `txToAppendTo`.
 
-**Returns:** Transaction
+## Seal Encryption
 
-## Usage Examples
+Seal provides end-to-end encryption for vector items. The access policy (`seal_approve_endless_vector_owner`) ensures only the vector owner can decrypt.
+
+### Creating an encrypted vector
 
 ```javascript
-const vector = new EndlessVector({
+const ev = await EndlessVector.create({
     suiClient: client,
-    id: '0xVECTOR_ID'
+    packageId: 'testnet',
+    sealClient,
+    signer: keypair,
+    signAndExecuteTransaction: sign,
 });
+// AES key generated → Seal-wrapped → stored on-chain
+// All push()/at() calls encrypt/decrypt transparently
+```
 
-await vector.initialize();
+### Reading an encrypted vector
 
-// Read metadata
-console.log('Items:', vector.length);
-console.log('Size:', vector.binaryLength, 'bytes');
-console.log('History segments:', vector.historyItemsCount);
-console.log('Archive segments:', vector.archiveItemsCount);
+You can provide a `signer` and the SDK auto-creates a SessionKey:
 
-// Read specific item
-const item = await vector.at(42); // Uint8Array
+```javascript
+const ev = new EndlessVector({
+    suiClient: client,
+    id: '0x...',
+    sealClient,
+    signer: keypair,          // SDK creates a 5-minute SessionKey automatically
+});
+const data = await ev.at(0); // decrypted
 ```
 
-### Custom Gas Coin for Parallel Operations
-
-To execute transactions in parallel, speeding up data upload process, you would probably need separate gas coin for each tx:
+Or provide a pre-built `sessionKey` (useful in browser wallets where signing is interactive):
 
 ```javascript
-// Get available gas coins
-const coins = await client.getCoins({
-    owner: address,
-    coinType: '0x2::sui::SUI'
+const ev = new EndlessVector({
+    suiClient: client,
+    id: '0x...',
+    sealClient,
+    sessionKey: mySessionKey, // created externally, e.g. via wallet adapter
 });
-const gasCoinRefs = coins.data.map(c => ({
-    objectId: c.coinObjectId,
-    digest: c.digest,
-    version: c.version
-}));
+const data = await ev.at(0); // decrypted using the provided SessionKey
+```
+
+You can also set the session key after construction:
+
+```javascript
+ev.seal._sessionKey = mySessionKey;
+```
+
+### How it works
+
+1. `create()` generates a random AES-256-GCM key
+2. The key is Seal-wrapped scoped to the vector's object ID and stored on-chain as `seal_encrypted_key`
+3. `push()` encrypts each item before storage (adds 28 bytes: 12B nonce + 16B GCM tag)
+4. `at()` unwraps the AES key via Seal (requires a valid SessionKey), then decrypts the item
+5. The unwrapped AES key is cached in memory for subsequent reads
+
+Passing `sealClient` to an unencrypted vector is safe — `push()` and `at()` check for the on-chain `sealEncryptedKey` before attempting any encryption/decryption.
+
+## Examples
+
+### Archive and burn lifecycle
 
-// Create vectors in parallel, each with its own gas coin
+```javascript
+await ev.push(largeData);
+await ev.archive();
+await ev.initialize();
+
+console.log(ev.archiveItemsCount);  // 1
+console.log(ev.archivedAtLength);   // e.g. 5
+
+const item = await ev.at(0); // still readable
+
+await ev.burnArchive();
+await ev.initialize();
+
+console.log(ev.burnedArchiveCount);  // 1
+console.log(ev.archivedFromLength);  // 5 — items 0..4 are gone
+await ev.at(0); // throws
+```
+
+### Parallel vector creation with gas coins
+
+```javascript
 const vectors = await Promise.all(
-    testData.map((items, i) =>
+    dataChunks.map((chunk, i) =>
         EndlessVector.create({
             suiClient: client,
-            packageId: 'testnet',  // or 'mainnet' or '0xPACKAGE_ID'
-            array: items,
+            packageId: 'testnet',
+            array: chunk,
             gasCoin: gasCoinRefs[i],
-            signAndExecuteTransaction: async (tx) => {
-                const result = await wallet.signAndExecuteTransaction({ transaction: tx });
-                return result.digest;
-            }
+            signAndExecuteTransaction: sign,
         })
     )
 );
@@ -287,38 +357,23 @@ const vectors = await Promise.all(
 
 ## Testing
 
-Run the test suite:
-
 ```bash
-npm test
+pnpm test:base                   # core tests
+pnpm test:seal                   # seal encryption tests
+pnpm test:walrus-blobs           # walrus blob storage
+pnpm test:walrus-blobs-sdk       # walrus blob storage (SDK client)
+pnpm test:walrus-blobs-history   # blob storage across history segments
+pnpm test:walrus-blobs-extend    # blob lifetime: min end epoch, cost, extend
 ```
 
-Tests use the TAP framework and require a configured local sui validator installed.
-
-### Performance Considerations
-
-- **Caching**: Loaded history and archive segments are cached
-- **Lazy Loading**: History/archive data is only loaded when accessed
-- **Batch Writes**: Use `getPushTransaction()` to combine multiple pushes in one transaction
-- **Re-initialization**: After `push()` or `concat()`, metadata is refreshed on next read
-
-### Concatenation
-
-The `concat()` method efficiently merges vectors by transferring ownership of internal data structures rather than copying items one by one. This makes it very efficient for combining large datasets.
-
-**Restrictions:**
-- Cannot concatenate vectors that have archived items
-- The concatenated vector is consumed (destroyed) in the process
-- All items from concatenated vector(s) are appended in order
+Tests use [vitest](https://vitest.dev/) and require a local Sui validator with Walrus and Seal localnet services.
 
 ## License
 
 Apache-2.0
 
-## Repository
-
-https://github.com/fizzyFlow/endless_vector
-
-## Author
+## Links
 
-[suidouble](https://github.com/suidouble)
+- [Repository](https://github.com/fizzyFlow/endless_vector)
+- [npm](https://www.npmjs.com/package/@fizzyflow/endless-vector)
+- [Author](https://github.com/suidouble)

package/ids.js

@@ -1,11 +1,11 @@
 
 export default {
     'testnet': {
-        packageId: '0x609cf37d16c135b784f32db4755ed0dafbaa2d5e24ab6edfa25a45a9b4ca7ffd', // should be the same as in Move.lock
-        originalPackageId: '0x30bf8e91f40774ed08234f767360b4154562232903928b5f2f8fe5b2d51655bc',
+        packageId: '0xaa7b048bc68a31204955da199b18020d68bb9823ae021b4c5d3f7245330ce825', // should be the same as in Move.lock
+        originalPackageId: '0xaa7b048bc68a31204955da199b18020d68bb9823ae021b4c5d3f7245330ce825',
     },
     'mainnet': {
-        packageId: '0x94cf19d164f014cfe73dda1121f27b54b69f0dbfa89d8b6ed48ae8a9144f5d65', // should be the same as in Move.lock
-        originalPackageId: '0x94cf19d164f014cfe73dda1121f27b54b69f0dbfa89d8b6ed48ae8a9144f5d65',
+        packageId: '0xec49cff1adbf19e0fdfa7933047162a749005fdfb64a1cebfb1ea5a8727b16f3', // should be the same as in Move.lock
+        originalPackageId: '0xec49cff1adbf19e0fdfa7933047162a749005fdfb64a1cebfb1ea5a8727b16f3',
     },  
 };