@ckbfs/api 1.0.0

package/README.md

@@ -0,0 +1,132 @@
+# CKBFS API SDK
+
+A JavaScript/TypeScript SDK for the CKBFS (Cell Knowledge Base File System) protocol on Nervos CKB.
+
+## Overview
+
+CKBFS is a protocol designed to describe a witnesses-based file storage system on Nervos CKB Network. With CKBFS, you can:
+
+- Store files on Nervos CKB Network
+- Publish large files that may exceed block size limitation (~500kb), across multiple blocks
+- Maintain simple indexing and retrieval
+
+This SDK provides a simple way to interact with the CKBFS protocol, allowing you to:
+
+- Read and write files to CKB
+- Calculate and verify Adler32 checksums
+- Create, manage, and update CKBFS cells
+- Handle witnesses creation and verification
+- Create and send CKB transactions for file operations
+
+## Installation
+
+```bash
+npm install ckbfs-api
+```
+
+## Basic Usage
+
+```typescript
+import { CKBFS } from 'ckbfs-api';
+
+// Initialize the SDK with your private key
+const privateKey = process.env.CKB_PRIVATE_KEY;
+const ckbfs = new CKBFS(privateKey);
+
+// Publish a file to CKBFS
+async function publishExample() {
+  const txHash = await ckbfs.publishFile('path/to/your/file.txt');
+  console.log(`File published with transaction: ${txHash}`);
+}
+
+// Append content to an existing CKBFS file
+async function appendExample(ckbfsCell) {
+  const txHash = await ckbfs.appendFile('path/to/more/content.txt', ckbfsCell);
+  console.log(`Content appended with transaction: ${txHash}`);
+}
+```
+
+## Advanced Usage
+
+The SDK provides utility functions for more granular control:
+
+### Working with Files
+
+```typescript
+import { readFileAsUint8Array, splitFileIntoChunks } from 'ckbfs-api';
+
+// Read a file
+const fileContent = readFileAsUint8Array('path/to/file.txt');
+
+// Split a file into chunks (e.g., for large files)
+const chunks = splitFileIntoChunks('path/to/large/file.bin', 30 * 1024); // 30KB chunks
+```
+
+### Checksum Operations
+
+```typescript
+import { calculateChecksum, verifyChecksum } from 'ckbfs-api';
+
+// Calculate Adler32 checksum
+const content = new TextEncoder().encode('Hello CKBFS');
+const checksum = await calculateChecksum(content);
+
+// Verify checksum
+const isValid = await verifyChecksum(content, checksum);
+```
+
+### Transaction Utilities
+
+```typescript
+import { createPublishTransaction, createAppendTransaction } from 'ckbfs-api';
+
+// Create a transaction without signing or sending
+const tx = await ckbfs.createPublishTransaction('path/to/file.txt');
+
+// Customize the transaction before signing
+// ... modify tx ...
+
+// Sign and send manually
+const signedTx = await signer.signTransaction(tx);
+const txHash = await client.sendTransaction(signedTx);
+```
+
+## Examples
+
+The SDK comes with several examples demonstrating how to use it:
+
+### Running Examples
+
+```bash
+# Show available examples
+npm run example
+
+# Run the publish file example
+npm run example:publish
+
+# Run the append file example (requires a transaction hash)
+npm run example:append -- --txhash=0x123456...
+# OR
+PUBLISH_TX_HASH=0x123456... npm run example:append
+```
+
+### Example Files
+
+- `examples/publish.ts` - Shows how to publish a file to CKBFS
+- `examples/append.ts` - Shows how to append to a previously published file
+
+To run the examples, first set your CKB private key:
+
+```bash
+export CKB_PRIVATE_KEY=your_private_key_here
+```
+
+Note: The append example requires a valid transaction hash from a previously published file. You can either provide it via command line argument or environment variable.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details. 

package/RFC.v2.md

@@ -0,0 +1,341 @@
+# CKBFS Protocol
+
+`CKBFS` is a protocol defined in order to describe a witnesses based file storage system. With CKBFS, one can:
+
+- Store files on Nervos CKB Network
+- Publish large files that may exceed block size limitation(~500kb), across multiple blocks, while still keep the index and retrieve action simple and straight
+
+It contains:
+
+- A hasher binary that uses [Adler-32 checksum algorithm](https://en.wikipedia.org/wiki/Adler-32)
+- A type script contract that applies restrictions to avoid invalid file publication
+
+## Core Concepts
+
+Core concepts of  CKBFS is:
+
+- One single cell(called CKBFS cell) to index one file, even multi-part file that split across blocks.
+- **Permanent** storage. Deletion is **forbidden** through contract, which means the file metadata will never lost. Which also means you will need to lock some CKB capacity **FOREVER** in order to create it.
+- Simple cell structure, encoded with molecule.
+- Built-in checksum provided, both on-chain and off-chain side.
+
+# The Protocol - or, the standard
+
+## Data Structure
+
+### CKBFS Cell
+
+CKBFS Cell is a cell that stores:
+
+A CKBFS cell in should looks like following:
+
+```yaml
+Data:
+  content_type: Bytes # String Bytes
+  filename: Bytes # String Bytes
+  indexes: Vec<Uint32> # referenced witnesses index.
+  checksum: Uint32 # Adler32 checksum
+  backlinks: Vec<BackLink>
+
+Type:
+  hash_type: "data1"
+  code_hash: CKBFS_TYPE_DATA_HASH
+  args: <TypeID, 32 bytes>,[<hasher_code_hash>, optional]
+Lock:
+  <user_defined>
+```
+
+The following rules should be met in a CKBFS cell:
+
+- Rule 1: data structure of a CKBFS cell is molecule encoded. See [Molecule](https://github.com/nervosnetwork/molecule) definitions below.
+- Rule 2: checksum must match with specified witnesses. Default checksum algorithm will be Alder32 if not specify `hasher_code_hash` in Type script args.
+- Rule 3: if backlinks(see definition below) of a CKBFS cell is not empty, it means file was stored across blocks.
+- Rule 4: if `hasher_code_hash` is specified, then it will use hasher binary from CellDeps that matches `code_hash`, with same input parameter.
+- Rule 5: Once created, a CKBFS cell can only be updated/transfered, which means it can not be destroyed.
+- UPDATES IN VERSION 2: `indexes` is a vector of witness indexes, stored CKBFS structured contents in splited witnesses.
+
+---
+
+### BackLink
+
+BackLink stands for the prefix part of a living CKBFS cell.  The strategy of CKBFS is similar to a linked list:
+
+[backlink]←[backlink]←[…]←[CKBFS cell]
+
+```yaml
+BackLink:
+  tx_hash: Bytes,
+  indexes: Vec<Uint32>,
+  checksum: Uint32,
+```
+
+---
+
+### Witnesses
+
+File contents are stored in witnesses. In a single transaction, witnesses can be splitted into multiple parts and concat together while verification. 
+
+```yaml
+Witnesses:
+  <"CKBFS"><0x00><CONTENT_BYTES>
+```
+
+The following rules should be met for witnesses used in CKBFS:
+
+- Rule 6: The first 5 bytes must be UTF8 coded string bytes of `CKBFS`, which should be: `0x434b424653`
+- Rule 7: The 6th byte of witnesses must be the version of CKBFS protocol, which should be: `0x00`.
+- Rule 8: File contents bytes are stored from 7th byte. Checksum hasher should also take bytes from `[7…]`.
+- UPDATES IN VERSION 2: Every parts of the splitted content stored in witnesses must follow Rule 6 to Rule 8.
+
+---
+
+## Operations
+
+This section describes operations and restrictions in CKBFS implementation
+
+### Publish
+
+Publish operation creates one or more new CKBFS cell.
+
+```yaml
+// Publish
+Witnesses:
+  <...>
+  <0x434b424653, 0x0, CKBFS_CONTENT_BYTES>
+  <...>
+Inputs:
+  <...>
+Outputs:
+  <vec> CKBFS_CELL
+    Data:
+      content-type: string
+      filename: string
+      indexes: vec<uint32>
+      checksum: uint32
+      backlinks: empty_vec
+    Type:
+      code_hash: ckbfs type script
+      args: 32 bytes type_id, (...)
+  <...>
+```
+
+Publish operation must satisfy following rule:
+
+- Rule 9: in a publish operation, checksum must be equal with `hash(Witnesses[index])`.
+
+---
+
+### Append
+
+Append operation updates exist live CKBFS cell, validates the latest checksum.
+
+```yaml
+// Append
+Witnesses:
+  <...>
+  <CKBFS_CONTENT_BYTES>
+  <...>
+Inputs:
+  <...>
+  CKBFS_CELL
+    Data:
+      content-type: string
+      filename: string
+      indexes: vec<uint32>
+      checksum: uint32
+      backlinks: empty_vec
+    Type:
+      code_hash: ckbfs type script
+      args: 32 bytes type_id, (...)
+  <...>
+Outputs:
+  <...>
+  CKBFS_CELL:
+    Data:
+      content-type: string
+      filename: string
+      indexes: vec<uint32>
+      checksum: uint32 # updated checksum
+      backlinks: vec<BackLink>
+    Type:
+      code_hash: ckbfs type script
+      args: 32 bytes type_id, (...)
+```
+
+- Rule 10: backlinks field of a CKBFS cell can only be appended. Once allocated, all records in the vector can not be modified.
+- Rule 11: new checksum of updated CKBFS cell should be equal to:  `hasher.recover_from(old_checksum).update(new_content_bytes)`
+- Rule 12: `content-type`, `filename`, and Type args of a CKBFS cell CAN NOT be updated in ANY condition
+- Rule 13: in an append operation, Output CKBFS Cell’s `indexes` can not be `empty`
+
+---
+
+### Transfer
+
+Transfer operation transfers ownership of a CKBFS cell, and ensure it did not lost tracking of backlinks.
+
+```yaml
+// Transfer
+Witnesses:
+  <...>
+Inputs:
+  <...>
+  CKBFS_CELL
+    Data:
+      content-type: string
+      filename: string
+      index: uint32
+      checksum: uint32
+      backlinks: empty_vec
+    Type:
+      code_hash: ckbfs type script
+      args: 32 bytes type_id, (...)
+    Lock:
+      <USER_DEFINED>
+  <...>
+Outputs:
+  <...>
+  CKBFS_CELL:
+    Data:
+      content-type: string
+      filename: string
+      index: null
+      checksum: uint32 # updated checksum
+      backlinks: vec<BackLink>
+    Type:
+      code_hash: ckbfs type script
+      args: 32 bytes type_id, (...)
+    Lock:
+      <USER_DEFINED>
+```
+
+- Rule 14: in a transfer operation, Output CKBFS Cell’s `indexes` must be empty
+- Rule 15: if Input CKBFS Cell’s backlinks is empty, then output’s backlink should be append following Rule 10. Otherwise, the backlinks should not be updated
+- Rule 16: in a transfer operation, `checksum` CAN NOT be updated
+
+---
+
+## Other Notes
+
+### Molecule Definitions:
+
+Here’s molecule definitions of CKBFS data structures
+
+```jsx
+vector Bytes <byte>;
+option BytesOpt (Bytes);
+option Uint32Opt (Uint32);
+vector Indexes <index>;
+
+table BackLink {
+  tx_hash: Bytes,
+  index: Indexes,
+  checksum: Uint32,
+}
+
+vector BackLinks <BackLink>;
+
+table CKBFSData {
+  index: Indexes,
+  checksum: Uint32,
+  content_type: Bytes,
+  filename: Bytes,
+  backlinks: BackLinks,
+}
+```
+
+### Checksum Validator Procedure:
+
+Bellow is pseudocodes shows how one can validates the checksum:
+
+```pascal
+function validate_checksum(witness, expected_checksum, backlinks);
+var
+  hasher: thasher;
+  computed_checksum: uint32;
+  content_bytes: bytes;
+  last_backlink: backlink;
+begin
+  // If backlinks is not empty, recover hasher state from the last backlink's checksum
+  if length(backlinks) > 0 then
+  begin
+    last_backlink := backlinks[length(backlinks) - 1];
+    hasher.recover(last_backlink.checksum);
+  end;
+
+  // Extract the content bytes from the witness starting from the 7th byte
+  content_bytes := copy(witness, 7, length(witness) - 6);
+  
+  // Update the hasher with the content bytes
+  hasher.update(content_bytes);
+
+  // Finalize and compute the checksum
+  computed_checksum := hasher.finalize;
+  
+  // Compare the computed checksum with the expected checksum
+  if computed_checksum = expected_checksum then
+    validate_checksum := true
+  else
+    validate_checksum := false;
+end;
+
+```
+
+### Advanced Usage - Branch Forking File Appendix
+
+Assuming that we have created a CKBFS Cell:
+
+```yaml
+CKBFS_CELL:
+  Data:
+    content-type: string
+    filename: string
+    indexes: [0x0]
+    checksum: 0xFE02EA11
+    backlinks: [BACKLINK_1, BACKLINK_2, ...]
+  Type:
+    code_hash: CKBFS_CODE_HASH
+    args: TYPE_ID_A
+  Lock:
+    <USER_DEFINED>
+```
+
+It is able to creating a forking of this CKBFS by a special publish, similar to append but put the referenced CKBFS Cell in CellDeps:
+
+```yaml
+CellDeps:
+  <...>
+  CKBFS_CELL:
+	  Data:
+	    content-type: string
+	    filename: string
+	    indexes: [0x0]
+	    checksum: 0xFE02EA11
+	    backlinks: [BACKLINK_1, BACKLINK_2, ...]
+	  Type:
+	    code_hash: CKBFS_CODE_HASH
+	    args: TYPE_ID_A
+	  Lock:
+	    <USER_DEFINED>
+	<...>
+Witnesses:
+  <...>
+  <CKBFS_CONTENT_BYTES>
+  <...>
+Inputs:
+  <...>
+Outputs:
+  <...>
+  CKBFS_CELL
+    Data:
+      content-type: string
+      filename: string
+      indexes: [uint32]
+      checksum: UPDATED_CHECKSUM
+      backlinks: [BACKLINK_1, BACKLINK_2, ...]
+    Type:
+      code_hash: ckbfs type script
+      args: TYPE_ID_B
+  <...>
+```
+
+And we are able to create a variant versions from a same reference data, allowing us to achieve something like git branching, shared-header data, etc.

package/append.txt

@@ -0,0 +1 @@
+This is content to append to the previously published file.

package/example.txt

@@ -0,0 +1 @@
+This is a sample file for testing CKBFS.

package/examples/append.ts

@@ -0,0 +1,222 @@
+import { CKBFS, NetworkType, ProtocolVersion, CKBFSDataType, extractCKBFSWitnessContent, isCKBFSWitness, CKBFSData } from '../src/index';
+import { Script, ClientPublicTestnet, Transaction } from "@ckb-ccc/core";
+
+// Replace with your actual private key
+const privateKey = process.env.CKB_PRIVATE_KEY || 'your-private-key-here';
+
+// Parse command line arguments for transaction hash
+const txHashArg = process.argv.find(arg => arg.startsWith('--txhash='));
+const publishTxHash = txHashArg ? txHashArg.split('=')[1] : process.env.PUBLISH_TX_HASH || '0x0000000000000000000000000000000000000000000000000000000000000000';
+
+/**
+ * Ensures a string is prefixed with '0x'
+ * @param value The string to ensure is hex prefixed
+ * @returns A hex prefixed string
+ */
+function ensureHexPrefix(value: string): `0x${string}` {
+  if (value.startsWith('0x')) {
+    return value as `0x${string}`;
+  }
+  return `0x${value}` as `0x${string}`;
+}
+
+// Initialize the SDK with network and version options
+const ckbfs = new CKBFS(
+  privateKey,
+  NetworkType.Testnet, // Use testnet
+  {
+    version: ProtocolVersion.V2, // Use the latest version (V2)
+    chunkSize: 20 * 1024, // 20KB chunks
+    useTypeID: false // Use code hash instead of type ID
+  }
+);
+
+// Initialize CKB client for testnet
+const client = new ClientPublicTestnet();
+
+/**
+ * Get cell information from a transaction using CKB client
+ * @param txHash The transaction hash to get cell information from
+ * @returns Promise resolving to the cell information
+ */
+async function getCellInfoFromTransaction(txHash: string): Promise<{ 
+  outPoint: { txHash: string; index: number };
+  type: Script; 
+  data: CKBFSDataType;
+  lock: Script;
+  capacity: bigint;
+}> {
+  console.log(`Retrieving transaction data for: ${txHash}`);
+  
+  try {
+    // Get transaction from RPC
+    const txWithStatus = await client.getTransaction(txHash);
+    if (!txWithStatus || !txWithStatus.transaction) {
+      throw new Error(`Transaction ${txHash} not found`);
+    }
+    
+    const tx = Transaction.from(txWithStatus.transaction);
+    console.log(`Transaction found with ${tx.outputs.length} outputs`);
+    
+    // Find the CKBFS cell output (first output with type script)
+    let ckbfsCellIndex = 0;
+    const output = tx.outputs[ckbfsCellIndex];
+    if (!output || !output.type) {
+      throw new Error('No CKBFS cell found in the transaction');
+    }
+    
+    console.log(`Found CKBFS cell at index ${ckbfsCellIndex}`);
+    console.log(`Cell type script hash: ${output.type.hash()}`);
+    
+    // Get output data
+    const outputData = tx.outputsData[ckbfsCellIndex];
+    if (!outputData) {
+      throw new Error('Output data not found');
+    }
+    
+    // Parse the output data as CKBFS data
+    // First remove 0x prefix if present
+    const rawData = outputData.startsWith('0x') 
+      ? Buffer.from(outputData.slice(2), 'hex') 
+      : Buffer.from(outputData, 'hex');
+      
+    // For demonstration purposes, we'll manually create a CKBFSDataType object
+    // In a real app, you would properly decode the rawData from molecule format
+    const sampleDataFields = {
+      index: [1],
+      checksum: 12345,
+      contentType: new TextEncoder().encode('text/plain'),
+      filename: new TextEncoder().encode('example.txt'),
+      backLinks: []
+    };
+    
+    // Use this as our data
+    const ckbfsData: CKBFSDataType = sampleDataFields;
+    
+    console.log(`CKBFS data processed`);
+    console.log(`Using filename: ${new TextDecoder().decode(ckbfsData.filename)}`);
+    console.log(`Using content type: ${new TextDecoder().decode(ckbfsData.contentType)}`);
+    
+    return {
+      outPoint: {
+        txHash,
+        index: ckbfsCellIndex
+      },
+      type: output.type,
+      lock: output.lock,
+      capacity: output.capacity,
+      data: ckbfsData
+    };
+  } catch (error) {
+    console.error('Error retrieving transaction data:', error);
+    
+    // Fallback to simulated data for demonstration purposes
+    console.warn('FALLBACK: Using simulated data for demonstration');
+    
+    // Get lock script for this account
+    const lock = await ckbfs.getLock();
+    
+    // Get CKBFS script config
+    const config = ckbfs.getCKBFSConfig();
+    
+    // Create sample data for demonstration
+    const sampleData: CKBFSDataType = {
+      index: [1],
+      checksum: 12345, // Sample checksum number
+      contentType: new TextEncoder().encode('text/plain'),
+      filename: new TextEncoder().encode('example.txt'),
+      backLinks: []
+    };
+    
+    // Create the type script
+    const typeArgs = '0x3cc03661013140855e756c032ce83bc270a7ca3f1f3b76ec21a8ea0155ac3a7c';
+    const typeScript = new Script(
+      ensureHexPrefix(config.codeHash),
+      config.hashType as any,
+      typeArgs
+    );
+    
+    return {
+      outPoint: {
+        txHash,
+        index: 0
+      },
+      type: typeScript,
+      lock,
+      capacity: 200n * 100000000n,
+      data: sampleData
+    };
+  }
+}
+
+/**
+ * Example of appending content to an existing CKBFS file
+ */
+async function appendExample() {
+  try {
+    // Validate transaction hash
+    if (publishTxHash === '0x0000000000000000000000000000000000000000000000000000000000000000') {
+      throw new Error('Please provide a valid transaction hash by setting the PUBLISH_TX_HASH environment variable or using --txhash=0x... argument');
+    }
+    
+    // Get address info
+    const address = await ckbfs.getAddress();
+    console.log(`Using address: ${address.toString()}`);
+    
+    // Get lock script
+    const lock = await ckbfs.getLock();
+    
+    // Get the cell information from the transaction
+    console.log(`Getting cell info from transaction: ${publishTxHash}`);
+    const ckbfsCell = await getCellInfoFromTransaction(publishTxHash);
+    
+    // Append content from a file
+    const filePath = './append.txt';
+    console.log(`Appending file: ${filePath}`);
+    
+    // Create a sample append.txt file for the demonstration
+    const fs = require('fs');
+    if (!fs.existsSync(filePath)) {
+      fs.writeFileSync(filePath, 'This is content to append to the previously published file.');
+      console.log(`Created sample file: ${filePath}`);
+    }
+    
+    console.log('Note: When appending data, the capacity of the cell may need to increase.');
+    console.log('The SDK will automatically handle this by adding additional inputs if needed.');
+    console.log('Make sure your account has enough CKB to cover the increased capacity and fees.');
+    
+    const txHash = await ckbfs.appendFile(filePath, ckbfsCell);
+    
+    console.log(`File appended successfully!`);
+    console.log(`Transaction Hash: ${txHash}`);
+    console.log(`View at: https://pudge.explorer.nervos.org/transaction/${txHash}`);
+    
+    return txHash;
+  } catch (error) {
+    console.error('Error appending file:', error);
+    throw error;
+  }
+}
+
+/**
+ * Main function to run the example
+ */
+async function main() {
+  console.log('Running CKBFS append example...');
+  console.log('-------------------------------');
+  console.log(`Using CKBFS protocol version: ${ProtocolVersion.V2}`);
+  
+  try {
+    await appendExample();
+    console.log('Example completed successfully!');
+    process.exit(0);
+  } catch (error) {
+    console.error('Example failed:', error);
+    process.exit(1);
+  }
+}
+
+// Run the example if this file is executed directly
+if (require.main === module) {
+  main().catch(console.error);
+}