hippius 0.1.0__tar.gz → 0.1.6__tar.gz

{hippius-0.1.0 → hippius-0.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: hippius
-Version: 0.1.0
+Version: 0.1.6
 Summary: Python SDK and CLI for Hippius blockchain storage
 Home-page: https://github.com/thenervelab/hippius-sdk
 Author: Dubs
@@ -23,6 +23,7 @@ Requires-Dist: pyperclip (>=1.8.2,<2.0.0) ; extra == "clipboard"
 Requires-Dist: python-dotenv (>=1.0.0,<2.0.0)
 Requires-Dist: requests (>=2.28.1,<3.0.0)
 Requires-Dist: substrate-interface (>=1.4.2,<2.0.0)
+Requires-Dist: zfec (>=1.5.3,<2.0.0)
 Project-URL: Documentation, https://github.com/thenervelab/hippius-sdk/docs
 Project-URL: Repository, https://github.com/thenervelab/hippius-sdk
 Description-Content-Type: text/markdown
@@ -172,6 +173,104 @@ raw_result = client.download_file(encrypted_result['cid'], "still_encrypted.txt"
 content = client.cat(encrypted_result['cid'], decrypt=True)
 ```
 
+## Erasure Coding
+
+Hippius SDK supports Reed-Solomon erasure coding for reliable and resilient file storage. This allows files to be split into chunks with added redundancy, so that the original file can be reconstructed even if some chunks are lost.
+
+### Erasure Coding Concepts
+
+- **k**: The number of data chunks needed to reconstruct the original file
+- **m**: The total number of chunks created (m > k)
+- The file can be reconstructed from any k chunks out of m total chunks
+- Higher redundancy (m-k) provides better protection against chunk loss
+
+### Using Erasure Coding
+
+```python
+from hippius_sdk import HippiusClient
+
+client = HippiusClient()
+
+# Erasure code a file with default parameters (k=3, m=5)
+result = client.erasure_code_file("large_file.mp4")
+metadata_cid = result["metadata_cid"]
+
+# Use custom parameters for more redundancy
+result = client.erasure_code_file(
+    file_path="important_data.zip",
+    k=4,               # Need 4 chunks to reconstruct
+    m=10,              # Create 10 chunks total (6 redundant)
+    chunk_size=2097152,  # 2MB chunks
+    encrypt=True       # Encrypt before splitting
+)
+
+# Store erasure-coded file in Hippius marketplace
+result = client.store_erasure_coded_file(
+    file_path="critical_backup.tar",
+    k=3,
+    m=5,
+    encrypt=True,
+    miner_ids=["miner1", "miner2", "miner3"]
+)
+
+# Reconstruct a file from its metadata
+reconstructed_path = client.reconstruct_from_erasure_code(
+    metadata_cid=metadata_cid,
+    output_file="reconstructed_file.mp4"
+)
+```
+
+### When to Use Erasure Coding
+
+Erasure coding is particularly useful for:
+
+- Large files where reliability is critical
+- Long-term archival storage
+- Data that must survive partial network failures
+- Situations where higher redundancy is needed without full replication
+
+### Advanced Features
+
+#### Small File Handling
+
+The SDK automatically adjusts parameters for small files:
+
+- If a file is too small to be split into `k` chunks, the SDK will adjust the chunk size
+- For very small files, the content is split into exactly `k` sub-blocks
+- Parameters are always optimized to provide the requested level of redundancy
+
+#### Robust Storage in Marketplace
+
+When using `store_erasure_coded_file`, the SDK now:
+
+- Stores both the metadata file AND all encoded chunks in the marketplace
+- Ensures miners can access all necessary data for redundancy and retrieval
+- Reports total number of files stored for verification
+
+#### CLI Commands
+
+The CLI provides powerful commands for erasure coding:
+
+```bash
+# Basic usage with automatic parameter adjustment
+hippius erasure-code myfile.txt
+
+# Specify custom parameters
+hippius erasure-code large_video.mp4 --k 4 --m 8 --chunk-size 4194304
+
+# For smaller files, using smaller parameters
+hippius erasure-code small_doc.txt --k 2 --m 5 --chunk-size 4096
+
+# Reconstruct a file from its metadata CID
+hippius reconstruct QmMetadataCID reconstructed_file.mp4
+```
+
+The CLI provides detailed output during the process, including:
+- Automatic parameter adjustments for optimal encoding
+- Progress of chunk creation and upload
+- Storage confirmation in the marketplace
+- Instructions for reconstruction
+
 ## Command Line Interface
 
 The Hippius SDK includes a powerful command-line interface (CLI) that provides access to all major features of the SDK directly from your terminal.
@@ -240,6 +339,19 @@ hippius store my_file.txt --encrypt
 hippius download QmCID123 output_file.txt --decrypt
 ```
 
+### Erasure Coding
+
+```bash
+# Erasure code a file with default parameters (k=3, m=5)
+hippius erasure-code large_file.mp4
+
+# Erasure code with custom parameters
+hippius erasure-code important_data.zip --k 4 --m 10 --chunk-size 2097152 --encrypt
+
+# Reconstruct a file from its metadata
+hippius reconstruct QmMetadataCID reconstructed_file.mp4
+```
+
 ### Using Environment Variables
 
 The CLI automatically reads from your `.env` file for common settings:

{hippius-0.1.0 → hippius-0.1.6}/README.md

@@ -143,6 +143,104 @@ raw_result = client.download_file(encrypted_result['cid'], "still_encrypted.txt"
 content = client.cat(encrypted_result['cid'], decrypt=True)
 ```
 
+## Erasure Coding
+
+Hippius SDK supports Reed-Solomon erasure coding for reliable and resilient file storage. This allows files to be split into chunks with added redundancy, so that the original file can be reconstructed even if some chunks are lost.
+
+### Erasure Coding Concepts
+
+- **k**: The number of data chunks needed to reconstruct the original file
+- **m**: The total number of chunks created (m > k)
+- The file can be reconstructed from any k chunks out of m total chunks
+- Higher redundancy (m-k) provides better protection against chunk loss
+
+### Using Erasure Coding
+
+```python
+from hippius_sdk import HippiusClient
+
+client = HippiusClient()
+
+# Erasure code a file with default parameters (k=3, m=5)
+result = client.erasure_code_file("large_file.mp4")
+metadata_cid = result["metadata_cid"]
+
+# Use custom parameters for more redundancy
+result = client.erasure_code_file(
+    file_path="important_data.zip",
+    k=4,               # Need 4 chunks to reconstruct
+    m=10,              # Create 10 chunks total (6 redundant)
+    chunk_size=2097152,  # 2MB chunks
+    encrypt=True       # Encrypt before splitting
+)
+
+# Store erasure-coded file in Hippius marketplace
+result = client.store_erasure_coded_file(
+    file_path="critical_backup.tar",
+    k=3,
+    m=5,
+    encrypt=True,
+    miner_ids=["miner1", "miner2", "miner3"]
+)
+
+# Reconstruct a file from its metadata
+reconstructed_path = client.reconstruct_from_erasure_code(
+    metadata_cid=metadata_cid,
+    output_file="reconstructed_file.mp4"
+)
+```
+
+### When to Use Erasure Coding
+
+Erasure coding is particularly useful for:
+
+- Large files where reliability is critical
+- Long-term archival storage
+- Data that must survive partial network failures
+- Situations where higher redundancy is needed without full replication
+
+### Advanced Features
+
+#### Small File Handling
+
+The SDK automatically adjusts parameters for small files:
+
+- If a file is too small to be split into `k` chunks, the SDK will adjust the chunk size
+- For very small files, the content is split into exactly `k` sub-blocks
+- Parameters are always optimized to provide the requested level of redundancy
+
+#### Robust Storage in Marketplace
+
+When using `store_erasure_coded_file`, the SDK now:
+
+- Stores both the metadata file AND all encoded chunks in the marketplace
+- Ensures miners can access all necessary data for redundancy and retrieval
+- Reports total number of files stored for verification
+
+#### CLI Commands
+
+The CLI provides powerful commands for erasure coding:
+
+```bash
+# Basic usage with automatic parameter adjustment
+hippius erasure-code myfile.txt
+
+# Specify custom parameters
+hippius erasure-code large_video.mp4 --k 4 --m 8 --chunk-size 4194304
+
+# For smaller files, using smaller parameters
+hippius erasure-code small_doc.txt --k 2 --m 5 --chunk-size 4096
+
+# Reconstruct a file from its metadata CID
+hippius reconstruct QmMetadataCID reconstructed_file.mp4
+```
+
+The CLI provides detailed output during the process, including:
+- Automatic parameter adjustments for optimal encoding
+- Progress of chunk creation and upload
+- Storage confirmation in the marketplace
+- Instructions for reconstruction
+
 ## Command Line Interface
 
 The Hippius SDK includes a powerful command-line interface (CLI) that provides access to all major features of the SDK directly from your terminal.
@@ -211,6 +309,19 @@ hippius store my_file.txt --encrypt
 hippius download QmCID123 output_file.txt --decrypt
 ```
 
+### Erasure Coding
+
+```bash
+# Erasure code a file with default parameters (k=3, m=5)
+hippius erasure-code large_file.mp4
+
+# Erasure code with custom parameters
+hippius erasure-code important_data.zip --k 4 --m 10 --chunk-size 2097152 --encrypt
+
+# Reconstruct a file from its metadata
+hippius reconstruct QmMetadataCID reconstructed_file.mp4
+```
+
 ### Using Environment Variables
 
 The CLI automatically reads from your `.env` file for common settings:

{hippius-0.1.0 → hippius-0.1.6}/hippius_sdk/cli.py

@@ -435,25 +435,202 @@ def handle_files(client, account_address, debug=False, show_all_miners=False):
     return 0
 
 
+def handle_erasure_code(
+    client, file_path, k, m, chunk_size, miner_ids, encrypt=None, verbose=True
+):
+    """Handle the erasure-code command"""
+    if not os.path.exists(file_path):
+        print(f"Error: File {file_path} not found")
+        return 1
+
+    # Check if zfec is installed
+    try:
+        import zfec
+    except ImportError:
+        print(
+            "Error: zfec is required for erasure coding. Install it with: pip install zfec"
+        )
+        print("Then update your environment: poetry add zfec")
+        return 1
+
+    # Parse miner IDs if provided
+    miner_id_list = None
+    if miner_ids:
+        miner_id_list = [m.strip() for m in miner_ids.split(",") if m.strip()]
+        if verbose:
+            print(f"Targeting {len(miner_id_list)} miners: {', '.join(miner_id_list)}")
+
+    # Get the file size and adjust parameters if needed
+    file_size = os.path.getsize(file_path)
+    file_size_mb = file_size / (1024 * 1024)
+    
+    print(f"Processing {file_path} ({file_size_mb:.2f} MB) with erasure coding...")
+    
+    # Check if the file is too small for the current chunk size and k value
+    original_k = k
+    original_m = m
+    original_chunk_size = chunk_size
+    
+    # Calculate how many chunks we would get with current settings
+    potential_chunks = max(1, file_size // chunk_size)
+    
+    # If we can't get at least k chunks, adjust the chunk size
+    if potential_chunks < k:
+        # Calculate a new chunk size that would give us exactly k chunks
+        new_chunk_size = max(1024, file_size // k)  # Ensure at least 1KB chunks
+        
+        print(f"Warning: File is too small for the requested parameters.")
+        print(f"Original parameters: k={k}, m={m}, chunk size={chunk_size/1024/1024:.2f} MB")
+        print(f"Would create only {potential_chunks} chunks, which is less than k={k}")
+        print(f"Automatically adjusting chunk size to {new_chunk_size/1024/1024:.6f} MB to create at least {k} chunks")
+        
+        chunk_size = new_chunk_size
+    
+    print(f"Final parameters: k={k}, m={m} (need {k} of {m} chunks to reconstruct)")
+    print(f"Chunk size: {chunk_size/1024/1024:.6f} MB")
+
+    if encrypt:
+        print("Encryption: Enabled")
+
+    start_time = time.time()
+
+    try:
+        # Use the store_erasure_coded_file method directly from HippiusClient
+        result = client.store_erasure_coded_file(
+            file_path=file_path,
+            k=k,
+            m=m,
+            chunk_size=chunk_size,
+            encrypt=encrypt,
+            miner_ids=miner_id_list,
+            max_retries=3,
+            verbose=verbose,
+        )
+
+        elapsed_time = time.time() - start_time
+
+        print(f"\nErasure coding and storage completed in {elapsed_time:.2f} seconds!")
+
+        # Display metadata
+        metadata = result.get("metadata", {})
+        metadata_cid = result.get("metadata_cid", "unknown")
+        total_files_stored = result.get("total_files_stored", 0)
+
+        original_file = metadata.get("original_file", {})
+        erasure_coding = metadata.get("erasure_coding", {})
+
+        print("\nErasure Coding Summary:")
+        print(
+            f"  Original file: {original_file.get('name')} ({original_file.get('size', 0)/1024/1024:.2f} MB)"
+        )
+        print(f"  File ID: {erasure_coding.get('file_id')}")
+        print(f"  Parameters: k={erasure_coding.get('k')}, m={erasure_coding.get('m')}")
+        print(f"  Total chunks: {len(metadata.get('chunks', []))}")
+        print(f"  Total files stored in marketplace: {total_files_stored}")
+        print(f"  Metadata CID: {metadata_cid}")
+
+        # If we stored in the marketplace
+        if "transaction_hash" in result:
+            print(
+                f"\nStored in marketplace. Transaction hash: {result['transaction_hash']}"
+            )
+
+        # Instructions for reconstruction
+        print("\nTo reconstruct this file, you will need:")
+        print(f"  1. The metadata CID: {metadata_cid}")
+        print("  2. Access to at least k chunks for each original chunk")
+        print("\nReconstruction command:")
+        print(
+            f"  hippius reconstruct {metadata_cid} reconstructed_{original_file.get('name')}"
+        )
+
+        return 0
+
+    except Exception as e:
+        print(f"Error during erasure coding: {e}")
+        
+        # Provide helpful advice based on the error
+        if "Wrong length" in str(e) and "input blocks" in str(e):
+            print("\nThis error typically occurs with very small files.")
+            print("Suggestions:")
+            print("  1. Try using a smaller chunk size: --chunk-size 4096")
+            print("  2. Try using a smaller k value: --k 2")
+            print("  3. For very small files, consider using regular storage instead of erasure coding.")
+        
+        return 1
+
+
+def handle_reconstruct(client, metadata_cid, output_file, verbose=True):
+    """Handle the reconstruct command for erasure-coded files"""
+    # Check if zfec is installed
+    try:
+        import zfec
+    except ImportError:
+        print(
+            "Error: zfec is required for erasure coding. Install it with: pip install zfec"
+        )
+        print("Then update your environment: poetry add zfec")
+        return 1
+
+    print(f"Reconstructing file from metadata CID: {metadata_cid}")
+    print(f"Output file: {output_file}")
+
+    start_time = time.time()
+
+    try:
+        # Use the reconstruct_from_erasure_code method
+        result = client.reconstruct_from_erasure_code(
+            metadata_cid=metadata_cid, output_file=output_file, verbose=verbose
+        )
+
+        elapsed_time = time.time() - start_time
+        print(f"\nFile reconstruction completed in {elapsed_time:.2f} seconds!")
+        print(f"Reconstructed file saved to: {result}")
+
+        return 0
+
+    except Exception as e:
+        print(f"Error during file reconstruction: {e}")
+        return 1
+
+
 def main():
-    """Main entry point for the Hippius CLI."""
+    """Main CLI entry point for hippius command."""
+    # Set up the argument parser
     parser = argparse.ArgumentParser(
         description="Hippius SDK Command Line Interface",
         formatter_class=argparse.RawDescriptionHelpFormatter,
         epilog="""
-Examples:
-  hippius download QmCID123 downloaded_file.txt
-  hippius exists QmCID123
-  hippius cat QmCID123
-  hippius store test_file.txt
-  hippius store-dir ./test_directory
+examples:
+  # Store a file
+  hippius store example.txt
+  
+  # Store a directory
+  hippius store-dir ./my_directory
+  
+  # Download a file
+  hippius download QmHash output.txt
+  
+  # Check if a CID exists
+  hippius exists QmHash
+  
+  # View the content of a CID
+  hippius cat QmHash
+  
+  # View your available credits
   hippius credits
-  hippius credits 5H1QBRF7T7dgKwzVGCgS4wioudvMRf9K4NEDzfuKLnuyBNzH
+  
+  # View your stored files
   hippius files
-  hippius files 5H1QBRF7T7dgKwzVGCgS4wioudvMRf9K4NEDzfuKLnuyBNzH
+  
+  # View all miners for stored files
   hippius files --all-miners
-  hippius keygen
-  hippius keygen --copy
+  
+  # Erasure code a file (Reed-Solomon)
+  hippius erasure-code large_file.mp4 --k 3 --m 5
+  
+  # Reconstruct an erasure-coded file
+  hippius reconstruct QmMetadataHash reconstructed_file.mp4
 """,
     )
 
@@ -588,6 +765,53 @@ Examples:
         "--copy", action="store_true", help="Copy the generated key to the clipboard"
     )
 
+    # Erasure code command
+    erasure_code_parser = subparsers.add_parser(
+        "erasure-code", help="Erasure code a file"
+    )
+    erasure_code_parser.add_argument("file_path", help="Path to file to erasure code")
+    erasure_code_parser.add_argument(
+        "--k",
+        type=int,
+        default=3,
+        help="Number of data chunks needed to reconstruct (default: 3)",
+    )
+    erasure_code_parser.add_argument(
+        "--m", type=int, default=5, help="Total number of chunks to create (default: 5)"
+    )
+    erasure_code_parser.add_argument(
+        "--chunk-size",
+        type=int,
+        default=1048576,
+        help="Chunk size in bytes (default: 1MB)",
+    )
+    erasure_code_parser.add_argument(
+        "--miner-ids", help="Comma-separated list of miner IDs"
+    )
+    erasure_code_parser.add_argument(
+        "--encrypt", action="store_true", help="Encrypt the file"
+    )
+    erasure_code_parser.add_argument(
+        "--no-encrypt", action="store_true", help="Do not encrypt the file"
+    )
+    erasure_code_parser.add_argument(
+        "--verbose", action="store_true", help="Enable verbose output", default=True
+    )
+
+    # Reconstruct command
+    reconstruct_parser = subparsers.add_parser(
+        "reconstruct", help="Reconstruct an erasure-coded file"
+    )
+    reconstruct_parser.add_argument(
+        "metadata_cid", help="Metadata CID of the erasure-coded file"
+    )
+    reconstruct_parser.add_argument(
+        "output_file", help="Path to save reconstructed file"
+    )
+    reconstruct_parser.add_argument(
+        "--verbose", action="store_true", help="Enable verbose output", default=True
+    )
+
     args = parser.parse_args()
 
     if not args.command:
@@ -647,6 +871,23 @@ Examples:
                 else False,
             )
 
+        elif args.command == "erasure-code":
+            return handle_erasure_code(
+                client,
+                args.file_path,
+                args.k,
+                args.m,
+                args.chunk_size,
+                miner_ids,
+                encrypt=args.encrypt,
+                verbose=args.verbose,
+            )
+
+        elif args.command == "reconstruct":
+            return handle_reconstruct(
+                client, args.metadata_cid, args.output_file, verbose=args.verbose
+            )
+
     except Exception as e:
         print(f"Error: {e}")
         return 1