solana-docs 0.0.0

package/src/proposals/program-instruction-macro.md

@@ -0,0 +1,226 @@
+# Program Instruction Macro
+
+## Problem
+
+Currently, inspecting an on-chain transaction requires depending on a
+client-side, language-specific decoding library to parse the instruction. If
+rpc methods could return decoded instruction details, these custom solutions
+would be unnecessary.
+
+We can deserialize instruction data using a program's Instruction enum, but
+decoding the account-key list into human-readable identifiers requires manual
+parsing. Our current Instruction enums have that account information, but only
+in variant docs.
+
+Similarly, we have instruction constructor functions that duplicate nearly all
+the information in the enum, but we can't generate that constructor from the
+enum definition because the list of account references is in code comments.
+
+Also, Instruction docs can vary between implementations, as there is no
+mechanism to ensure consistency.
+
+## Proposed Solution
+
+Move the data from code comments to attributes, such that the constructors
+can be generated, and include all the documentation from the enum definition.
+
+Here is an example of an Instruction enum using the new accounts format:
+
+```rust,ignore
+#[instructions(test_program::id())]
+pub enum TestInstruction {
+    /// Transfer lamports
+    #[accounts(
+        from_account(SIGNER, WRITABLE, desc = "Funding account"),
+        to_account(WRITABLE, desc = "Recipient account"),
+    )]
+    Transfer {
+        lamports: u64,
+    },
+
+    /// Provide M of N required signatures
+    #[accounts(
+        data_account(WRITABLE, desc = "Data account"),
+        signers(SIGNER, multiple, desc = "Signer"),
+    )]
+    Multisig,
+
+    /// Consumes a stored nonce, replacing it with a successor
+    #[accounts(
+        nonce_account(SIGNER, WRITABLE, desc = "Nonce account"),
+        recent_blockhashes_sysvar(desc = "RecentBlockhashes sysvar"),
+        nonce_authority(SIGNER, optional, desc = "Nonce authority"),
+    )]
+    AdvanceNonceAccount,
+}
+```
+
+An example of the generated TestInstruction with docs:
+
+```rust,ignore
+pub enum TestInstruction {
+    /// Transfer lamports
+    ///
+    /// * Accounts expected by this instruction:
+    ///   0. `[WRITABLE, SIGNER]` Funding account
+    ///   1. `[WRITABLE]` Recipient account
+    Transfer {
+        lamports: u64,
+    },
+
+    /// Provide M of N required signatures
+    ///
+    /// * Accounts expected by this instruction:
+    ///   0. `[WRITABLE]` Data account
+    ///   * (Multiple) `[SIGNER]` Signers
+    Multisig,
+
+    /// Consumes a stored nonce, replacing it with a successor
+    ///
+    /// * Accounts expected by this instruction:
+    ///   0. `[WRITABLE, SIGNER]` Nonce account
+    ///   1. `[]` RecentBlockhashes sysvar
+    ///   2. (Optional) `[SIGNER]` Nonce authority
+    AdvanceNonceAccount,
+}
+```
+
+Generated constructors:
+
+```rust,ignore
+/// Transfer lamports
+///
+/// * `from_account` - `[WRITABLE, SIGNER]` Funding account
+/// * `to_account` - `[WRITABLE]` Recipient account
+pub fn transfer(from_account: Pubkey, to_account: Pubkey, lamports: u64) -> Instruction {
+    let account_metas = vec![
+        AccountMeta::new(from_pubkey, true),
+        AccountMeta::new(to_pubkey, false),
+    ];
+    Instruction::new_with_bincode(
+        test_program::id(),
+        &SystemInstruction::Transfer { lamports },
+        account_metas,
+    )
+}
+
+/// Provide M of N required signatures
+///
+/// * `data_account` - `[WRITABLE]` Data account
+/// * `signers` - (Multiple) `[SIGNER]` Signers
+pub fn multisig(data_account: Pubkey, signers: &[Pubkey]) -> Instruction {
+    let mut account_metas = vec![
+        AccountMeta::new(nonce_pubkey, false),
+    ];
+    for pubkey in signers.iter() {
+        account_metas.push(AccountMeta::new_readonly(pubkey, true));
+    }
+
+    Instruction::new_with_bincode(
+        test_program::id(),
+        &TestInstruction::Multisig,
+        account_metas,
+    )
+}
+
+/// Consumes a stored nonce, replacing it with a successor
+///
+/// * nonce_account - `[WRITABLE, SIGNER]` Nonce account
+/// * recent_blockhashes_sysvar - `[]` RecentBlockhashes sysvar
+/// * nonce_authority - (Optional) `[SIGNER]` Nonce authority
+pub fn advance_nonce_account(
+    nonce_account: Pubkey,
+    recent_blockhashes_sysvar: Pubkey,
+    nonce_authority: Option<Pubkey>,
+) -> Instruction {
+    let mut account_metas = vec![
+        AccountMeta::new(nonce_account, false),
+        AccountMeta::new_readonly(recent_blockhashes_sysvar, false),
+    ];
+    if let Some(pubkey) = authorized_pubkey {
+        account_metas.push(AccountMeta::new_readonly*nonce_authority, true));
+    }
+    Instruction::new_with_bincode(
+        test_program::id(),
+        &TestInstruction::AdvanceNonceAccount,
+        account_metas,
+    )
+}
+
+```
+
+Generated TestInstructionVerbose enum:
+
+```rust,ignore
+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]
+pub enum TestInstruction {
+    /// Transfer lamports
+    Transfer {
+        /// Funding account
+        funding_account: u8
+
+        /// Recipient account
+        recipient_account: u8
+
+        lamports: u64,
+    },
+
+    /// Provide M of N required signatures
+    Multisig {
+        data_account: u8,
+        signers: Vec<u8>,
+    },
+
+    /// Consumes a stored nonce, replacing it with a successor
+    AdvanceNonceAccount {
+        nonce_account: u8,
+        recent_blockhashes_sysvar: u8,
+        nonce_authority: Option<u8>,
+    }
+}
+
+impl TestInstructionVerbose {
+    pub fn from_instruction(instruction: TestInstruction, account_keys: Vec<u8>) -> Self {
+        match instruction {
+            TestInstruction::Transfer { lamports } => TestInstructionVerbose::Transfer {
+                funding_account: account_keys[0],
+                recipient_account: account_keys[1],
+                lamports,
+            }
+            TestInstruction::Multisig => TestInstructionVerbose::Multisig {
+                data_account: account_keys[0],
+                signers: account_keys[1..],
+            }
+            TestInstruction::AdvanceNonceAccount => TestInstructionVerbose::AdvanceNonceAccount {
+                nonce_account: account_keys[0],
+                recent_blockhashes_sysvar: account_keys[1],
+                nonce_authority: &account_keys.get(2),
+            }
+        }
+    }
+}
+
+```
+
+## Considerations
+
+1. **Named fields** - Since the resulting Verbose enum constructs variants with
+   named fields, any unnamed fields in the original Instruction variant will need
+   to have names generated. As such, it would be considerably more straightforward
+   if all Instruction enum fields are converted to named types, instead of unnamed
+   tuples. This seems worth doing anyway, adding more precision to the variants and
+   enabling real documentation (so developers don't have to do
+   [this](https://github.com/solana-labs/solana/blob/3aab13a1679ba2b7846d9ba39b04a52f2017d3e0/sdk/src/system_instruction.rs#L140)
+   This will cause a little churn in our current code base, but not a lot.
+2. **Variable account lists** - This approach offers a couple options for
+   variable account lists. First, optional accounts may be added and tagged with
+   the `optional` keyword. However, currently only one optional account is
+   supported per instruction. Additional data will need to be added to the
+   instruction to support multiples, enabling identification of which accounts are
+   present when some but not all are included. Second, accounts that share the same
+   features may be added as a set, tagged with the `multiple` keyword. Like
+   optional accounts, only one multiple account set is supported per instruction
+   (and optional and multiple may not coexist). More complex instructions that
+   cannot be accommodated by `optional` or `multiple`, requiring logic to figure
+   out account order/representation, should probably be made into separate
+   instructions.

package/src/proposals/return-data.md

@@ -0,0 +1,144 @@
+# Return data from BPF programs
+
+## Problem
+
+In the Solidity langauge it is permitted to return any number of values from a function,
+for example a variable length string can be returned:
+
+```
+function foo1() public returns (string) {
+    return "Hello, world!\n";
+}
+```
+
+Multiple values, arrays and structs are permitted too.
+
+```
+struct S {
+    int f1;
+    bool f2
+};
+
+function foo2() public returns (string, int[], S) {
+    return (a, b, c);
+}
+```
+
+All the return values are eth abi encoded to a variable-length byte array.
+
+On ethereum errors can be returned too:
+
+```
+function withdraw() public {
+    require(msg.sender == owner, "Permission denied");
+}
+
+function failure() public {
+    revert("I afraid I can't do that dave");
+}
+```
+These errors help the developer debug any issue they are having, and can
+also be caught in a Solidity `try` .. `catch` block. Outside of a `try` .. `catch`
+block, any of these would cause the transaction or rpc to fail.
+
+## Existing solution
+
+The existing solution that Solang uses, writes the return data to the callee account data.
+The caller's account cannot be used, since the callee may not be the same BPF program, so
+it will not have permission to write to the callee's account data.
+
+Another solution would be to have a single return data account which is passed
+around through CPI. Again this does not work for CPI as the callee may not have
+permission to write to it.
+
+The problem with this solution is:
+
+- It does not work for RPC calls
+- It is very racey; a client has to submit the Tx and then retrieve the account
+  data. This is not atomic so the return data can be overwritten by another transaction.
+
+## Requirements for Solution
+
+It must work for:
+
+- RPC: An RPC should be able to return any number of values without writing to account data
+- Transaction: An transaction should be able to return any number of values without needing to write them account data
+- CPI: The callee must "set" return value, and the caller must be able to retrieve it.
+
+## Review of other chains
+
+### Ethereum (EVM)
+
+The `RETURN` opcode allows a contract to set a buffer as a returndata. This opcode takes a pointer to memory and a size. The `REVERT` opcode works similarly but signals that the call failed, and all account data changes must be reverted.
+
+For CPI, the caller can retrieve the returned data of the callee using the `RETURNDATASIZE` opcode which returns the length, and the `RETURNDATACOPY` opcode, which takes a memory destination pointer, offset into the returndata, and a length argument.
+
+Ethereum stores the returndata in blocks.
+
+### Parity Substrate
+
+The return data can be set using the `seal_return(u32 flags, u32 pointer, u32 size)` syscall.
+- Flags can be 1 for revert, 0 for success (nothing else defined)
+- Function does not return
+
+CPI: The `seal_call()` syscall takes pointer to buffer and pointer to buffer size where return data goes
+ - There is a 32KB limit for return data.
+
+Parity Substrate does not write the return data to blocks.
+
+## Rejected Solution
+
+The concept of ephemeral accounts has been proposed a solution for this. This would
+certainly work for the CPI case, but this would not work RPC or Transaction case.
+
+## Proposed Solution
+
+The callee can set the return data using a new system call `sol_set_return_data(buf: *const u8, length: u64)`.
+There is a limit of 1024 bytes for the returndata. This function can be called multiple times, and
+will simply overwrite what was written in the last call.
+
+The return data can be retrieved with `sol_get_return_data(buf: *mut u8, length: u64, program_id: *mut Pubkey) -> u64`.
+This function copies the return buffer, and the program_id that set the return data, and
+returns the length of the return data, or `0` if no return data is set. In this case, program_id is not set.
+
+When an instruction calls `sol_invoke()`, the return data of the callee is copied into the return data
+of the current instruction. This means that any return data is automatically passed up the call stack,
+to the callee of the current instruction (or the RPC call).
+
+Note that `sol_invoke()` clears the returns data before invoking the callee, so that any return data from
+a previous invoke is not reused if the invoked fails to set a return data. For example:
+
+ - A invokes B
+ - Before entry to B, return data is cleared.0
+ - B sets some return data and returns
+ - A invokes C
+ - Before entry to C, return data is cleared.
+ - C does not set return data and returns
+ - A checks return data and finds it empty
+
+Another scenario to consider:
+
+ - A invokes B
+ - B invokes C
+ - C sets return data and returns
+ - B does not touch return data and returns
+ - A gets return data from C
+ - A does not touch return data
+ - Return data from transaction is what C set.
+
+The compute costs are calculated for getting and setting the return data using
+the syscalls.
+
+For a normal RPC or Transaction, the returndata is base64-encoded and stored along side the sol_log
+strings in the [stable log](https://github.com/solana-labs/solana/blob/95292841947763bdd47ef116b40fc34d0585bca8/sdk/src/process_instruction.rs#L275-L281).
+
+## Note on returning errors
+
+Solidity on Ethereum allows the contract to return an error in the return data. In this case, all
+the account data changes for the account should be reverted. On Solana, any non-zero exit code
+for a BPF prorgram means the entire transaction fails. We do not wish to support an error return
+by returning success and then returning an error in the return data. This would mean we would have
+to support reverting the account data changes; this too expensive both on the VM side and the BPF
+contract side.
+
+Errors will be reported via sol_log.

package/src/proposals/rip-curl.md

@@ -0,0 +1,56 @@
+# RiP Curl: low-latency, transaction-oriented RPC
+
+## Problem
+
+Solana's initial RPC implementation was created for the purpose of allowing
+users to confirm transactions that had just recently been sent to the cluster.
+It was designed with memory usage in mind such that any validator should be
+able to support the API without concern of DoS attacks.
+
+Later down the line, it became desirable to use that same API to support the
+Solana explorer. The original design only supported minutes of history, so we
+changed it to instead store transaction statuses in a local RocksDB instance
+and offer days of history. We then extended that to 6 months via BigTable.
+
+With each modification, the API became more suitable for applications serving
+static content and less appealing for transaction processing. The clients poll
+for transaction status instead of being notified, giving the false impression
+of higher confirmation times. Furthermore, what clients can poll for is
+limited, preventing them from making reasonable real-time decisions, such as
+recognizing a transaction is confirmed as soon as particular, known
+validators vote on it.
+
+## Proposed Solution
+
+A web-friendly, transaction-oriented, streaming API built around the
+validator's ReplayStage.
+
+Improved client experience:
+
+- Support connections directly from WebAssembly apps.
+- Clients can be notified of confirmation progress in real-time, including votes
+  and voter stake weight.
+- Clients can be notified when the heaviest fork changes, if it affects the
+  transactions confirmation count.
+
+Easier for validators to support:
+
+- Each validator supports some number of concurrent connections and otherwise
+  has no significant resource constraints.
+- Transaction status is never stored in memory and cannot be polled for.
+- Signatures are only stored in memory until the desired commitment level or
+  until the blockhash expires, which ever is later.
+
+How it works:
+
+1. The client connects to a validator using a reliable communication channel,
+   such as a web socket.
+2. The validator registers the signature with ReplayStage.
+3. The validator sends the transaction into the Gulf Stream and retries all
+   known forks until the blockhash expires (not until the transaction is
+   accepted on only the heaviest fork). If the blockhash expires, the
+   signature is unregistered, the client is notified, and connection is closed.
+4. As ReplayStage detects events affecting the transaction's status, it
+   notifies the client in real-time.
+5. After confirmation that the transaction is rooted (`CommitmentLevel::Max`),
+   the signature is unregistered and the server closes the upstream channel.

package/src/proposals/rust-clients.md

@@ -0,0 +1,54 @@
+---
+title: Rust Clients
+---
+
+## Problem
+
+High-level tests, such as bench-tps, are written in terms of the `Client`
+trait. When we execute these tests as part of the test suite, we use the
+low-level `BankClient` implementation. When we need to run the same test
+against a cluster, we use the `ThinClient` implementation. The problem with
+that approach is that it means the trait will continually expand to include new
+utility functions and all implementations of it need to add the new
+functionality. By separating the user-facing object from the trait that abstracts
+the network interface, we can expand the user-facing object to include all sorts
+of useful functionality, such as the "spinner" from RpcClient, without concern
+for needing to extend the trait and its implementations.
+
+## Proposed Solution
+
+Instead of implementing the `Client` trait, `ThinClient` should be constructed
+with an implementation of it. That way, all utility functions currently in the
+`Client` trait can move into `ThinClient`. `ThinClient` could then move into
+`solana-sdk` since all its network dependencies would be in the implementation
+of `Client`. We would then add a new implementation of `Client`, called
+`ClusterClient`, and that would live in the `solana-client` crate, where
+`ThinClient` currently resides.
+
+After this reorg, any code needing a client would be written in terms of
+`ThinClient`. In unit tests, the functionality would be invoked with
+`ThinClient<BankClient>`, whereas `main()` functions, benchmarks and
+integration tests would invoke it with `ThinClient<ClusterClient>`.
+
+If higher-level components require more functionality than what could be
+implemented by `BankClient`, it should be implemented by a second object
+that implements a second trait, following the same pattern described here.
+
+### Error Handling
+
+The `Client` should use the existing `TransportError` enum for errors, except
+that the `Custom(String)` field should be changed to `Custom(Box<dyn Error>)`.
+
+### Implementation Strategy
+
+1. Add new object to `solana-sdk`, `RpcClientTng`, where the `Tng` suffix is
+   temporary and stands for "The Next Generation"
+2. Initialize `RpcClientTng` with a `SyncClient` implementation.
+3. Add new object to `solana-sdk`, `ThinClientTng`; initialize it with
+   `RpcClientTng` and an `AsyncClient` implementation
+4. Move all unit-tests from `BankClient` to `ThinClientTng<BankClient>`
+5. Add `ClusterClient`
+6. Move `ThinClient` users to `ThinClientTng<ClusterClient>`
+7. Delete `ThinClient` and rename `ThinClientTng` to `ThinClient`
+8. Move `RpcClient` users to new `ThinClient<ClusterClient>`
+9. Delete `RpcClient` and rename `RpcClientTng` to `RpcClient`