solana-docs 0.0.0

package/src/proposals/transactions-v2.md

@@ -0,0 +1,334 @@
+# Transactions v2 - On-chain Address Lookup Tables
+
+## Problem
+
+Messages transmitted to Solana validators must not exceed the IPv6 MTU size to
+ensure fast and reliable network transmission of cluster info over UDP.
+Solana's networking stack uses a conservative MTU size of 1280 bytes which,
+after accounting for headers, leaves 1232 bytes for packet data like serialized
+transactions.
+
+Developers building applications on Solana must design their on-chain program
+interfaces within the above transaction size limit constraint. One common
+work-around is to store state temporarily on-chain and consume that state in
+later transactions. This is the approach used by the BPF loader program for
+deploying Solana programs.
+
+However, this workaround doesn't work well when developers compose many on-chain
+programs in a single atomic transaction. With more composition comes more
+account inputs, each of which takes up 32 bytes. There is currently no available
+workaround for increasing the number of accounts used in a single transaction
+since each transaction must list all accounts that it needs to properly lock
+accounts for parallel execution. Therefore the current cap is about 35 accounts
+after accounting for signatures and other transaction metadata.
+
+## Proposed Solution
+
+1) Introduce a new program which manages on-chain address lookup tables
+2) Add a new transaction format which can make use of on-chain
+address lookup tables to efficiently load more accounts in a single transaction.
+
+### Address Lookup Table Program
+
+Here we describe a program-based solution to the problem, whereby a protocol
+developer or end-user can create collections of related addresses on-chain for
+concise use in a transaction's account inputs.
+
+After addresses are stored on-chain in an address lookup table account, they may be
+succinctly referenced in a transaction using a 1-byte u8 index rather than a
+full 32-byte address. This will require a new transaction format to make use of
+these succinct references as well as runtime handling for looking up and loading
+addresses from the on-chain lookup tables.
+
+#### State
+
+Address lookup tables must be rent-exempt when initialized and after
+each time new addresses are appended. Lookup tables can either be extended
+from an on-chain buffered list of addresses or directly by appending
+addresses through instruction data. Newly appended addresses require
+one slot to warmup before being available to transactions for lookups.
+
+Since transactions use a `u8` index to look up addresses, address tables can
+store up to 256 addresses each. In addition to stored addresses, address table
+accounts also tracks various metadata explained below.
+
+```rust
+pub struct LookupTableMeta {
+    /// The slot used to derive the table's address. The table cannot
+    /// be closed until the derivation slot is no longer "recent"
+    /// (not accessible in the `SlotHashes` sysvar).
+    pub derivation_slot: Slot,
+    /// The slot that the table was last extended. Address tables may
+    /// only be used to lookup addresses that were extended before
+    /// the current bank's slot.
+    pub last_extended_slot: Slot,
+    /// The start index where the table was last extended from during
+    /// the `last_extended_slot`.
+    pub last_extended_slot_start_index: u8,
+    /// Authority address which must sign for each modification.
+    pub authority: Option<Pubkey>,
+    // Raw list of addresses follows this serialized structure in
+    // the account's data, starting from `LOOKUP_TABLE_META_SIZE`.
+}
+```
+
+To make it easier for address lookup tables to be updated by multi-sig or
+governance-controlled authorities, addresses can be buffered on-chain in
+a buffer account. Buffer accounts can be used to extend a lookup table
+with many addresses in a single small transaction.
+
+```rust
+pub struct BufferMeta {
+    /// Authority address which must sign for each modification.
+    pub authority: Pubkey,
+
+    // Serialized list of stored addresses follows the above metadata.
+}
+```
+
+#### Cleanup
+
+Once an address lookup table is no longer needed, it can be closed
+and have its rent balance reclaimed. Address lookup tables may not be recreated
+at the same address because each new lookup table must be initialized at an address
+derived from a recent slot.
+
+#### Cost
+
+Since address lookups require extra overhead during transaction processing,
+they should incur higher costs for a transaction.
+
+### Versioned Transactions
+
+In order to support address table lookups, the structure of serialized
+transactions must be modified. The new transaction format should not
+affect transaction processing in the Solana program runtime beyond the
+increased capacity for accounts and program invocations. Invoked
+programs will be unaware of which transaction format was used.
+
+The new transaction format must be distinguished from the current transaction
+format. Current transactions can fit at most 19 signatures (64-bytes each) but
+the message header encodes `num_required_signatures` as a `u8`. Since the upper
+bit of the `u8` will never be set for a valid transaction, we can enable it to
+denote whether a transaction should be decoded with the versioned format or not.
+
+#### New Transaction Format
+
+```rust
+#[derive(Serialize, Deserialize)]
+pub struct Transaction {
+    #[serde(with = "short_vec")]
+    pub signatures: Vec<Signature>,
+    /// The message to sign.
+    pub message: Message,
+}
+
+// Uses custom serialization. If the first bit is set, the remaining bits
+// in the first byte will encode a version number. If the first bit is not
+// set, the first byte will be treated as the first byte of an encoded
+// legacy message.
+pub enum VersionedMessage {
+    Legacy(Message),
+    V0(v0::Message),
+}
+
+// The structure of the new v0 Message
+#[derive(Serialize, Deserialize)]
+pub struct Message {
+  // unchanged
+  pub header: MessageHeader,
+
+  // unchanged
+  #[serde(with = "short_vec")]
+  pub account_keys: Vec<Pubkey>,
+
+  // unchanged
+  pub recent_blockhash: Hash,
+
+  // unchanged
+  //
+  // # Notes
+  //
+  // Account and program indexes will index into the list of addresses
+  // constructed from the concatenation of three key lists:
+  //   1) message `account_keys`
+  //   2) ordered list of keys loaded from address table `writable_indexes`
+  //   3) ordered list of keys loaded from address table `readable_indexes`
+  #[serde(with = "short_vec")]
+  pub instructions: Vec<CompiledInstruction>,
+
+  /// List of address table lookups used to load additional accounts
+  /// for this transaction.
+  #[serde(with = "short_vec")]
+  pub address_table_lookups: Vec<MessageAddressTableLookup>,
+}
+
+/// Address table lookups describe an on-chain address lookup table to use
+/// for loading more readonly and writable accounts in a single tx.
+#[derive(Serialize, Deserialize)]
+pub struct MessageAddressTableLookup {
+  /// Address lookup table account key
+  pub account_key: Pubkey,
+  /// List of indexes used to load writable account addresses
+  #[serde(with = "short_vec")]
+  pub writable_indexes: Vec<u8>,
+  /// List of indexes used to load readonly account addresses
+  #[serde(with = "short_vec")]
+  pub readonly_indexes: Vec<u8>,
+}
+```
+
+#### Size changes
+
+- 1 extra byte for `version` field
+- 1 extra byte for `address_table_lookups` length
+- 34 extra bytes for the address and lengths of the `writable_indexes`
+and `readonly_indexes` indexes in each address table lookup
+- 1 extra byte for each lookup table index
+
+#### Metadata changes
+
+Each resolved address from an address lookup table should be stored in
+the transaction metadata for quick reference. This will avoid the need for
+clients to make multiple RPC round trips to fetch all accounts loaded by a
+v2 transaction. It will also make it easier to use the ledger tool to
+analyze account access patterns.
+
+#### RPC changes
+
+Fetched transaction responses will likely require a new version field to
+indicate to clients which transaction structure to use for deserialization.
+Clients using pre-existing RPC methods will receive error responses when
+attempting to fetch a versioned transaction which will indicate that they
+must upgrade.
+
+The RPC API should also support an option for returning fully expanded
+transactions to abstract away the address lookup table details from
+downstream clients.
+
+### Limitations
+
+- Max of 256 unique accounts may be loaded by a transaction because `u8`
+is used by compiled instructions to index into transaction message `account_keys`.
+- Address lookup tables can hold up to 256 entries because lookup table indexes are also `u8`.
+- Transaction signers may not be loaded through an address lookup table, the full
+address of each signer must be serialized in the transaction. This ensures that
+the performance of transaction signature checks is not affected.
+- Hardware wallets will probably not be able to display details about accounts
+referenced through address lookup tables due to inability to verify on-chain data.
+- Only single level address lookup tables can be used. Recursive lookups will not be supported.
+
+## Security Concerns
+
+### Lookup table re-initialization
+
+If an address lookup table can be closed and re-initialized with new addresses,
+any client which is unaware of the change could inadvertently lookup unexpected
+addresses. To avoid this, all address lookup tables must be initialized at an
+address derived from a recent slot and they cannot be closed until the slot
+used for derivation is no longer "recent."
+
+### Resource consumption
+
+Enabling more account inputs in a transaction allows for more program
+invocations, write-locks, and data reads / writes. Before address tables are
+enabled, transaction-wide compute limits and increased costs for write locks and
+data reads are required.
+
+### Front running
+
+If the addresses listed within an address lookup table are mutable, front
+running attacks could modify which addresses are resolved for a later
+transaction. For this reason, address lookup tables are append-only and may
+only be closed if it's no longer possible to create a new lookup table at the
+same derived address.
+
+Additionally, a malicious actor could try to fork the chain immediately after a
+new address lookup table account is added to a block. If successful, they could
+add a different unexpected table entry in the fork. In order to deter this attack,
+clients should wait for address lookup tables to be finalized before using them in a
+transaction. Clients may also append integrity check instructions to the
+transaction which verify that the correct accounts are looked up.
+
+### Denial of service
+
+Address lookup table accounts may be read very frequently and will therefore
+be a more high profile target for denial of service attacks through write locks
+similar to sysvar accounts.
+
+For this reason, special handling should be given to address lookup tables.
+When an address lookup table is used to lookup addresses for a transaction,
+it can be loaded without waiting for a read lock. To avoid race conditions,
+only the addresses appended in previous blocks can be used for lookups.
+
+### Duplicate accounts
+
+Transactions may not load an account more than once whether directly through
+`account_keys` or indirectly through `address_table_lookups`.
+
+## Other Proposals
+
+1) Account prefixes
+
+Needing to pre-register accounts in an on-chain address lookup table is cumbersome
+because it adds an extra step for transaction processing. Instead, Solana
+transactions could use variable length address prefixes to specify accounts.
+These prefix shortcuts can save on data usage without needing to setup on-chain
+state.
+
+However, this model requires nodes to keep a mapping of prefixes to active account
+addresses. Attackers can create accounts with the same prefix as a popular account
+to disrupt transactions.
+
+2) Transaction builder program
+
+Solana can provide a new on-chain program which allows "Big" transactions to be
+constructed on-chain by normal transactions. Once the transaction is
+constructed, a final "Execute" transaction can trigger a node to process the big
+transaction as a normal transaction without needing to fit it into an MTU sized
+packet.
+
+The UX of this approach is tricky. A user could in theory sign a big transaction
+but it wouldn't be great if they had to use their wallet to sign multiple
+transactions to build that transaction that they already signed and approved. This
+could be a use-case for transaction relay services, though. A user could pay a
+relayer to construct the large pre-signed transaction on-chain for them.
+
+In order to prevent the large transaction from being reconstructed and replayed,
+its message hash will need to be added to the status cache when executed.
+
+3) Epoch account indexes
+
+Similarly to leader schedule calculation, validators could create a global index
+of the most accessed accounts in the previous epoch and make that index
+available to transactions in the following epoch.
+
+This approach has a downside of only updating the index at epoch boundaries
+which means there would be a few day delay before popular new accounts could be
+referenced. It also needs to be consistently generated by all validators by
+using some criteria like adding accounts in order by access count.
+
+4) Address lists
+
+Extend the transaction structure to support addresses that, when loaded, expand
+to a list of addresses. After expansion, all account inputs are concatenated to
+form a single list of account keys which can be indexed into by instructions.
+Address lists would likely need to be immutable to prevent attacks. They would
+also need to be limited in length to limit resource consumption.
+
+This proposal can be thought of a special case of the proposed index account
+approach. Since the full account list would be expanded, there's no need to add
+additional offsets that use up the limited space in a serialized transaction.
+However, the expected size of an address list may need to be encoded into the
+transaction to aid the sanitization of account indexes.  We would also need to
+encode how many addresses in the list should be loaded as readonly vs
+read-write. Lastly, special attention must be given to watch out for addresses
+that exist in multiple account lists.
+
+5) Increase transaction size
+
+Significantly larger serialized transactions have an increased likelihood of being
+dropped over the wire but this might not be a big issue since clients can retry
+transactions anyways. The only time validators need to send individual transactions
+over the network is when a leader forwards unprocessed transactions to the next
+leader.

package/src/proposals/validator-proposal.md

@@ -0,0 +1,54 @@
+---
+title: Validator
+---
+
+## History
+
+When we first started Solana, the goal was to de-risk our TPS claims. We knew
+that between optimistic concurrency control and sufficiently long leader slots,
+that PoS consensus was not the biggest risk to TPS. It was GPU-based signature
+verification, software pipelining and concurrent banking. Thus, the TPU was
+born. After topping 100k TPS, we split the team into one group working toward
+710k TPS and another to flesh out the validator pipeline. Hence, the TVU was
+born. The current architecture is a consequence of incremental development with
+that ordering and project priorities. It is not a reflection of what we ever
+believed was the most technically elegant cross-section of those technologies.
+In the context of leader rotation, the strong distinction between leading and
+validating is blurred.
+
+## Difference between validating and leading
+
+The fundamental difference between the pipelines is when the PoH is present. In
+a leader, we process transactions, removing bad ones, and then tag the result
+with a PoH hash. In the validator, we verify that hash, peel it off, and
+process the transactions in exactly the same way. The only difference is that
+if a validator sees a bad transaction, it can't simply remove it like the
+leader does, because that would cause the PoH hash to change. Instead, it
+rejects the whole block. The other difference between the pipelines is what
+happens _after_ banking. The leader broadcasts entries to downstream validators
+whereas the validator will have already done that in RetransmitStage, which is
+a confirmation time optimization. The validation pipeline, on the other hand,
+has one last step. Any time it finishes processing a block, it needs to weigh
+any forks it's observing, possibly cast a vote, and if so, reset its PoH hash
+to the block hash it just voted on.
+
+## Proposed Design
+
+We unwrap the many abstraction layers and build a single pipeline that can
+toggle leader mode on whenever the validator's ID shows up in the leader
+schedule.
+
+![Validator block diagram](/img/validator-proposal.svg)
+
+## Notable changes
+
+- Hoist FetchStage and BroadcastStage out of TPU
+- BankForks renamed to Banktree
+- TPU moves to new socket-free crate called solana-tpu.
+- TPU's BankingStage absorbs ReplayStage
+- TVU goes away
+- New RepairStage absorbs Shred Fetch Stage and repair requests
+- JSON RPC Service is optional - used for debugging. It should instead be part
+  of a separate `solana-blockstreamer` executable.
+- New MulticastStage absorbs retransmit part of RetransmitStage
+- MulticastStage downstream of Blockstore

package/src/proposals/vote-signing-to-implement.md

@@ -0,0 +1,116 @@
+---
+title: Secure Vote Signing
+---
+
+## Secure Vote Signing
+
+This design describes additional vote signing behavior that will make the process more secure.
+
+Currently, Solana implements a vote-signing service that evaluates each vote to ensure it does not violate a slashing condition. The service could potentially have different variations, depending on the hardware platform capabilities. In particular, it could be used in conjunction with a secure enclave \(such as SGX\). The enclave could generate an asymmetric key, exposing an API for user \(untrusted\) code to sign the vote transactions, while keeping the vote-signing private key in its protected memory.
+
+The following sections outline how this architecture would work:
+
+### Message Flow
+
+1. The node initializes the enclave at startup
+
+   - The enclave generates an asymmetric key and returns the public key to the
+
+     node
+
+   - The keypair is ephemeral. A new keypair is generated on node bootup. A
+
+     new keypair might also be generated at runtime based on some to be determined
+
+     criteria.
+
+   - The enclave returns its attestation report to the node
+
+2. The node performs attestation of the enclave \(e.g using Intel's IAS APIs\)
+
+   - The node ensures that the Secure Enclave is running on a TPM and is
+
+     signed by a trusted party
+
+3. The stakeholder of the node grants ephemeral key permission to use its stake.
+
+   This process is to be determined.
+
+4. The node's untrusted, non-enclave software calls trusted enclave software
+
+   using its interface to sign transactions and other data.
+
+   - In case of vote signing, the node needs to verify the PoH. The PoH
+
+     verification is an integral part of signing. The enclave would be
+
+     presented with some verifiable data to check before signing the vote.
+
+   - The process of generating the verifiable data in untrusted space is to be determined
+
+### PoH Verification
+
+1. When the node votes on an en entry `X`, there's a lockout period `N`, for
+
+   which it cannot vote on a fork that does not contain `X` in its history.
+
+2. Every time the node votes on the derivative of `X`, say `X+y`, the lockout
+
+   period for `X` increases by a factor `F` \(i.e. the duration node cannot vote on
+
+   a fork that does not contain `X` increases\).
+
+   - The lockout period for `X+y` is still `N` until the node votes again.
+
+3. The lockout period increment is capped \(e.g. factor `F` applies maximum 32
+
+   times\).
+
+4. The signing enclave must not sign a vote that violates this policy. This
+
+   means
+
+   - Enclave is initialized with `N`, `F` and `Factor cap`
+   - Enclave stores `Factor cap` number of entry IDs on which the node had
+
+     previously voted
+
+   - The sign request contains the entry ID for the new vote
+   - Enclave verifies that new vote's entry ID is on the correct fork
+
+     \(following the rules \#1 and \#2 above\)
+
+### Ancestor Verification
+
+This is alternate, albeit, less certain approach to verifying voting fork. 1. The validator maintains an active set of nodes in the cluster 2. It observes the votes from the active set in the last voting period 3. It stores the ancestor/last_tick at which each node voted 4. It sends new vote request to vote-signing service
+
+- It includes previous votes from nodes in the active set, and their
+
+  corresponding ancestors
+
+  1. The signer checks if the previous votes contains a vote from the validator,
+
+     and the vote ancestor matches with majority of the nodes
+
+- It signs the new vote if the check is successful
+- It asserts \(raises an alarm of some sort\) if the check is unsuccessful
+
+The premise is that the validator can be spoofed at most once to vote on incorrect data. If someone hijacks the validator and submits a vote request for bogus data, that vote will not be included in the PoH \(as it'll be rejected by the cluster\). The next time the validator sends a request to sign the vote, the signing service will detect that validator's last vote is missing \(as part of
+
+## 5 above\).
+
+### Fork determination
+
+Due to the fact that the enclave cannot process PoH, it has no direct knowledge of fork history of a submitted validator vote. Each enclave should be initiated with the current _active set_ of public keys. A validator should submit its current vote along with the votes of the active set \(including itself\) that it observed in the slot of its previous vote. In this way, the enclave can surmise the votes accompanying the validator's previous vote and thus the fork being voted on. This is not possible for the validator's initial submitted vote, as it will not have a 'previous' slot to reference. To account for this, a short voting freeze should apply until the second vote is submitted containing the votes within the active set, along with it's own vote, at the height of the initial vote.
+
+### Enclave configuration
+
+A staking client should be configurable to prevent voting on inactive forks. This mechanism should use the client's known active set `N_active` along with a threshold vote `N_vote` and a threshold depth `N_depth` to determine whether or not to continue voting on a submitted fork. This configuration should take the form of a rule such that the client will only vote on a fork if it observes more than `N_vote` at `N_depth`. Practically, this represents the client from confirming that it has observed some probability of economic finality of the submitted fork at a depth where an additional vote would create a lockout for an undesirable amount of time if that fork turns out not to be live.
+
+### Challenges
+
+1. Generation of verifiable data in untrusted space for PoH verification in the
+
+   enclave.
+
+2. Need infrastructure for granting stake to an ephemeral key.

package/src/running-validator/restart-cluster.md

@@ -0,0 +1,117 @@
+## Restarting a cluster
+
+### Step 1. Identify the slot that the cluster will be restarted at
+
+The highest optimistically confirmed slot is the best slot to start from, which
+can be found by looking for
+[this](https://github.com/solana-labs/solana/blob/0264147d42d506fb888f5c4c021a998e231a3e74/core/src/optimistic_confirmation_verifier.rs#L71)
+metrics datapoint. Otherwise use the last root.
+
+Call this slot `SLOT_X`
+
+### Step 2. Stop the validator(s)
+
+### Step 3. Optionally install the new solana version
+
+### Step 4. Create a new snapshot for slot `SLOT_X` with a hard fork at slot `SLOT_X`
+
+```bash
+$ solana-ledger-tool -l ledger create-snapshot SLOT_X ledger --hard-fork SLOT_X
+```
+
+The ledger directory should now contain the new snapshot.
+`solana-ledger-tool create-snapshot` will also output the new shred version, and bank hash value,
+call this NEW_SHRED_VERSION and NEW_BANK_HASH respectively.
+
+Adjust your validator's arguments:
+
+```bash
+ --wait-for-supermajority SLOT_X
+ --expected-bank-hash NEW_BANK_HASH
+```
+
+Then restart the validator.
+
+Confirm with the log that the validator booted and is now in a holding pattern at `SLOT_X`, waiting for a super majority.
+
+Once NEW_SHRED_VERSION is determined, nudge foundation entrypoint operators to update entrypoints.
+
+### Step 5. Announce the restart on Discord:
+
+Post something like the following to #announcements (adjusting the text as appropriate):
+
+> Hi @Validators,
+>
+> We've released v1.1.12 and are ready to get testnet back up again.
+>
+> Steps:
+>
+> 1. Install the v1.1.12 release: https://github.com/solana-labs/solana/releases/tag/v1.1.12
+> 2. a. Preferred method, start from your local ledger with:
+>
+> ```bash
+> solana-validator
+>   --wait-for-supermajority SLOT_X     # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
+>   --expected-bank-hash NEW_BANK_HASH  # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
+>   --hard-fork SLOT_X                  # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
+>   --no-snapshot-fetch                 # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
+>   --entrypoint entrypoint.testnet.solana.com:8001
+>   --known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on
+>   --expected-genesis-hash 4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY
+>   --only-known-rpc
+>   --limit-ledger-size
+>   ...                                # <-- your other --identity/--vote-account/etc arguments
+> ```
+>
+> b. If your validator doesn't have ledger up to slot SLOT_X or if you have deleted your ledger, have it instead download a snapshot with:
+>
+> ```bash
+> solana-validator
+>   --wait-for-supermajority SLOT_X     # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
+>   --expected-bank-hash NEW_BANK_HASH  # <-- NEW! IMPORTANT! REMOVE AFTER THIS RESTART
+>   --entrypoint entrypoint.testnet.solana.com:8001
+>   --known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on
+>   --expected-genesis-hash 4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY
+>   --only-known-rpc
+>   --limit-ledger-size
+>   ...                                # <-- your other --identity/--vote-account/etc arguments
+> ```
+>
+>      You can check for which slots your ledger has with: `solana-ledger-tool -l path/to/ledger bounds`
+>
+> 3. Wait until 80% of the stake comes online
+>
+> To confirm your restarted validator is correctly waiting for the 80%:
+> a. Look for `N% of active stake visible in gossip` log messages
+> b. Ask it over RPC what slot it's on: `solana --url http://127.0.0.1:8899 slot`. It should return `SLOT_X` until we get to 80% stake
+>
+> Thanks!
+
+### Step 7. Wait and listen
+
+Monitor the validators as they restart. Answer questions, help folks,
+
+## Troubleshooting
+
+### 80% of the stake didn't participate in the restart, now what?
+If less than 80% of the stake join the restart after a reasonable amount of
+time, it will be necessary to retry the restart attempt with the stake from the
+non-responsive validators removed.
+
+The community should identify and come to social consensus on the set of
+non-responsive validators. Then all participating validators return to Step 4
+and create a new snapshot with additional `--destake-vote-account <PUBKEY>`
+arguments for each of the non-responsive validator's vote account address
+
+```bash
+$ solana-ledger-tool -l ledger create-snapshot SLOT_X ledger --hard-fork SLOT_X \
+    --destake-vote-account <VOTE_ACCOUNT_1> \
+    --destake-vote-account <VOTE_ACCOUNT_2> \
+    .
+    .
+    --destake-vote-account <VOTE_ACCOUNT_N> \
+```
+
+This will cause all stake associated with the non-responsive validators to be
+immediately deactivated. All their stakers will need to re-delegate their stake
+once the cluster restart is successful.