solana-docs 0.0.0

package/src/cli/install-solana-cli-tools.md

@@ -0,0 +1,164 @@
+---
+title: Install the Solana Tool Suite
+---
+
+There are multiple ways to install the Solana tools on your computer
+depending on your preferred workflow:
+
+- [Use Solana's Install Tool (Simplest option)](#use-solanas-install-tool)
+- [Download Prebuilt Binaries](#download-prebuilt-binaries)
+- [Build from Source](#build-from-source)
+
+## Use Solana's Install Tool
+
+### MacOS & Linux
+
+- Open your favorite Terminal application
+
+- Install the Solana release
+  [LATEST_SOLANA_RELEASE_VERSION](https://github.com/solana-labs/solana/releases/tag/LATEST_SOLANA_RELEASE_VERSION) on your
+  machine by running:
+
+```bash
+sh -c "$(curl -sSfL https://release.solana.com/LATEST_SOLANA_RELEASE_VERSION/install)"
+```
+
+- You can replace `LATEST_SOLANA_RELEASE_VERSION` with the release tag matching
+  the software version of your desired release, or use one of the three symbolic
+  channel names: `stable`, `beta`, or `edge`.
+
+- The following output indicates a successful update:
+
+```text
+downloading LATEST_SOLANA_RELEASE_VERSION installer
+Configuration: /home/solana/.config/solana/install/config.yml
+Active release directory: /home/solana/.local/share/solana/install/active_release
+* Release version: LATEST_SOLANA_RELEASE_VERSION
+* Release URL: https://github.com/solana-labs/solana/releases/download/LATEST_SOLANA_RELEASE_VERSION/solana-release-x86_64-unknown-linux-gnu.tar.bz2
+Update successful
+```
+
+- Depending on your system, the end of the installer messaging may prompt you
+  to
+
+```bash
+Please update your PATH environment variable to include the solana programs:
+```
+
+- If you get the above message, copy and paste the recommended command below
+  it to update `PATH`
+- Confirm you have the desired version of `solana` installed by running:
+
+```bash
+solana --version
+```
+
+- After a successful install, `solana-install update` may be used to easily
+  update the Solana software to a newer version at any time.
+
+---
+
+### Windows
+
+- Open a Command Prompt (`cmd.exe`) as an Administrator
+
+  - Search for Command Prompt in the Windows search bar. When the Command
+    Prompt app appears, right-click and select “Open as Administrator”.
+    If you are prompted by a pop-up window asking “Do you want to allow this app to
+    make changes to your device?”, click Yes.
+
+- Copy and paste the following command, then press Enter to download the Solana
+  installer into a temporary directory:
+
+```bash
+curl https://release.solana.com/LATEST_SOLANA_RELEASE_VERSION/solana-install-init-x86_64-pc-windows-msvc.exe --output C:\solana-install-tmp\solana-install-init.exe --create-dirs
+```
+
+- Copy and paste the following command, then press Enter to install the latest
+  version of Solana. If you see a security pop-up by your system, please select
+  to allow the program to run.
+
+```bash
+C:\solana-install-tmp\solana-install-init.exe LATEST_SOLANA_RELEASE_VERSION
+```
+
+- When the installer is finished, press Enter.
+
+- Close the command prompt window and re-open a new command prompt window as a
+  normal user
+  - Search for "Command Prompt" in the search bar, then left click on the
+    Command Prompt app icon, no need to run as Administrator)
+- Confirm you have the desired version of `solana` installed by entering:
+
+```bash
+solana --version
+```
+
+- After a successful install, `solana-install update` may be used to easily
+  update the Solana software to a newer version at any time.
+
+## Download Prebuilt Binaries
+
+If you would rather not use `solana-install` to manage the install, you can
+manually download and install the binaries.
+
+### Linux
+
+Download the binaries by navigating to
+[https://github.com/solana-labs/solana/releases/latest](https://github.com/solana-labs/solana/releases/latest),
+download **solana-release-x86_64-unknown-linux-msvc.tar.bz2**, then extract the
+archive:
+
+```bash
+tar jxf solana-release-x86_64-unknown-linux-gnu.tar.bz2
+cd solana-release/
+export PATH=$PWD/bin:$PATH
+```
+
+### MacOS
+
+Download the binaries by navigating to
+[https://github.com/solana-labs/solana/releases/latest](https://github.com/solana-labs/solana/releases/latest),
+download **solana-release-x86_64-apple-darwin.tar.bz2**, then extract the
+archive:
+
+```bash
+tar jxf solana-release-x86_64-apple-darwin.tar.bz2
+cd solana-release/
+export PATH=$PWD/bin:$PATH
+```
+
+### Windows
+
+- Download the binaries by navigating to
+  [https://github.com/solana-labs/solana/releases/latest](https://github.com/solana-labs/solana/releases/latest),
+  download **solana-release-x86_64-pc-windows-msvc.tar.bz2**, then extract the
+  archive using WinZip or similar.
+
+- Open a Command Prompt and navigate to the directory into which you extracted
+  the binaries and run:
+
+```bash
+cd solana-release/
+set PATH=%cd%/bin;%PATH%
+```
+
+## Build From Source
+
+If you are unable to use the prebuilt binaries or prefer to build it yourself
+from source, navigate to
+[https://github.com/solana-labs/solana/releases/latest](https://github.com/solana-labs/solana/releases/latest),
+and download the **Source Code** archive. Extract the code and build the
+binaries with:
+
+```bash
+./scripts/cargo-install-all.sh .
+export PATH=$PWD/bin:$PATH
+```
+
+You can then run the following command to obtain the same result as with
+prebuilt binaries:
+
+```bash
+solana-install init
+```

package/src/cli/transfer-tokens.md

@@ -0,0 +1,158 @@
+---
+title: Send and Receive Tokens
+---
+
+This page decribes how to receive and send SOL tokens using the command line
+tools with a command line wallet such as a [paper wallet](../wallet-guide/paper-wallet.md),
+a [file system wallet](../wallet-guide/file-system-wallet.md), or a
+[hardware wallet](../wallet-guide/hardware-wallets.md). Before you begin, make sure
+you have created a wallet and have access to its address (pubkey) and the
+signing keypair. Check out our
+[conventions for entering keypairs for different wallet types](../cli/conventions.md#keypair-conventions).
+
+## Testing your Wallet
+
+Before sharing your public key with others, you may want to first ensure the
+key is valid and that you indeed hold the corresponding private key.
+
+In this example, we will create a second wallet in addition to your first wallet,
+and then transfer some tokens to it. This will confirm that you can send and
+receive tokens on your wallet type of choice.
+
+This test example uses our Developer Testnet, called devnet. Tokens issued
+on devnet have **no** value, so don't worry if you lose them.
+
+#### Airdrop some tokens to get started
+
+First, _airdrop_ yourself some play tokens on the devnet.
+
+```bash
+solana airdrop 1 <RECIPIENT_ACCOUNT_ADDRESS> --url https://api.devnet.solana.com
+```
+
+where you replace the text `<RECIPIENT_ACCOUNT_ADDRESS>` with your base58-encoded
+public key/wallet address.
+
+#### Check your balance
+
+Confirm the airdrop was successful by checking the account's balance.
+It should output `1 SOL`:
+
+```bash
+solana balance <ACCOUNT_ADDRESS> --url https://api.devnet.solana.com
+```
+
+#### Create a second wallet address
+
+We will need a new address to receive our tokens. Create a second
+keypair and record its pubkey:
+
+```bash
+solana-keygen new --no-passphrase --no-outfile
+```
+
+The output will contain the address after the text `pubkey:`. Copy the
+address. We will use it in the next step.
+
+```text
+pubkey: GKvqsuNcnwWqPzzuhLmGi4rzzh55FhJtGizkhHaEJqiV
+```
+
+You can also create a second (or more) wallet of any type:
+[paper](../wallet-guide/paper-wallet#creating-multiple-paper-wallet-addresses),
+[file system](../wallet-guide/file-system-wallet.md#creating-multiple-file-system-wallet-addresses),
+or [hardware](../wallet-guide/hardware-wallets.md#multiple-addresses-on-a-single-hardware-wallet).
+
+#### Transfer tokens from your first wallet to the second address
+
+Next, prove that you own the airdropped tokens by transferring them.
+The Solana cluster will only accept the transfer if you sign the transaction
+with the private keypair corresponding to the sender's public key in the
+transaction.
+
+```bash
+solana transfer --from <KEYPAIR> <RECIPIENT_ACCOUNT_ADDRESS> 0.5 --allow-unfunded-recipient --url https://api.devnet.solana.com --fee-payer <KEYPAIR>
+```
+
+where you replace `<KEYPAIR>` with the path to a keypair in your first wallet,
+and replace `<RECIPIENT_ACCOUNT_ADDRESS>` with the address of your second
+wallet.
+
+Confirm the updated balances with `solana balance`:
+
+```bash
+solana balance <ACCOUNT_ADDRESS> --url http://api.devnet.solana.com
+```
+
+where `<ACCOUNT_ADDRESS>` is either the public key from your keypair or the
+recipient's public key.
+
+#### Full example of test transfer
+
+```bash
+$ solana-keygen new --outfile my_solana_wallet.json   # Creating my first wallet, a file system wallet
+Generating a new keypair
+For added security, enter a passphrase (empty for no passphrase):
+Wrote new keypair to my_solana_wallet.json
+==========================================================================
+pubkey: DYw8jCTfwHNRJhhmFcbXvVDTqWMEVFBX6ZKUmG5CNSKK                          # Here is the address of the first wallet
+==========================================================================
+Save this seed phrase to recover your new keypair:
+width enhance concert vacant ketchup eternal spy craft spy guard tag punch    # If this was a real wallet, never share these words on the internet like this!
+==========================================================================
+
+$ solana airdrop 1 DYw8jCTfwHNRJhhmFcbXvVDTqWMEVFBX6ZKUmG5CNSKK --url https://api.devnet.solana.com  # Airdropping 1 SOL to my wallet's address/pubkey
+Requesting airdrop of 1 SOL from 35.233.193.70:9900
+1 SOL
+
+$ solana balance DYw8jCTfwHNRJhhmFcbXvVDTqWMEVFBX6ZKUmG5CNSKK --url https://api.devnet.solana.com # Check the address's balance
+1 SOL
+
+$ solana-keygen new --no-outfile  # Creating a second wallet, a paper wallet
+Generating a new keypair
+For added security, enter a passphrase (empty for no passphrase):
+====================================================================
+pubkey: 7S3P4HxJpyyigGzodYwHtCxZyUQe9JiBMHyRWXArAaKv                   # Here is the address of the second, paper, wallet.
+====================================================================
+Save this seed phrase to recover your new keypair:
+clump panic cousin hurt coast charge engage fall eager urge win love   # If this was a real wallet, never share these words on the internet like this!
+====================================================================
+
+$ solana transfer --from my_solana_wallet.json 7S3P4HxJpyyigGzodYwHtCxZyUQe9JiBMHyRWXArAaKv 0.5 --allow-unfunded-recipient --url https://api.devnet.solana.com --fee-payer my_solana_wallet.json  # Transferring tokens to the public address of the paper wallet
+3gmXvykAd1nCQQ7MjosaHLf69Xyaqyq1qw2eu1mgPyYXd5G4v1rihhg1CiRw35b9fHzcftGKKEu4mbUeXY2pEX2z  # This is the transaction signature
+
+$ solana balance DYw8jCTfwHNRJhhmFcbXvVDTqWMEVFBX6ZKUmG5CNSKK --url https://api.devnet.solana.com
+0.499995 SOL  # The sending account has slightly less than 0.5 SOL remaining due to the 0.000005 SOL transaction fee payment
+
+$ solana balance 7S3P4HxJpyyigGzodYwHtCxZyUQe9JiBMHyRWXArAaKv --url https://api.devnet.solana.com
+0.5 SOL  # The second wallet has now received the 0.5 SOL transfer from the first wallet
+
+```
+
+## Receive Tokens
+
+To receive tokens, you will need an address for others to send tokens to. In
+Solana, the wallet address is the public key of a keypair. There are a variety
+of techniques for generating keypairs. The method you choose will depend on how
+you choose to store keypairs. Keypairs are stored in wallets. Before receiving
+tokens, you will need to [create a wallet](../wallet-guide/cli.md).
+Once completed, you should have a public key
+for each keypair you generated. The public key is a long string of base58
+characters. Its length varies from 32 to 44 characters.
+
+## Send Tokens
+
+If you already hold SOL and want to send tokens to someone, you will need
+a path to your keypair, their base58-encoded public key, and a number of
+tokens to transfer. Once you have that collected, you can transfer tokens
+with the `solana transfer` command:
+
+```bash
+solana transfer --from <KEYPAIR> <RECIPIENT_ACCOUNT_ADDRESS> <AMOUNT> --fee-payer <KEYPAIR>
+```
+
+Confirm the updated balances with `solana balance`:
+
+```bash
+solana balance <ACCOUNT_ADDRESS>
+```

package/src/cli.md

@@ -0,0 +1,22 @@
+---
+title: Command-line Guide
+---
+
+In this section, we will describe how to use the Solana command-line tools to
+create a _wallet_, to send and receive SOL tokens, and to participate in
+the cluster by delegating stake.
+
+To interact with a Solana cluster, we will use its command-line interface, also
+known as the CLI. We use the command-line because it is the first place the
+Solana core team deploys new functionality. The command-line interface is not
+necessarily the easiest to use, but it provides the most direct, flexible, and
+secure access to your Solana accounts.
+
+## Getting Started
+
+To get started using the Solana Command Line (CLI) tools:
+
+- [Install the Solana Tools](cli/install-solana-cli-tools.md)
+- [Choose a Cluster](cli/choose-a-cluster.md)
+- [Create a Wallet](wallet-guide/cli.md)
+- [Check out our CLI conventions](cli/conventions.md)

package/src/cluster/bench-tps.md

@@ -0,0 +1,129 @@
+---
+title: Benchmark a Cluster
+---
+
+The Solana git repository contains all the scripts you might need to spin up your own local testnet. Depending on what you're looking to achieve, you may want to run a different variation, as the full-fledged, performance-enhanced multinode testnet is considerably more complex to set up than a Rust-only, singlenode testnode. If you are looking to develop high-level features, such as experimenting with smart contracts, save yourself some setup headaches and stick to the Rust-only singlenode demo. If you're doing performance optimization of the transaction pipeline, consider the enhanced singlenode demo. If you're doing consensus work, you'll need at least a Rust-only multinode demo. If you want to reproduce our TPS metrics, run the enhanced multinode demo.
+
+For all four variations, you'd need the latest Rust toolchain and the Solana source code:
+
+First, setup Rust, Cargo and system packages as described in the Solana [README](https://github.com/solana-labs/solana#1-install-rustc-cargo-and-rustfmt)
+
+Now checkout the code from github:
+
+```bash
+git clone https://github.com/solana-labs/solana.git
+cd solana
+```
+
+The demo code is sometimes broken between releases as we add new low-level features, so if this is your first time running the demo, you'll improve your odds of success if you check out the [latest release](https://github.com/solana-labs/solana/releases) before proceeding:
+
+```bash
+TAG=$(git describe --tags $(git rev-list --tags --max-count=1))
+git checkout $TAG
+```
+
+### Configuration Setup
+
+Ensure important programs such as the vote program are built before any nodes are started. Note that we are using the release build here for good performance.
+If you want the debug build, use just `cargo build` and omit the `NDEBUG=1` part of the command.
+
+```bash
+cargo build --release
+```
+
+The network is initialized with a genesis ledger generated by running the following script.
+
+```bash
+NDEBUG=1 ./multinode-demo/setup.sh
+```
+
+### Faucet
+
+In order for the validators and clients to work, we'll need to spin up a faucet to give out some test tokens. The faucet delivers Milton Friedman-style "air drops" \(free tokens to requesting clients\) to be used in test transactions.
+
+Start the faucet with:
+
+```bash
+NDEBUG=1 ./multinode-demo/faucet.sh
+```
+
+### Singlenode Testnet
+
+Before you start a validator, make sure you know the IP address of the machine you want to be the bootstrap validator for the demo, and make sure that udp ports 8000-10000 are open on all the machines you want to test with.
+
+Now start the bootstrap validator in a separate shell:
+
+```bash
+NDEBUG=1 ./multinode-demo/bootstrap-validator.sh
+```
+
+Wait a few seconds for the server to initialize. It will print "leader ready..." when it's ready to receive transactions. The leader will request some tokens from the faucet if it doesn't have any. The faucet does not need to be running for subsequent leader starts.
+
+### Multinode Testnet
+
+To run a multinode testnet, after starting a leader node, spin up some additional validators in separate shells:
+
+```bash
+NDEBUG=1 ./multinode-demo/validator-x.sh
+```
+
+To run a performance-enhanced validator on Linux, [CUDA 10.0](https://developer.nvidia.com/cuda-downloads) must be installed on your system:
+
+```bash
+./fetch-perf-libs.sh
+NDEBUG=1 SOLANA_CUDA=1 ./multinode-demo/bootstrap-validator.sh
+NDEBUG=1 SOLANA_CUDA=1 ./multinode-demo/validator.sh
+```
+
+### Testnet Client Demo
+
+Now that your singlenode or multinode testnet is up and running let's send it some transactions!
+
+In a separate shell start the client:
+
+```bash
+NDEBUG=1 ./multinode-demo/bench-tps.sh # runs against localhost by default
+```
+
+What just happened? The client demo spins up several threads to send 500,000 transactions to the testnet as quickly as it can. The client then pings the testnet periodically to see how many transactions it processed in that time. Take note that the demo intentionally floods the network with UDP packets, such that the network will almost certainly drop a bunch of them. This ensures the testnet has an opportunity to reach 710k TPS. The client demo completes after it has convinced itself the testnet won't process any additional transactions. You should see several TPS measurements printed to the screen. In the multinode variation, you'll see TPS measurements for each validator node as well.
+
+### Testnet Debugging
+
+There are some useful debug messages in the code, you can enable them on a per-module and per-level basis. Before running a leader or validator set the normal RUST_LOG environment variable.
+
+For example
+
+- To enable `info` everywhere and `debug` only in the solana::banking_stage module:
+
+  ```bash
+  export RUST_LOG=solana=info,solana::banking_stage=debug
+  ```
+
+- To enable BPF program logging:
+
+  ```bash
+  export RUST_LOG=solana_bpf_loader=trace
+  ```
+
+Generally we are using `debug` for infrequent debug messages, `trace` for potentially frequent messages and `info` for performance-related logging.
+
+You can also attach to a running process with GDB. The leader's process is named _solana-validator_:
+
+```bash
+sudo gdb
+attach <PID>
+set logging on
+thread apply all bt
+```
+
+This will dump all the threads stack traces into gdb.txt
+
+## Developer Testnet
+
+In this example the client connects to our public testnet. To run validators on the testnet you would need to open udp ports `8000-10000`.
+
+```bash
+NDEBUG=1 ./multinode-demo/bench-tps.sh --entrypoint entrypoint.devnet.solana.com:8001 --faucet api.devnet.solana.com:9900 --duration 60 --tx_count 50
+```
+
+You can observe the effects of your client's transactions on our [metrics dashboard](https://metrics.solana.com:3000/d/monitor/cluster-telemetry?var-testnet=devnet)

package/src/cluster/fork-generation.md

@@ -0,0 +1,64 @@
+---
+title: Fork Generation
+---
+
+This section describes how forks naturally occur as a consequence of [leader rotation](leader-rotation.md).
+
+## Overview
+
+Nodes take turns being leader and generating the PoH that encodes state changes. The cluster can tolerate loss of connection to any leader by synthesizing what the leader _**would**_ have generated had it been connected but not ingesting any state changes. The possible number of forks is thereby limited to a "there/not-there" skip list of forks that may arise on leader rotation slot boundaries. At any given slot, only a single leader's transactions will be accepted.
+
+## Message Flow
+
+1. Transactions are ingested by the current leader.
+2. Leader filters valid transactions.
+3. Leader executes valid transactions updating its state.
+4. Leader packages transactions into entries based off its current PoH slot.
+5. Leader transmits the entries to validator nodes \(in signed shreds\)
+   1. The PoH stream includes ticks; empty entries that indicate liveness of the leader and the passage of time on the cluster.
+   2. A leader's stream begins with the tick entries necessary to complete PoH back to the leader's most recently observed prior leader slot.
+6. Validators retransmit entries to peers in their set and to further downstream nodes.
+7. Validators validate the transactions and execute them on their state.
+8. Validators compute the hash of the state.
+9. At specific times, i.e. specific PoH tick counts, validators transmit votes to the leader.
+   1. Votes are signatures of the hash of the computed state at that PoH tick count.
+   2. Votes are also propagated via gossip.
+10. Leader executes the votes, the same as any other transaction, and broadcasts them to the cluster.
+11. Validators observe their votes and all the votes from the cluster.
+
+## Partitions, Forks
+
+Forks can arise at PoH tick counts that correspond to a vote. The next leader may not have observed the last vote slot and may start their slot with generated virtual PoH entries. These empty ticks are generated by all nodes in the cluster at a cluster-configured rate for hashes/per/tick `Z`.
+
+There are only two possible versions of the PoH during a voting slot: PoH with `T` ticks and entries generated by the current leader, or PoH with just ticks. The "just ticks" version of the PoH can be thought of as a virtual ledger, one that all nodes in the cluster can derive from the last tick in the previous slot.
+
+Validators can ignore forks at other points \(e.g. from the wrong leader\), or slash the leader responsible for the fork.
+
+Validators vote based on a greedy choice to maximize their reward described in [Tower BFT](../implemented-proposals/tower-bft.md).
+
+### Validator's View
+
+#### Time Progression
+
+The diagram below represents a validator's view of the PoH stream with possible forks over time. L1, L2, etc. are leader slots, and `E`s represent entries from that leader during that leader's slot. The `x`s represent ticks only, and time flows downwards in the diagram.
+
+![Fork generation](/img/fork-generation.svg)
+
+Note that an `E` appearing on 2 forks at the same slot is a slashable condition, so a validator observing `E3` and `E3'` can slash L3 and safely choose `x` for that slot. Once a validator commits to a fork, other forks can be discarded below that tick count. For any slot, validators need only consider a single "has entries" chain or a "ticks only" chain to be proposed by a leader. But multiple virtual entries may overlap as they link back to the a previous slot.
+
+#### Time Division
+
+It's useful to consider leader rotation over PoH tick count as time division of the job of encoding state for the cluster. The following table presents the above tree of forks as a time-divided ledger.
+
+| leader slot      | L1  | L2  | L3  | L4  | L5  |
+| :--------------- | :-- | :-- | :-- | :-- | :-- |
+| data             | E1  | E2  | E3  | E4  | E5  |
+| ticks since prev |     |     |     | x   | xx  |
+
+Note that only data from leader L3 will be accepted during leader slot L3. Data from L3 may include "catchup" ticks back to a slot other than L2 if L3 did not observe L2's data. L4 and L5's transmissions include the "ticks to prev" PoH entries.
+
+This arrangement of the network data streams permits nodes to save exactly this to the ledger for replay, restart, and checkpoints.
+
+### Leader's View
+
+When a new leader begins a slot, it must first transmit any PoH \(ticks\) required to link the new slot with the most recently observed and voted slot. The fork the leader proposes would link the current slot to a previous fork that the leader has voted on with virtual ticks.

package/src/cluster/leader-rotation.md

@@ -0,0 +1,88 @@
+---
+title: Leader Rotation
+---
+
+At any given moment, a cluster expects only one validator to produce ledger entries. By having only one leader at a time, all validators are able to replay identical copies of the ledger. The drawback of only one leader at a time, however, is that a malicious leader is capable of censoring votes and transactions. Since censoring cannot be distinguished from the network dropping packets, the cluster cannot simply elect a single node to hold the leader role indefinitely. Instead, the cluster minimizes the influence of a malicious leader by rotating which node takes the lead.
+
+Each validator selects the expected leader using the same algorithm, described below. When the validator receives a new signed ledger entry, it can be certain that an entry was produced by the expected leader. The order of slots which each leader is assigned a slot is called a _leader schedule_.
+
+## Leader Schedule Rotation
+
+A validator rejects blocks that are not signed by the _slot leader_. The list of identities of all slot leaders is called a _leader schedule_. The leader schedule is recomputed locally and periodically. It assigns slot leaders for a duration of time called an _epoch_. The schedule must be computed far in advance of the slots it assigns, such that the ledger state it uses to compute the schedule is finalized. That duration is called the _leader schedule offset_. Solana sets the offset to the duration of slots until the next epoch. That is, the leader schedule for an epoch is calculated from the ledger state at the start of the previous epoch. The offset of one epoch is fairly arbitrary and assumed to be sufficiently long such that all validators will have finalized their ledger state before the next schedule is generated. A cluster may choose to shorten the offset to reduce the time between stake changes and leader schedule updates.
+
+While operating without partitions lasting longer than an epoch, the schedule only needs to be generated when the root fork crosses the epoch boundary. Since the schedule is for the next epoch, any new stakes committed to the root fork will not be active until the next epoch. The block used for generating the leader schedule is the first block to cross the epoch boundary.
+
+Without a partition lasting longer than an epoch, the cluster will work as follows:
+
+1. A validator continuously updates its own root fork as it votes.
+2. The validator updates its leader schedule each time the slot height crosses an epoch boundary.
+
+For example:
+
+Let's assume an epoch duration of 100 slots, which in reality is magnitudes higher. The root fork is updated from fork computed at slot height 99 to a fork computed at slot height 102. Forks with slots at height 100, 101 were skipped because of failures. The new leader schedule is computed using fork at slot height 102. It is active from slot 200 until it is updated again.
+
+No inconsistency can exist because every validator that is voting with the cluster has skipped 100 and 101 when its root passes 102. All validators, regardless of voting pattern, would be committing to a root that is either 102, or a descendant of 102.
+
+### Leader Schedule Rotation with Epoch Sized Partitions.
+
+The duration of the leader schedule offset has a direct relationship to the likelihood of a cluster having an inconsistent view of the correct leader schedule.
+
+Consider the following scenario:
+
+Two partitions that are generating half of the blocks each. Neither is coming to a definitive supermajority fork. Both will cross epoch 100 and 200 without actually committing to a root and therefore a cluster-wide commitment to a new leader schedule.
+
+In this unstable scenario, multiple valid leader schedules exist.
+
+- A leader schedule is generated for every fork whose direct parent is in the previous epoch.
+- The leader schedule is valid after the start of the next epoch for descendant forks until it is updated.
+
+Each partition's schedule will diverge after the partition lasts more than an epoch. For this reason, the epoch duration should be selected to be much much larger then slot time and the expected length for a fork to be committed to root.
+
+After observing the cluster for a sufficient amount of time, the leader schedule offset can be selected based on the median partition duration and its standard deviation. For example, an offset longer then the median partition duration plus six standard deviations would reduce the likelihood of an inconsistent ledger schedule in the cluster to 1 in 1 million.
+
+## Leader Schedule Generation at Genesis
+
+The genesis config declares the first leader for the first epoch. This leader ends up scheduled for the first two epochs because the leader schedule is also generated at slot 0 for the next epoch. The length of the first two epochs can be specified in the genesis config as well. The minimum length of the first epochs must be greater than or equal to the maximum rollback depth as defined in [Tower BFT](../implemented-proposals/tower-bft.md).
+
+## Leader Schedule Generation Algorithm
+
+Leader schedule is generated using a predefined seed. The process is as follows:
+
+1. Periodically use the PoH tick height \(a monotonically increasing counter\) to seed a stable pseudo-random algorithm.
+2. At that height, sample the bank for all the staked accounts with leader identities that have voted within a cluster-configured number of ticks. The sample is called the _active set_.
+3. Sort the active set by stake weight.
+4. Use the random seed to select nodes weighted by stake to create a stake-weighted ordering.
+5. This ordering becomes valid after a cluster-configured number of ticks.
+
+## Schedule Attack Vectors
+
+### Seed
+
+The seed that is selected is predictable but unbiasable. There is no grinding attack to influence its outcome.
+
+### Active Set
+
+A leader can bias the active set by censoring validator votes. Two possible ways exist for leaders to censor the active set:
+
+- Ignore votes from validators
+- Refuse to vote for blocks with votes from validators
+
+To reduce the likelihood of censorship, the active set is calculated at the leader schedule offset boundary over an _active set sampling duration_. The active set sampling duration is long enough such that votes will have been collected by multiple leaders.
+
+### Staking
+
+Leaders can censor new staking transactions or refuse to validate blocks with new stakes. This attack is similar to censorship of validator votes.
+
+### Validator operational key loss
+
+Leaders and validators are expected to use ephemeral keys for operation, and stake owners authorize the validators to do work with their stake via delegation.
+
+The cluster should be able to recover from the loss of all the ephemeral keys used by leaders and validators, which could occur through a common software vulnerability shared by all the nodes. Stake owners should be able to vote directly by co-signing a validator vote even though the stake is currently delegated to a validator.
+
+## Appending Entries
+
+The lifetime of a leader schedule is called an _epoch_. The epoch is split into _slots_, where each slot has a duration of `T` PoH ticks.
+
+A leader transmits entries during its slot. After `T` ticks, all the validators switch to the next scheduled leader. Validators must ignore entries sent outside a leader's assigned slot.
+
+All `T` ticks must be observed by the next leader for it to build its own entries on. If entries are not observed \(leader is down\) or entries are invalid \(leader is buggy or malicious\), the next leader must produce ticks to fill the previous leader's slot. Note that the next leader should do repair requests in parallel, and postpone sending ticks until it is confident other validators also failed to observe the previous leader's entries. If a leader incorrectly builds on its own ticks, the leader following it must replace all its ticks.

package/src/cluster/managing-forks.md

@@ -0,0 +1,36 @@
+---
+title: Managing Forks
+---
+
+The ledger is permitted to fork at slot boundaries. The resulting data structure forms a tree called a _blockstore_. When the validator interprets the blockstore, it must maintain state for each fork in the chain. We call each instance an _active fork_. It is the responsibility of a validator to weigh those forks, such that it may eventually select a fork.
+
+A validator selects a fork by submiting a vote to a slot leader on that fork. The vote commits the validator for a duration of time called a _lockout period_. The validator is not permitted to vote on a different fork until that lockout period expires. Each subsequent vote on the same fork doubles the length of the lockout period. After some cluster-configured number of votes \(currently 32\), the length of the lockout period reaches what's called _max lockout_. Until the max lockout is reached, the validator has the option to wait until the lockout period is over and then vote on another fork. When it votes on another fork, it performs an operation called _rollback_, whereby the state rolls back in time to a shared checkpoint and then jumps forward to the tip of the fork that it just voted on. The maximum distance that a fork may roll back is called the _rollback depth_. Rollback depth is the number of votes required to achieve max lockout. Whenever a validator votes, any checkpoints beyond the rollback depth become unreachable. That is, there is no scenario in which the validator will need to roll back beyond rollback depth. It therefore may safely _prune_ unreachable forks and _squash_ all checkpoints beyond rollback depth into the root checkpoint.
+
+## Active Forks
+
+An active fork is as a sequence of checkpoints that has a length at least one longer than the rollback depth. The shortest fork will have a length exactly one longer than the rollback depth. For example:
+
+![Forks](/img/forks.svg)
+
+The following sequences are _active forks_:
+
+- {4, 2, 1}
+- {5, 2, 1}
+- {6, 3, 1}
+- {7, 3, 1}
+
+## Pruning and Squashing
+
+A validator may vote on any checkpoint in the tree. In the diagram above, that's every node except the leaves of the tree. After voting, the validator prunes nodes that fork from a distance farther than the rollback depth and then takes the opportunity to minimize its memory usage by squashing any nodes it can into the root.
+
+Starting from the example above, with a rollback depth of 2, consider a vote on 5 versus a vote on 6. First, a vote on 5:
+
+![Forks after pruning](/img/forks-pruned.svg)
+
+The new root is 2, and any active forks that are not descendants from 2 are pruned.
+
+Alternatively, a vote on 6:
+
+![Forks](/img/forks-pruned2.svg)
+
+The tree remains with a root of 1, since the active fork starting at 6 is only 2 checkpoints from the root.