solana-docs 0.0.0

package/src/running-validator/validator-failover.md

@@ -0,0 +1,146 @@
+---
+title: Failover Setup
+---
+
+A simple two machine instance failover method is described here, which allows you to:
+* upgrade your validator software with virtually no down time, and
+* failover to the secondary instance when your monitoring detects a problem with
+  the primary instance
+without any safety issues that would otherwise be associated with running two
+instances of your validator.
+
+You will need two validator-class machines for your primary and secondary
+validator. A third machine for running an [etcd](https://etcd.io/) cluster,
+which is used to store the tower voting record for your validator.
+
+## Setup
+
+### etcd cluster setup
+
+There is ample documentation regarding etcd setup and configuration at
+https://etcd.io/, please generally familiarize yourself with etcd before
+continuing.
+
+It's recommended that etcd be installed on a separate machine from your primary
+and secondary validator machines. This machine must be highly available, and
+depending on your needs you may wish to configure etcd with more than just
+one node.
+
+First install `etcd` as desired for your machine. Then TLS certificates must be
+created for authentication between the etcd cluster and your validator.  Here is
+one way to do this:
+
+With [Golang](https://golang.org/) installed, run `go get
+github.com/cloudflare/cfssl/cmd/cfssl`.  The `cfssl` program should now be
+available at `~/go/bin/cfssl`.  Ensure `~/go/bin` is in your PATH by running
+`PATH=$PATH:~/go/bin/`.
+
+Now create a certificate directory and configuration file:
+```
+mkdir -p certs/
+echo '{"CN":"etcd","hosts":["localhost", "127.0.0.1"],"key":{"algo":"rsa","size":2048}}' > certs/config.json
+```
+
+then create certificates for the etcd server and the validator:
+```
+cfssl gencert -initca certs/config.json | cfssljson -bare certs/etcd-ca
+cfssl gencert -ca certs/etcd-ca.pem -ca-key certs/etcd-ca-key.pem certs/config.json | cfssljson -bare certs/validator
+cfssl gencert -ca certs/etcd-ca.pem -ca-key certs/etcd-ca-key.pem certs/config.json | cfssljson -bare certs/etcd
+```
+
+Copy these files to your primary and secondary validator machines:
+* `certs/validator-key.pem`
+* `certs/validator.pem`
+* `certs/etcd-ca.pem`
+
+and these files to the machine running the etcd server:
+* `certs/etcd.pem`
+* `certs/etcd-key.pem`
+* `certs/etcd-ca.pem`
+
+With this configuration, both the validator and etdc will share the same
+TLS certificate authority and will each authenticate the other with it.
+
+
+Start `etcd` with the following arguments:
+```bash
+etcd --auto-compaction-retention 2 --auto-compaction-mode revision \
+  --cert-file=certs/etcd.pem --key-file=certs/etcd-key.pem \
+  --client-cert-auth \
+  --trusted-ca-file=certs/etcd-ca.pem \
+  --listen-client-urls=https://127.0.0.1:2379 \
+  --advertise-client-urls=https://127.0.0.1:2379
+```
+
+and use `curl` to confirm the etcd TLS certificates are properly configured:
+```bash
+curl --cacert certs/etcd-ca.pem https://127.0.0.1:2379/ --cert certs/validator.pem --key certs/validator-key.pem
+```
+On success, curl will return a 404 response.
+
+For more information on etcd TLS setup, please refer to
+https://etcd.io/docs/v3.5/op-guide/security/#example-2-client-to-server-authentication-with-https-client-certificates
+
+### Primary Validator
+The following additional `solana-validator` parameters are required to enable
+tower storage into etcd:
+
+```
+solana-validator ... \
+  --tower-storage etcd \
+  --etcd-cacert-file certs/etcd-ca.pem \
+  --etcd-cert-file certs/validator.pem \
+  --etcd-key-file certs/validator-key.pem \
+  --etcd-endpoint 127.0.0.1:2379  # <-- replace 127.0.0.1 with the actual IP address
+```
+
+Note that once running your validator *will terminate* if it's not able to write
+its tower into etcd before submitting a vote transactioin, so it's essential
+that your etcd endpoint remain accessible at all times.
+
+### Secondary Validator
+Configure the secondary validator like the primary with the exception of the
+following `solana-validator` command-line argument changes:
+* Generate and use a secondary validator identity: `--identity secondary-validator-keypair.json`
+* Add `--no-check-vote-account`
+* Add `--authorized-voter validator-keypair.json` (where
+  `validator-keypair.json` is the identity keypair for your primary validator)
+
+## Triggering a failover manually
+When both validators are running normally and caught up to the cluster, a
+failover from primary to secondary can be triggered by running the following
+command on the secondary validator:
+```bash
+$ solana-validator wait-for-restart-window --identity validator-keypair.json \
+  && solana-validator set-identity validator-keypair.json
+```
+
+The secondary validator will acquire a lock on the tower in etcd to ensure
+voting and block production safely switches over from the primary validator.
+
+The primary validator will then terminate as soon as it detects the secondary
+validator using its identity.
+
+Note: When the primary validator restarts (which may be immediate if you have
+configured your primary validator to do so) it will reclaim its identity
+from the secondary validator. This will in turn cause the secondary validator to
+exit. However if/when the secondary validator restarts, it will do so using the
+secondary validator identity and thus the restart cycle is broken.
+
+## Triggering a failover via monitoring
+Monitoring of your choosing can invoke the `solana-validator set-identity
+validator-keypair.json` command mentioned in the previous section.
+
+It is not necessary to guarantee the primary validator has halted before failing
+over to the secondary, as the failover process will prevent the primary
+validator from voting and producing blocks even if it is in an unknown state.
+
+## Validator Software Upgrades
+To perform a software upgrade using this failover method:
+1. Install the new software version on your primary validator system but do not
+   restart it yet.
+2. Trigger a manual failover to your secondary validator. This should cause your
+   primary validator to terminate.
+3. When your primary validator restarts it will now be using the new software version.
+4. Once the primary validator catches up upgrade the secondary validator at
+   your convenience.

package/src/running-validator/validator-info.md

@@ -0,0 +1,64 @@
+---
+title: Publishing Validator Info
+---
+
+You can publish your validator information to the chain to be publicly visible to other users.
+
+## Run solana validator-info
+
+Run the solana CLI to populate a validator info account:
+
+```bash
+solana validator-info publish --keypair ~/validator-keypair.json <VALIDATOR_INFO_ARGS> <VALIDATOR_NAME>
+```
+
+For details about optional fields for VALIDATOR_INFO_ARGS:
+
+```bash
+solana validator-info publish --help
+```
+
+## Example Commands
+
+Example publish command:
+
+```bash
+solana validator-info publish "Elvis Validator" -n elvis -w "https://elvis-validates.com"
+```
+
+Example query command:
+
+```bash
+solana validator-info get
+```
+
+which outputs
+
+```text
+Validator info from 8WdJvDz6obhADdxpGCiJKZsDYwTLNEDFizayqziDc9ah
+  Validator pubkey: 6dMH3u76qZ7XG4bVboVRnBHR2FfrxEqTTTyj4xmyDMWo
+  Info: {"keybaseUsername":"elvis","name":"Elvis Validator","website":"https://elvis-validates.com"}
+```
+
+## Keybase
+
+Including a Keybase username allows client applications \(like the Solana
+Network Explorer\) to automatically pull in your validator public profile,
+including cryptographic proofs, brand identity, etc. To connect your validator
+pubkey with Keybase:
+
+1. Join [https://keybase.io/](https://keybase.io/) and complete the profile for your validator
+2. Add your validator **identity pubkey** to Keybase:
+
+   - Create an empty file on your local computer called `validator-<PUBKEY>`
+   - In Keybase, navigate to the Files section, and upload your pubkey file to
+
+     a `solana` subdirectory in your public folder: `/keybase/public/<KEYBASE_USERNAME>/solana`
+
+   - To check your pubkey, ensure you can successfully browse to
+
+     `https://keybase.pub/<KEYBASE_USERNAME>/solana/validator-<PUBKEY>`
+
+3. Add or update your `solana validator-info` with your Keybase username. The
+
+   CLI will verify the `validator-<PUBKEY>` file

package/src/running-validator/validator-monitor.md

@@ -0,0 +1,47 @@
+---
+title: Monitoring a Validator
+---
+
+## Check Gossip
+
+Confirm the IP address and **identity pubkey** of your validator is visible in
+the gossip network by running:
+
+```bash
+solana gossip
+```
+
+## Check Your Balance
+
+Your account balance should decrease by the transaction fee amount as your
+validator submits votes, and increase after serving as the leader. Pass the
+`--lamports` are to observe in finer detail:
+
+```bash
+solana balance --lamports
+```
+
+## Check Vote Activity
+
+The `solana vote-account` command displays the recent voting activity from
+your validator:
+
+```bash
+solana vote-account ~/vote-account-keypair.json
+```
+
+## Get Cluster Info
+
+There are several useful JSON-RPC endpoints for monitoring your validator on the
+cluster, as well as the health of the cluster:
+
+```bash
+# Similar to solana-gossip, you should see your validator in the list of cluster nodes
+curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getClusterNodes"}' http://api.devnet.solana.com
+# If your validator is properly voting, it should appear in the list of `current` vote accounts. If staked, `stake` should be > 0
+curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getVoteAccounts"}' http://api.devnet.solana.com
+# Returns the current leader schedule
+curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getLeaderSchedule"}' http://api.devnet.solana.com
+# Returns info about the current epoch. slotIndex should progress on subsequent calls.
+curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getEpochInfo"}' http://api.devnet.solana.com
+```

package/src/running-validator/validator-reqs.md

@@ -0,0 +1,115 @@
+---
+title: Validator Requirements
+---
+
+## Minimum SOL requirements
+
+There is no strict minimum amount of SOL required to run a validator on Solana.
+
+However in order to participate in consensus, a vote account is required which
+has a rent-exempt reserve of 0.02685864 SOL. Voting also requires sending a vote
+transaction for each block the validator agrees with, which can cost up to
+1.1 SOL per day.
+
+## Hardware Recommendations
+
+- CPU
+  - 12 cores / 24 threads, or more
+  - 2.8GHz, or faster
+  - AVX2 instruction support (to use official release binaries, self-compile
+    otherwise)
+  - Support for AVX512f and/or SHA-NI instructions is helpful
+  - The AMD Zen3 series is popular with the validator community
+- RAM
+  - 128GB, or more
+  - Motherboard with 256GB capacity suggested
+- Disk
+  - PCIe Gen3 x4 NVME SSD, or better
+  - Accounts: 500GB, or larger. High TBW (Total Bytes Written)
+  - Ledger: 1TB or larger. High TBW suggested
+  - OS: (Optional) 500GB, or larger. SATA OK
+  - The OS may be installed on the ledger disk, though testing has shown better
+    performance with the ledger on its own disk
+  - Accounts and ledger _can_ be stored on the same disk, however due to high
+    IOPS, this is not recommended
+  - The Samsung 970 and 980 Pro series SSDs are popular with the validator community
+- GPUs
+  - Not strictly necessary at this time
+  - Motherboard and power supply speced to add one or more high-end GPUs in the
+    future suggested
+
+### RPC Node Recommendations
+The [hardware recommendations](#hardware-recommendations) above should be considered
+bare minimums if the validator is intended to be employed as an RPC node. To provide
+full functionality and improved reliability, the following adjustments should be
+made.
+
+- CPU
+  - 16 cores / 32 threads, or more
+- RAM
+  - 256 GB, or more
+- Disk
+  - Consider a larger ledger disk if longer transaction history is required
+  - Accounts and ledger should not be stored on the same disk
+
+## Virtual machines on Cloud Platforms
+
+While you can run a validator on a cloud computing platform, it may not
+be cost-efficient over the long term.
+
+However, it may be convenient to run non-voting api nodes on VM instances for
+your own internal usage. This use case includes exchanges and services built on
+Solana.
+
+In fact, the mainnet-beta validators operated by the team are currently
+(Mar. 2021) run on GCE `n2-standard-32` (32 vCPUs, 128 GB memory) instances with
+2048 GB SSD for operational convenience.
+
+For other cloud platforms, select instance types with similar specs.
+
+Also note that egress internet traffic usage may turn out to be high,
+especially for the case of running staked validators.
+
+## Docker
+
+Running validator for live clusters (including mainnet-beta) inside Docker is
+not recommended and generally not supported. This is due to concerns of general
+Docker's containerzation overhead and resultant performance degradation unless
+specially configured.
+
+We use Docker only for development purposes. Docker Hub contains images for all
+releases at [solanalabs/solana](https://hub.docker.com/r/solanalabs/solana).
+
+## Software
+
+- We build and run on Ubuntu 20.04.
+- See [Installing Solana](../cli/install-solana-cli-tools.md) for the current Solana software release.
+
+Prebuilt binaries are available for Linux x86_64 on CPUs supporting AVX2 \(Ubuntu 20.04 recommended\).
+MacOS or WSL users may build from source.
+
+## Networking
+Internet service should be at least 300Mbit/s symmetric, commercial. 1GBit/s preferred
+
+### Port Forwarding
+The following ports need to be open to the internet for both inbound and outbound
+
+It is not recommended to run a validator behind a NAT. Operators who choose to
+do so should be comfortable configuring their networking equipment and debugging
+any traversal issues on their own.
+
+#### Required
+- 8000-10000 TCP/UDP - P2P protocols (gossip, turbine, repair, etc). This can
+be limited to any free 11 port range with `--dynamic-port-range`
+
+#### Optional
+For security purposes, it is not suggested that the following ports be open to
+the internet on staked, mainnet-beta validators.
+- 8899 TCP - JSONRPC over HTTP. Change with `--rpc-port RPC_PORT``
+- 8900 TCP - JSONRPC over Websockets. Derived. Uses `RPC_PORT + 1`
+
+## GPU Requirements
+
+CUDA is required to make use of the GPU on your system. The provided Solana
+release binaries are built on Ubuntu 20.04 with [CUDA Toolkit 10.1 update 1](https://developer.nvidia.com/cuda-toolkit-archive). If your machine is using
+a different CUDA version then you will need to rebuild from source.

package/src/running-validator/validator-stake.md

@@ -0,0 +1,132 @@
+---
+title: Staking
+---
+
+**By default your validator will have no stake.** This means it will be
+ineligible to become leader.
+
+## Monitoring Catch Up
+
+To delegate stake, first make sure your validator is running and has caught up
+to the cluster. It may take some time to catch up after your validator boots.
+Use the `catchup` command to monitor your validator through this process:
+
+```bash
+solana catchup ~/validator-keypair.json
+```
+
+Until your validator has caught up, it will not be able to vote successfully and
+stake cannot be delegated to it.
+
+Also if you find the cluster's slot advancing faster than yours, you will likely
+never catch up. This typically implies some kind of networking issue between
+your validator and the rest of the cluster.
+
+## Create Stake Keypair
+
+If you haven’t already done so, create a staking keypair. If you have completed
+this step, you should see the “validator-stake-keypair.json” in your Solana
+runtime directory.
+
+```bash
+solana-keygen new -o ~/validator-stake-keypair.json
+```
+
+## Delegate Stake
+
+Now delegate 1 SOL to your validator by first creating your stake account:
+
+```bash
+solana create-stake-account ~/validator-stake-keypair.json 1
+```
+
+and then delegating that stake to your validator:
+
+```bash
+solana delegate-stake ~/validator-stake-keypair.json ~/vote-account-keypair.json
+```
+
+> Don’t delegate your remaining SOL, as your validator will use those tokens to vote.
+
+Stakes can be re-delegated to another node at any time with the same command,
+but only one re-delegation is permitted per epoch:
+
+```bash
+solana delegate-stake ~/validator-stake-keypair.json ~/some-other-vote-account-keypair.json
+```
+
+Assuming the node is voting, now you're up and running and generating validator
+rewards. Rewards are paid automatically on epoch boundaries.
+
+The rewards lamports earned are split between your stake account and the vote
+account according to the commission rate set in the vote account. Rewards can
+only be earned while the validator is up and running. Further, once staked, the
+validator becomes an important part of the network. In order to safely remove a
+validator from the network, first deactivate its stake.
+
+At the end of each slot, a validator is expected to send a vote transaction.
+These vote transactions are paid for by lamports from a validator's identity
+account.
+
+This is a normal transaction so the standard transaction fee will apply. The
+transaction fee range is defined by the genesis block. The actual fee will
+fluctuate based on transaction load. You can determine the current fee via the
+[RPC API “getRecentBlockhash”](developing/clients/jsonrpc-api.md#getrecentblockhash)
+before submitting a transaction.
+
+Learn more about [transaction fees here](../implemented-proposals/transaction-fees.md).
+
+## Validator Stake Warm-up
+
+To combat various attacks on consensus, new stake delegations are subject to
+a [warm-up](/staking/stake-accounts#delegation-warmup-and-cooldown)
+period.
+
+Monitor a validator's stake during warmup by:
+
+- View your vote account:`solana vote-account ~/vote-account-keypair.json` This displays the current state of all the votes the validator has submitted to the network.
+- View your stake account, the delegation preference and details of your stake:`solana stake-account ~/validator-stake-keypair.json`
+- `solana validators` displays the current active stake of all validators, including yours
+- `solana stake-history` shows the history of stake warming up and cooling down over recent epochs
+- Look for log messages on your validator indicating your next leader slot: `[2019-09-27T20:16:00.319721164Z INFO solana_core::replay_stage] <VALIDATOR_IDENTITY_PUBKEY> voted and reset PoH at tick height ####. My next leader slot is ####`
+- Once your stake is warmed up, you will see a stake balance listed for your validator by running `solana validators`
+
+## Monitor Your Staked Validator
+
+Confirm your validator becomes a [leader](../terminology.md#leader)
+
+- After your validator is caught up, use the `solana balance` command to monitor the earnings as your validator is selected as leader and collects transaction fees
+- Solana nodes offer a number of useful JSON-RPC methods to return information about the network and your validator's participation. Make a request by using curl \(or another http client of your choosing\), specifying the desired method in JSON-RPC-formatted data. For example:
+
+```bash
+  // Request
+  curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getEpochInfo"}' http://localhost:8899
+
+  // Result
+  {"jsonrpc":"2.0","result":{"epoch":3,"slotIndex":126,"slotsInEpoch":256},"id":1}
+```
+
+Helpful JSON-RPC methods:
+
+- `getEpochInfo`[An epoch](../terminology.md#epoch) is the time, i.e. number of [slots](../terminology.md#slot), for which a [leader schedule](../terminology.md#leader-schedule) is valid. This will tell you what the current epoch is and how far into it the cluster is.
+- `getVoteAccounts` This will tell you how much active stake your validator currently has. A % of the validator's stake is activated on an epoch boundary. You can learn more about staking on Solana [here](../cluster/stake-delegation-and-rewards.md).
+- `getLeaderSchedule` At any given moment, the network expects only one validator to produce ledger entries. The [validator currently selected to produce ledger entries](../cluster/leader-rotation.md#leader-rotation) is called the “leader”. This will return the complete leader schedule \(on a slot-by-slot basis\) for currently activated stake, the identity pubkey will show up 1 or more times here.
+
+## Deactivating Stake
+
+Before detaching your validator from the cluster, you should deactivate the
+stake that was previously delegated by running:
+
+```bash
+solana deactivate-stake ~/validator-stake-keypair.json
+```
+
+Stake is not deactivated immediately and instead cools down in a similar fashion
+as stake warm up. Your validator should remain attached to the cluster while
+the stake is cooling down. While cooling down, your stake will continue to earn
+rewards. Only after stake cooldown is it safe to turn off your validator or
+withdraw it from the network. Cooldown may take several epochs to complete,
+depending on active stake and the size of your stake.
+
+Note that a stake account may only be used once, so after deactivation, use the
+cli's `withdraw-stake` command to recover the previously staked lamports.