solana-docs 0.0.0

package/src/proposals/accounts-db-replication.md

@@ -0,0 +1,185 @@
+---
+title: AccountsDB Replication for RPC Services
+---
+
+## Problem
+
+Validators fall behind the network when bogged down by heavy RPC load. This
+seems to be due to a combination of CPU load and lock contention caused by
+serving RPC requests. The most expensive RPC requests involve account scans.
+
+## Solution Overview
+
+AccountsDB `replicas` that run separately from the main validator can be used to
+offload account-scan requests. Replicas would only: request and pull account
+updates from the validator, serve client account-state RPC requests, and manage
+AccountsDb and AccountsBackgroundService clean + shrink.
+
+The replica communicates to the main validator via a new RPC mechanism to fetch
+metadata information about the replication and the accounts update from the validator.
+The main validator supports only one replica node. A replica node can relay the
+information to 1 or more other replicas forming a replication tree.
+
+At the initial start of the replica node, it downloads the latest snapshot
+from a validator and constructs the bank and AccountsDb from it. After that, it queries
+its main validator for new slots and requests the validator to send the updated
+accounts for that slot and update to its own AccountsDb.
+
+The same RPC replication mechanism can be used between a replica to another replica.
+This requires the replica to serve both the client and server in the replication model.
+
+On the client RPC serving side, `JsonRpcAccountsService` is responsible for serving
+the client RPC calls for accounts related information similar to the existing
+`JsonRpcService`.
+
+The replica will also take snapshots periodically so that it can start quickly after
+a restart if the snapshot is not too old.
+
+## Detailed Solution
+The following sections provide more details of the design.
+
+### Consistency Model
+The AccountsDb information is replicated asynchronously from the main validator to the replica.
+When a query against the replica's AccountsDb is made, the replica may not have the latest
+information of the latest slot. In this regard, it will eventually be consistent. However, for
+a particular slot, the information provided is consistent with that of its peer validator
+for commitment levels confirmed and finalized. For V1, we only support queries at these two
+levels.
+
+### Solana Replica Node
+A new node named solana-replica-node will be introduced whose main responsibility is to maintain
+the AccountsDb replica. The RPC node or replica node is used interchangeably in this document.
+It will be a separate executable from the validator.
+
+The replica consists of the following major components:
+
+The `ReplicaSlotConfirmationRequestor`: this service is responsible for periodically sending the
+request `ReplicaSlotConfirmationRequest` to its peer validator or replica for the latest slots.
+It specifies the latest slot (last_replicated_slot) for which the replica has already
+fetched the accounts information for. This maintains the ReplWorkingSlotSet and manages
+the lifecycle of BankForks, BlockCommitmentCache (for the highest confirmed slot) and
+the optimistically confirmed bank.
+
+The `ReplicaSlotConfirmationServer`: this service is responsible for serving the
+`ReplicaSlotConfirmationRequest` and sends the `ReplicaSlotConfirmationResponse` back to the requestor.
+The response consists of a vector of new slots the validator knows of which is later than the
+specified last_replicated_slot. This service also runs in the main validator. This service
+gets the slots for replication from the BankForks, BlockCommitmentCache and OptimiscallyConfirmBank.
+
+The `ReplicaAccountsRequestor`: this service is responsible for sending the request
+`ReplicaAccountsRequest` to its peer validator or replica for the `ReplicaAccountInfo` for a
+slot for which it has not completed accounts db replication. The `ReplicaAccountInfo` contains
+the `ReplicaAccountMeta`, Hash and the AccountData. The `ReplicaAccountMeta` contains info about
+the existing `AccountMeta` in addition to the account data length in bytes.
+
+The `ReplicaAccountsServer`: this service is reponsible for serving the `ReplicaAccountsRequest`
+and sends `ReplicaAccountsResponse` to the requestor. The response contains the count of the
+ReplAccountInfo and the vector of ReplAccountInfo. This service runs both in the validator
+and the replica relaying replication information. The server can stream the account information
+from its AccountCache or from the storage if already flushed. This is similar to how a snapshot
+package is created from the AccountsDb with the difference that the storage does not need to be
+flushed to the disk before streaming to the client. If the account data is in the cache, it can
+be directly streamed. Care must be taken to avoid account data for a slot being cleaned while
+serving the streaming. When attempting a replication of a slot, if the slot is already cleaned
+up with accounts data cleaned as result of update in later rooted slots, the replica should
+forsake this slot and try the later uncleaned root slot.
+
+During replication we also need to replicate the information of accounts that have been cleaned
+up due to zero lamports, i.e. we need to be able to tell the difference between an account in a
+given slot which was not updated and hence has no storage entry in that slot, and one that
+holds 0 lamports and has been cleaned up through the history. We may record this via some
+"Tombstone" mechanism -- recording the dead accounts cleaned up fora slot. The tombstones
+themselves can be removed after exceeding the retention period expressed as epochs. Any
+attempt to replicate slots with tombstones removed will fail and the replica should skip
+this slot and try later ones.
+
+The `JsonRpcAccountsService`: this is the RPC service serving client requests for account
+information. The existing JsonRpcService serves other client calls than AccountsDb ones.
+The replica node only serves the AccountsDb calls.
+
+The existing JsonRpcService requires `BankForks`, `OptimisticallyConfirmedBank` and
+`BlockCommitmentCache` to load the Bank. The JsonRpcAccountsService will need to use
+information obtained from ReplicaSlotConfirmationResponse to construct the AccountsDb.
+
+The `AccountsBackgroundService`: this service also runs in the replica which is responsible
+for taking snapshots periodically and shrinking the AccountsDb and doing accounts cleaning.
+The existing code also uses BankForks which we need to keep in the replica.
+
+### Compatibility Consideration
+
+For protocol compatibility considerations, all the requests have the replication version which
+is initially set to 1. Alternatively, we can use the validator's version. The RPC server side
+shall check the request version and fail if it is not supported.
+
+### Replication Setup
+To limit adverse effects on the validator and the replica due to replication, they can be
+configured with a list of replica nodes which can form a replication pair with it. And the
+replica node is configured with the validator which can serve its requests.
+
+
+### Fault Tolerance
+The main responsibility of making sure the replication is tolerant of faults lies with the
+replica. In case of request failures, the replica shall retry the requests.
+
+
+### Interface
+
+Following are the client RPC APIs supported by the replica node in JsonRpcAccountsService.
+
+- getAccountInfo
+- getBlockCommitment
+- getMultipleAccounts
+- getProgramAccounts
+- getMinimumBalanceForRentExemption
+- getInflationGovenor
+- getInflationRate
+- getEpochSchedule
+- getRecentBlockhash
+- getFees
+- getFeeCalculatorForBlockhash
+- getFeeRateGovernor
+- getLargestAccounts
+- getSupply
+- getStakeActivation
+- getTokenAccountBalance
+- getTokenSupply
+- getTokenLargestAccounts
+- getTokenAccountsByOwner
+- getTokenAccountsByDelegate
+
+Following APIs are not included:
+
+- getInflationReward
+- getClusterNodes
+- getRecentPerformanceSamples
+- getGenesisHash
+- getSignatueStatuses
+- getMaxRetransmitSlot
+- getMaxShredInsertSlot
+- sendTransaction
+- simulateTransaction
+- getSlotLeader
+- getSlotLeaders
+- minimumLedgerSlot
+- getBlock
+- getBlockTime
+- getBlocks
+- getBlocksWithLimit
+- getTransaction
+- getSignaturesForAddress
+- getFirstAvailableBlock
+- getBlockProduction
+
+
+Action Items
+
+1. Build the replica framework and executable
+2. Integrate snapshot restore code for bootstrap the AccountsDb.
+3. Develop the ReplicaSlotConfirmationRequestor and ReplicaSlotConfirmationServer interface code
+4. Develop the ReplicaSlotConfirmationRequestor and ReplicaSlotConfirmationServer detailed implementations: managing the ReplEligibleSlotSet lifecycle: adding new roots and deleting root to it. And interfaces managing ReplWorkingSlotSet interface: adding and removing. Develop component synthesising information from BankForks, BlockCommitmentCache and OptimistcallyConfirmedBank on the server side and maintaining information on the client side.
+5. Develop the interface code for ReplicaAccountsRequestor and ReplicaAccountsServer
+6. Develop detailed implementation for ReplicaAccountsRequestor and ReplicaAccountsServer and develop the replication account storage serializer and deserializer.
+7. Develop the interface code JsonRpcAccountsService
+8. Detailed Implementation of JsonRpcAccountsService, refactor code to share with part of JsonRpcService.
+9. Integrate with the AccountsBackgroundService in the replica for shrinking, cleaning, snapshotting.
+10. Metrics and performance testing

package/src/proposals/bankless-leader.md

@@ -0,0 +1,57 @@
+---
+title: Bankless Leader
+---
+
+A bankless leader does the minimum amount of work to produce a valid block. The leader is tasked with ingress transactions, sorting and filtering valid transactions, arranging them into entries, shredding the entries and broadcasting the shreds. While a validator only needs to reassemble the block and replay execution of well formed entries. The leader does 3x more memory operations before any bank execution than the validator per processed transaction.
+
+## Rationale
+
+Normal bank operation for a spend needs to do 2 loads and 2 stores. With this design leader just does 1 load. so 4x less account_db work before generating the block. The store operations are likely to be more expensive than reads.
+
+When replay stage starts processing the same transactions, it can assume that PoH is valid, and that all the entries are safe for parallel execution. The fee accounts that have been loaded to produce the block are likely to still be in memory, so the additional load should be warm and the cost is likely to be amortized.
+
+## Fee Account
+
+The [fee account](../terminology.md#fee_account) pays for the transaction to be included in the block. The leader only needs to validate that the fee account has the balance to pay for the fee.
+
+## Balance Cache
+
+For the duration of the leaders consecutive blocks, the leader maintains a temporary balance cache for all the processed fee accounts. The cache is a map of pubkeys to lamports.
+
+At the start of the first block the balance cache is empty. At the end of the last block the cache is destroyed.
+
+The balance cache lookups must reference the same base fork for the entire duration of the cache. At the block boundary, the cache can be reset along with the base fork after replay stage finishes verifying the previous block.
+
+## Balance Check
+
+Prior to the balance check, the leader validates all the signatures in the transaction.
+
+1. Verify the accounts are not in use and BlockHash is valid.
+2. Check if the fee account is present in the cache, or load the account from accounts_db and store the lamport balance in the cache.
+3. If the balance is less than the fee, drop the transaction.
+4. Subtract the fee from the balance.
+5. For all the keys in the transaction that are Credit-Debit and are referenced by an instruction, reduce their balance to 0 in the cache. The account fee is declared as Credit-Debit, but as long as it is not used in any instruction its balance will not be reduced to 0.
+
+## Leader Replay
+
+Leaders will need to replay their blocks as part of the standard replay stage operation.
+
+## Leader Replay With Consecutive Blocks
+
+A leader can be scheduled to produce multiple blocks in a row. In that scenario the leader is likely to be producing the next block while the replay stage for the first block is playing.
+
+When the leader finishes the replay stage it can reset the balance cache by clearing it, and set a new fork as the base for the cache which can become active on the next block.
+
+## Resetting the Balance Cache
+
+1. At the start of the block, if the balance cache is uninitialized, set the base fork for the balance cache to be the parent of the block and create an empty cache.
+2. if the cache is initialized, check if block's parents has a new frozen bank that is newer than the current base fork for the balance cache.
+3. if a parent newer than the cache's base fork exist, reset the cache to the parent.
+
+## Impact on Clients
+
+The same fee account can be reused many times in the same block until it is used once as Credit-Debit by an instruction.
+
+Clients that transmit a large number of transactions per second should use a dedicated fee account that is not used as Credit-Debit in any instruction.
+
+Once an account fee is used as Credit-Debit, it will fail the balance check until the balance cache is reset.

package/src/proposals/block-confirmation.md

@@ -0,0 +1,85 @@
+---
+title: Block Confirmation
+---
+
+A validator votes on a PoH hash for two purposes. First, the vote indicates it
+believes the ledger is valid up until that point in time. Second, since many
+valid forks may exist at a given height, the vote also indicates exclusive
+support for the fork. This document describes only the former. The latter is
+described in [Tower BFT](../implemented-proposals/tower-bft.md).
+
+## Current Design
+
+To start voting, a validator first registers an account to which it will send
+its votes. It then sends votes to that account. The vote contains the tick
+height of the block it is voting on. The account stores the 32 highest heights.
+
+### Problems
+
+- Only the validator knows how to find its own votes directly.
+
+  Other components, such as the one that calculates confirmation time, needs to
+  be baked into the validator code. The validator code queries the bank for all
+  accounts owned by the vote program.
+
+- Voting ballots do not contain a PoH hash. The validator is only voting that
+  it has observed an arbitrary block at some height.
+
+- Voting ballots do not contain a hash of the bank state. Without that hash,
+  there is no evidence that the validator executed the transactions and
+  verified there were no double spends.
+
+## Proposed Design
+
+### No Cross-block State Initially
+
+At the moment a block is produced, the leader shall add a NewBlock transaction
+to the ledger with a number of tokens that represents the validation reward.
+It is effectively an incremental multisig transaction that sends tokens from
+the mining pool to the validators. The account should allocate just enough
+space to collect the votes required to achieve a supermajority. When a
+validator observes the NewBlock transaction, it has the option to submit a vote
+that includes a hash of its ledger state (the bank state). Once the account has
+sufficient votes, the vote program should disperse the tokens to the
+validators, which causes the account to be deleted.
+
+#### Logging Confirmation Time
+
+The bank will need to be aware of the vote program. After each transaction, it
+should check if it is a vote transaction and if so, check the state of that
+account. If the transaction caused the supermajority to be achieved, it should
+log the time since the NewBlock transaction was submitted.
+
+### Finality and Payouts
+
+[Tower BFT](../implemented-proposals/tower-bft.md) is the proposed fork selection algorithm. It proposes
+that payment to miners be postponed until the _stack_ of validator votes reaches
+a certain depth, at which point rollback is not economically feasible. The vote
+program may therefore implement Tower BFT. Vote instructions would need to
+reference a global Tower account so that it can track cross-block state.
+
+## Challenges
+
+### On-chain voting
+
+Using programs and accounts to implement this is a bit tedious. The hardest
+part is figuring out how much space to allocate in NewBlock. The two variables
+are the _active set_ and the stakes of those validators. If we calculate the
+active set at the time NewBlock is submitted, the number of validators to
+allocate space for is known upfront. If, however, we allow new validators to
+vote on old blocks, then we'd need a way to allocate space dynamically.
+
+Similar in spirit, if the leader caches stakes at the time of NewBlock, the
+vote program doesn't need to interact with the bank when it processes votes. If
+we don't, then we have the option to allow stakes to float until a vote is
+submitted. A validator could conceivably reference its own staking account, but
+that'd be the current account value instead of the account value of the most
+recently finalized bank state. The bank currently doesn't offer a means to
+reference accounts from particular points in time.
+
+### Voting Implications on Previous Blocks
+
+Does a vote on one height imply a vote on all blocks of lower heights of
+that fork? If it does, we'll need a way to lookup the accounts of all
+blocks that haven't yet reached supermajority. If not, the validator could
+send votes to all blocks explicitly to get the block rewards.

package/src/proposals/cluster-test-framework.md

@@ -0,0 +1,102 @@
+---
+title: Cluster Test Framework
+---
+
+This document proposes the Cluster Test Framework \(CTF\). CTF is a test harness that allows tests to execute against a local, in-process cluster or a deployed cluster.
+
+## Motivation
+
+The goal of CTF is to provide a framework for writing tests independent of where and how the cluster is deployed. Regressions can be captured in these tests and the tests can be run against deployed clusters to verify the deployment. The focus of these tests should be on cluster stability, consensus, fault tolerance, API stability.
+
+Tests should verify a single bug or scenario, and should be written with the least amount of internal plumbing exposed to the test.
+
+## Design Overview
+
+Tests are provided an entry point, which is a `contact_info::ContactInfo` structure, and a keypair that has already been funded.
+
+Each node in the cluster is configured with a `validator::ValidatorConfig` at boot time. At boot time this configuration specifies any extra cluster configuration required for the test. The cluster should boot with the configuration when it is run in-process or in a data center.
+
+Once booted, the test will discover the cluster through a gossip entry point and configure any runtime behaviors via validator RPC.
+
+## Test Interface
+
+Each CTF test starts with an opaque entry point and a funded keypair. The test should not depend on how the cluster is deployed, and should be able to exercise all the cluster functionality through the publicly available interfaces.
+
+```text
+use crate::contact_info::ContactInfo;
+use solana_sdk::signature::{Keypair, Signer};
+pub fn test_this_behavior(
+    entry_point_info: &ContactInfo,
+    funding_keypair: &Keypair,
+    num_nodes: usize,
+)
+```
+
+## Cluster Discovery
+
+At test start, the cluster has already been established and is fully connected. The test can discover most of the available nodes over a few second.
+
+```text
+use crate::gossip_service::discover_nodes;
+
+// Discover the cluster over a few seconds.
+let cluster_nodes = discover_nodes(&entry_point_info, num_nodes);
+```
+
+## Cluster Configuration
+
+To enable specific scenarios, the cluster needs to be booted with special configurations. These configurations can be captured in `validator::ValidatorConfig`.
+
+For example:
+
+```text
+let mut validator_config = ValidatorConfig::default();
+let local = LocalCluster::new_with_config(
+                num_nodes,
+                10_000,
+                100,
+                &validator_config
+                );
+```
+
+## How to design a new test
+
+For example, there is a bug that shows that the cluster fails when it is flooded with invalid advertised gossip nodes. Our gossip library and protocol may change, but the cluster still needs to stay resilient to floods of invalid advertised gossip nodes.
+
+Configure the RPC service:
+
+```text
+let mut validator_config = ValidatorConfig::default();
+validator_config.rpc_config.enable_rpc_gossip_push = true;
+validator_config.rpc_config.enable_rpc_gossip_refresh_active_set = true;
+```
+
+Wire the RPCs and write a new test:
+
+```text
+pub fn test_large_invalid_gossip_nodes(
+    entry_point_info: &ContactInfo,
+    funding_keypair: &Keypair,
+    num_nodes: usize,
+) {
+    let cluster = discover_nodes(&entry_point_info, num_nodes);
+
+    // Poison the cluster.
+    let client = create_client(entry_point_info.client_facing_addr(), VALIDATOR_PORT_RANGE);
+    for _ in 0..(num_nodes * 100) {
+        client.gossip_push(
+            cluster_info::invalid_contact_info()
+        );
+    }
+    sleep(Durration::from_millis(1000));
+
+    // Force refresh of the active set.
+    for node in &cluster {
+        let client = create_client(node.client_facing_addr(), VALIDATOR_PORT_RANGE);
+        client.gossip_refresh_active_set();
+    }
+
+    // Verify that spends still work.
+    verify_spends(&cluster);
+}
+```

package/src/proposals/comprehensive-compute-fees.md

@@ -0,0 +1,164 @@
+---
+title: Comprehensive Compute Fees
+---
+
+## Motivation
+
+The current fee structure lacks a comprehensive account of the work required by
+a validator to process a transaction.  The fee structure is only based on the
+number of signatures in a transaction but is meant to account for the work that
+the validator must perform to validate each transaction.  The validator performs
+a lot more user-defined work than just signature verification.  Processing a
+transaction typically includes signature verifications, account locking, account
+loading, and instruction processing.
+
+## Proposed Solution
+
+The following solution does not specify what native token costs are to be
+associated with the new fee structure.  Instead, it sets the criteria and
+provides the knobs that a cost model can use to determine those costs.
+
+### Fee
+
+The goal of the fees is to cover the computation cost of processing a
+transaction.  Each of the fee categories below will be represented as a compute
+unit cost that, when added together, encompasses the entire cost of processing
+the transaction.  By calculating the total cost of the transaction, the runtime
+can charge a more representative fee and make better transaction scheduling
+decisions.
+
+A fee will be calculated based on:
+
+1. Number of signatures
+   - Fixed rate per signature
+2. Number of write locks
+   - Fixed rate per writable account
+3. Data byte cost
+   - Fixed rate per byte of the sum of the length all a transactions instruction
+     datas
+4. Account sizes
+   - Account sizes can't be known up-front but can account for a considerable
+     amount of the load the transaction incurs on the network.  The payer will
+     be charged for a maximum account size (10m) upfront and refunded the
+     difference after the actual account sizes are known.
+5. Compute budget
+   - Each transaction will be given a default transaction-wide compute budget of
+     200k units with the option of requesting a larger budget via a compute
+     budget instruction up to a maximum of 1m units.  This budget is used to
+     limit the time it takes to process a transaction.  The compute budget
+     portion of the fee will be charged up-front based on the default or
+     requested amount.  After processing, the actual number of units consumed
+     will be known, and the payer will be refunded the difference, so the payer
+     only pays for what they used.  Builtin programs will have a fixed cost
+     while BPF program's cost will be measured at runtime.
+6. Precompiled programs
+   - Precompiled programs are performing compute-intensive operations.  The work
+     incurred by a precompiled program is predictable based on the instruction's
+     data array.  Therefore a cost will be assigned per precompiled program
+     based on the parsing of instruction data.  Because precompiled programs are
+     processed outside of the bank, their compute cost will not be reflected in
+     the compute budget and will not be used in transaction scheduling
+     decisions. The methods used to determine the fixed cost of the components
+     above are described in
+     [#19627](https://github.com/solana-labs/solana/issues/19627)
+
+### Cost model
+
+The cost model is used to assess what load a transaction will incur during
+in-slot processing and then make decisions on how to best schedule transaction
+into batches.
+
+The cost model's criteria are identical to the fee's criteria except for
+signatures and precompiled programs.  These two costs are incurred before a
+transaction is scheduled and therefore do not affect how long a transaction
+takes within a slot to process.
+
+### Cache account sizes and use them instead of the max
+
+https://github.com/solana-labs/solana/issues/20511
+
+### Transaction-wide compute caps
+
+The current compute budget caps are independently applied to each instruction
+within a transaction. This means the overall transaction cap varies depending on
+how many instructions are in the transaction.  To more accurately schedule a
+transaction, the compute budget will be applied transaction-wide.  One challenge
+of the transaction-wide cap is that each instruction (program) can no longer
+expect to be given an equal amount of compute units.  Each instruction will be
+given the remaining units left over after processing earlier instructions.  This
+will provide some additional tuning and composability challenges for developers.
+
+### Requestable compute budget caps and heap sizes
+
+The precompiled
+[ComputeBudget](https://github.com/solana-labs/solana/blob/00929f836348d76cb3503d0ba5f76f0d275bcc66/sdk/src/compute_budget.rs#L34)
+program can be used to request higher transaction-wide compute budget caps and
+program heap sizes.  The requested increases will be reflected in the
+transaction's fee.
+
+### Fees for precompiled program failures
+
+https://github.com/solana-labs/solana/issues/20481
+
+### Rate governing
+
+Current rate governing needs to be re-assessed.  Fees are being rate
+governed down to their minimums because the number of signatures in each slot is
+far lower than the "target" signatures per slot.
+
+Instead of using the number of signatures to rate govern, the cost model will
+feed back information based on the batch/queue load it is seeing.  The fees will
+sit at a target rate and only increase if the load goes above a specified but to
+be determined threshold.  The governing will be applied across all the fee
+criteria.
+
+### Deterministic fees
+
+Solana's fees are currently deterministic based on a given blockhash.  This
+determinism is a nice feature that simplifies client interactions.  An example
+is when draining an account that is also the payer, the transaction issuer can
+pre-compute the fee and then set the entire remaining balance to be transferred
+out without worrying that the fee will change leaving a very small amount
+remaining in the account.  Another example is for offline signing, the payer
+signer can guarantee what fee that will be charged for the transaction based on
+the nonce's blockhash.
+
+Determinism is achieved in two ways:
+- blockhash queue contains a list of recent (<=~2min) blockhashes and a
+  `lamports_per_signature` value.  The blockhash queue is one of the snapshot's
+  serialized members and thus bank hash depends on it.
+- Nonce accounts used for offline signing contain a `lamports_per_signature`
+  value in its account data
+
+In both cases, when a transaction is assessed a fee, the
+`lamports_per_signature` to use is looked up (either in the queue or in the
+nonce account's data) using the transaction's blockhash.
+
+This currently comes with the following challenges:
+- Exposing the `FeeCalculator` object to the clients (holds the
+  `lamports_per_signature`) makes it hard to evolve the fee criteria due to
+  backward-compatibility.  This issue is being solved by deprecating the
+  `FeeCalculator` object and instead the new apis take a message and return a
+  fee.
+- Blockhash queue entries contain the fee criteria specifics and are part of the
+  bankhash so evolving the fees over time involves more work/risk
+- Nonce accounts store the fee criteria directly in their account data so
+  evolving the fees over time requires changes to nonce account data and data
+  size.
+
+Two solutions to the latter two challenges
+- Get rid of the concept of deterministic fees.  Clients ask via RPC to
+  calculate the current fee estimate and the actual fee is assessed when the
+  transaction is processed.  Fee changes will be governed and change slowly
+  based on network load so the fee differences will be small within the 2min
+  window.  Nonce accounts no longer store the fee criteria but instead a fee
+  cap.  If the assessed fee at the time of processing exceeds the cap then the
+  transaction fails.  This solution removes fee criteria entirely from the
+  blockhash queue and nonce accounts and removes the need for either of those to
+  evolve if there is a need for fee criteria to evolve.
+- Retain the concept of deterministic fees.  Clients ask via RPC to calculate
+  the current fee and pass in a blockhash that fee will be associated with.
+  Blockhash queue and nonce accounts switch to a versioned but internal "Fee"
+  object (similar to "FeeCalculator").  Each time there is a need for fees to
+  evolve the fee object will add a new version and new blockhash queue entries
+  and new nonce accounts will use the new version.

package/src/proposals/embedding-move.md

@@ -0,0 +1,37 @@
+---
+title: Embedding the Move Language
+---
+
+## Problem
+
+Solana enables developers to write on-chain programs in general purpose programming languages such as C or Rust, but those programs contain Solana-specific mechanisms. For example, there isn't another chain that asks developers to create a Rust module with a `process_instruction(KeyedAccounts)` function. Whenever practical, Solana should offer application developers more portable options.
+
+Until just recently, no popular blockchain offered a language that could expose the value of Solana's massively parallel [runtime](../validator/runtime.md). Solidity contracts, for example, do not separate references to shared data from contract code, and therefore need to be executed serially to ensure deterministic behavior. In practice we see that the most aggressively optimized EVM-based blockchains all seem to peak out around 1,200 TPS - a small fraction of what Solana can do. The Libra project, on the other hand, designed an on-chain programming language called Move that is more suitable for parallel execution. Like Solana's runtime, Move programs depend on accounts for all shared state.
+
+The biggest design difference between Solana's runtime and Libra's Move VM is how they manage safe invocations between modules. Solana took an operating systems approach and Libra took the domain-specific language approach. In the runtime, a module must trap back into the runtime to ensure the caller's module did not write to data owned by the callee. Likewise, when the callee completes, it must again trap back to the runtime to ensure the callee did not write to data owned by the caller. Move, on the other hand, includes an advanced type system that allows these checks to be run by its bytecode verifier. Because Move bytecode can be verified, the cost of verification is paid just once, at the time the module is loaded on-chain. In the runtime, the cost is paid each time a transaction crosses between modules. The difference is similar in spirit to the difference between a dynamically-typed language like Python versus a statically-typed language like Java. Solana's runtime allows applications to be written in general purpose programming languages, but that comes with the cost of runtime checks when jumping between programs.
+
+This proposal attempts to define a way to embed the Move VM such that:
+
+- cross-module invocations within Move do not require the runtime's
+
+  cross-program runtime checks
+
+- Move programs can leverage functionality in other Solana programs and vice
+
+  versa
+
+- Solana's runtime parallelism is exposed to batches of Move and non-Move
+
+  transactions
+
+## Proposed Solution
+
+### Move VM as a Solana loader
+
+The Move VM shall be embedded as a Solana loader under the identifier `MOVE_PROGRAM_ID`, so that Move modules can be marked as `executable` with the VM as its `owner`. This will allow modules to load module dependencies, as well as allow for parallel execution of Move scripts.
+
+All data accounts owned by Move modules must set their owners to the loader, `MOVE_PROGRAM_ID`. Since Move modules encapsulate their account data in the same way Solana programs encapsulate theirs, the Move module owner should be embedded in the account data. The runtime will grant write access to the Move VM, and Move grants access to the module accounts.
+
+### Interacting with Solana programs
+
+To invoke instructions in non-Move programs, Solana would need to extend the Move VM with a `process_instruction()` system call. It would work the same as `process_instruction()` Rust BPF programs.