solana-docs 0.0.0

package/src/proposals/interchain-transaction-verification.md

@@ -0,0 +1,105 @@
+---
+title: Inter-chain Transaction Verification
+---
+
+## Problem
+
+Inter-chain applications are not new to the digital asset ecosystem; in fact, even the smaller centralized exchanges still categorically dwarf all single chain applications put together in terms of users and volume. They command massive valuations and have spent years effectively optimizing their core products for a broad range of end users. However, their basic operations center around mechanisms that require their users to unilaterally trust them, typically with little to no recourse or protection from accidental loss. This has led to the broader digital asset ecosystem being fractured along network lines because interoperability solutions typically:
+
+- Are technically complex to fully implement
+- Create unstable network scale incentive structures
+- Require consistent and high level cooperation between stakeholders
+
+## Proposed Solution
+
+Simple Payment Verification \(SPV\) is a generic term for a range of different methodologies used by light clients on most major blockchain networks to verify aspects of the network state without the burden of fully storing and maintaining the chain itself. In most cases, this means relying on a form of hash tree to supply a proof of the presence of a given transaction in a certain block by comparing against a root hash in that block’s header or equivalent. This allows a light client or wallet to reach a probabilistic level of certainty about on-chain events by itself with a minimum of trust required with regard to network nodes.
+
+Traditionally the process of assembling and validating these proofs is carried out off chain by nodes, wallets, or other clients, but it also offers a potential mechanism for inter-chain state verification. However, by moving the capability to validate SPV proofs on-chain as a smart contract while leveraging the archival properties inherent to the blockchain, it is possible to construct a system for programmatically detecting and verifying transactions on other networks without the involvement of any type of trusted oracle or complex multi-stage consensus mechanism. This concept is broadly generalisable to any network with an SPV mechanism and can even be operated bilaterally on other smart contract platforms, opening up the possibility of cheap, fast, inter-chain transfer of value without relying on collateral, hashlocks, or trusted intermediaries.
+
+Opting to take advantage of well established and developmentally stable mechanisms already common to all major blockchains allows SPV based interoperability solutions to be dramatically simpler than orchestrated multi-stage approaches. As part of this, they dispense with the need for widely agreed upon cross chain communication standards and the large multi-party organizations that write them in favor of a set of discrete contract-based services that can be easily utilized by caller contracts through a common abstraction format. This will set the groundwork for a broad range of applications and contracts able to interoperate across the variegated and every growing platform ecosystem.
+
+## Terminology
+
+SPV Program - Client-facing interface for the inter-chain SPV system, manages participant roles. SPV Engine - Validates transaction proofs, subset of the SPV Program. Client - The caller to the SPV Program, typically another solana contract. Prover - Party who generates proofs for transactions and submits them to the SPV Program. Transaction Proof - Created by Provers, contains a merkle proof, transaction, and blockheader reference. Merkle Proof - Basic SPV proof that validates the presence of a transaction in a certain block. Block Header - Represents the basic parameters and relative position of a given block. Proof Request - An order placed by a client for verification of transaction\(s\) by provers. Header Store - A data structure for storing and referencing ranges of block headers in proofs. Client Request - Transaction from the client to the SPV Program to trigger creation of a Proof Request. Sub-account - A Solana account owned by another contract account, without its own private key.
+
+## Service
+
+SPV Programs run as contracts deployed on the Solana network and maintain a type of public marketplace for SPV proofs that allows any party to submit both requests for proofs as well as proofs themselves for verification in response to requests. There will be multiple SPV Program instances active at any given time, at least one for each connected external network and potentially multiple instances per network. SPV program instances will be relatively consistent in their high level API and feature sets with some variation between currency platforms \(Bitcoin, Litecoin\) and smart contract platforms owing to the potential for verification of network state changes beyond simply transactions. In every case regardless of network, the SPV Program relies on an internal component called an SPV engine to provide stateless verification of the actual SPV proofs upon which the higher level client facing features and api are built. The SPV engine requires a network specific implementation, but allows easy extension of the larger inter-chain ecosystem by any team who chooses to carry out that implementation and drop it into the standard SPV program for deployment.
+
+For purposes of Proof Requests, the requester is referred to as the program client, which in most if not all cases will be another Solana Contract. The client can choose to submit a request pertaining to a specific transaction or to include a broader filter that can apply to any of a range of parameters of a transaction including its inputs, outputs, and amount. For example, A client could submit a request for any transaction sent from a given address A to address B with the amount X after a certain time. This structure can be used in a range of applications, such as verifying a specific intended payment in the case of an atomic swap or detecting the movement of collateral assets for a loan.
+
+Following submission of a Client Request, assuming that it is successfully validated, a proof request account is created by the SPV program to track the progress of the request. Provers use the account to specify the request they intend to fill in the proofs they submit for validation, at which point the SPV program validates those proofs and if successful, saves them to the account data of the request account. Clients can monitor the status of their requests and see any applicable transactions alongside their proofs by querying the account data of the request account. In future iterations when supported by Solana, this process will be simplified by contracts publishing events rather than requiring a polling style process as described.
+
+## Implementation
+
+The Solana Inter-chain SPV mechanism consists of the following components and participants:
+
+### SPV engine
+
+A contract deployed on Solana which statelessly verifies SPV proofs for the caller. It takes as arguments for validation:
+
+- An SPV proof in the correct format of the blockchain associated with the program
+- Reference\(s\) to the relevant block headers to compare that proof against
+- The necessary parameters of the transaction to verify
+
+  If the proof in question is successfully validated, the SPV program saves proof
+
+  of that verification to the request account, which can be saved by the caller to
+
+  its account data or otherwise handled as necessary. SPV programs also expose
+
+  utilities and structs used for representation and validation of headers,
+
+  transactions, hashes, etc. on a chain by chain basis.
+
+### SPV program
+
+A contract deployed on Solana which coordinates and intermediates the interaction between Clients and Provers and manages the validation of requests, headers, proofs, etc. It is the primary point of access for Client contracts to access the inter-chain. SPV mechanism. It offers the following core features:
+
+- Submit Proof Request - allows client to place a request for a specific proof or set of proofs
+- Cancel Proof Request - allows client to invalidate a pending request
+- Fill Proof Request - used by Provers to submit for validation a proof corresponding to a given Proof Request
+
+  The SPV program maintains a publicly available listing of valid pending Proof
+
+  Requests in its account data for the benefit of the Provers, who monitor it and
+
+  enclose references to target requests with their submitted proofs.
+
+### Proof Request
+
+A message sent by the Client to the SPV engine denoting a request for a proof of a specific transaction or set of transactions. Proof Requests can either manually specify a certain transaction by its hash or can elect to submit a filter that matches multiple transactions or classes of transactions. For example, a filter matching “any transaction from address xxx to address yyy” could be used to detect payment of a debt or settlement of an inter-chain swap. Likewise, a filter matching “any transaction from address xxx” could be used by a lending or synthetic token minting contract to monitor and react to changes in collateralization. Proof Requests are sent with a fee, which is disbursed by the SPV engine contract to the appropriate Prover once a proof matching that request is validated.
+
+### Request Book
+
+The public listing of valid, open Proof Requests available to provers to fill or for clients to cancel. Roughly analogous to an orderbook in an exchange, but with a single type of listing rather than two separate sides. It is stored in the account data of the SPV program.
+
+### Proof
+
+A proof of the presence of a given transaction in the blockchain in question. Proofs encompass both the actual merkle proof and reference\(s\) to a chain of valid sequential block headers. They are constructed and submitted by Provers in accordance with the specifications of the publicly available Proof Requests hosted on the request book by the SPV program. Upon Validation, they are saved to the account data of the relevant Proof Request, which can be used by the Client to monitor the state of the request.
+
+### Client
+
+The originator of a request for a transaction proof. Clients will most often be other contracts as parts of applications or specific financial products like loans, swaps, escrow, etc. The client in any given verification process cycle initially submits a ClientRequest which communicates the parameters and fee and if successfully validated, results in the creation of a Proof Request account by the SPV program. The Client may also submit a CancelRequest referencing an active Proof Request in order to denote it as invalid for purposes of proof submission.
+
+### Prover
+
+The submitter of a proof that fills a Proof Request. Provers monitor the request book of the SPV program for outstanding Proof Requests and generate matching proofs, which they submit to the SPV program for validation. If the proof is accepted, the fee associated with the Proof Request in question is disbursed to the Prover. Provers typically operate as Solana Blockstreamer nodes that also have access to a Bitcoin node, which they use for purposes of constructing proofs and accessing block headers.
+
+### Header Store
+
+An account-based data structure used to maintain block headers for the purpose of inclusion in submitted proofs by reference to the header store account. header stores can be maintained by independent entities, since header chain validation is a component of the SPV program proof validation mechanism. Fees that are paid out by Proof Requests to Provers are split between the submitter of the merkle proof itself and the header store that is referenced in the submitted proof. Due to the current inability to grow already allocated account data capacity, the use case necessitates a data structure that can grow indefinitely without rebalancing. Sub-accounts are accounts owned by the SPV program without their own private keys that are used for storage by allocating blockheaders to their account data. Multiple potential approaches to the implementation of the header store system are feasible:
+
+Store Headers in program sub-accounts indexed by Public address:
+
+- Each sub-account holds one header and has a public key matching the blockhash
+- Requires same number of account data lookups as confirmations per verification
+- Limit on number of confirmations \(15-20\) via max transaction data ceiling
+- No network-wide duplication of individual headers
+
+Linked List of multiple sub-accounts storing headers:
+
+- Maintain sequential index of storage accounts, many headers per storage account
+- Max 2 account data lookups for >99.9% of verifications \(1 for most\)
+- Compact sequential data address format allows any number of confirmations and fast lookups
+- Facilitates network-wide header duplication inefficiencies

package/src/proposals/ledger-replication-to-implement.md

@@ -0,0 +1,417 @@
+---
+title: Ledger Replication
+---
+
+Note: this ledger replication solution was partially implemented, but not
+completed. The partial implementation was removed by
+https://github.com/solana-labs/solana/pull/9992 in order to prevent the security
+risk of unused code. The first part of this design document reflects the
+once-implemented parts of ledger replication. The
+[second part of this document](#ledger-replication-not-implemented) describes the
+parts of the solution never implemented.
+
+## Proof of Replication
+
+At full capacity on a 1gbps network solana will generate 4 petabytes of data per year. To prevent the network from centralizing around validators that have to store the full data set this protocol proposes a way for mining nodes to provide storage capacity for pieces of the data.
+
+The basic idea to Proof of Replication is encrypting a dataset with a public symmetric key using CBC encryption, then hash the encrypted dataset. The main problem with the naive approach is that a dishonest storage node can stream the encryption and delete the data as it's hashed. The simple solution is to periodically regenerate the hash based on a signed PoH value. This ensures that all the data is present during the generation of the proof and it also requires validators to have the entirety of the encrypted data present for verification of every proof of every identity. So the space required to validate is `number_of_proofs * data_size`
+
+## Optimization with PoH
+
+Our improvement on this approach is to randomly sample the encrypted segments faster than it takes to encrypt, and record the hash of those samples into the PoH ledger. Thus the segments stay in the exact same order for every PoRep and verification can stream the data and verify all the proofs in a single batch. This way we can verify multiple proofs concurrently, each one on its own CUDA core. The total space required for verification is `1_ledger_segment + 2_cbc_blocks * number_of_identities` with core count equal to `number_of_identities`. We use a 64-byte chacha CBC block size.
+
+## Network
+
+Validators for PoRep are the same validators that are verifying transactions. If an archiver can prove that a validator verified a fake PoRep, then the validator will not receive a reward for that storage epoch.
+
+Archivers are specialized _light clients_. They download a part of the ledger \(a.k.a Segment\) and store it, and provide PoReps of storing the ledger. For each verified PoRep archivers earn a reward of sol from the mining pool.
+
+## Constraints
+
+We have the following constraints:
+
+- Verification requires generating the CBC blocks. That requires space of 2
+
+  blocks per identity, and 1 CUDA core per identity for the same dataset. So as
+
+  many identities at once should be batched with as many proofs for those
+
+  identities verified concurrently for the same dataset.
+
+- Validators will randomly sample the set of storage proofs to the set that
+
+  they can handle, and only the creators of those chosen proofs will be
+
+  rewarded. The validator can run a benchmark whenever its hardware configuration
+
+  changes to determine what rate it can validate storage proofs.
+
+## Validation and Replication Protocol
+
+### Constants
+
+1. SLOTS_PER_SEGMENT: Number of slots in a segment of ledger data. The
+
+   unit of storage for an archiver.
+
+2. NUM_KEY_ROTATION_SEGMENTS: Number of segments after which archivers
+
+   regenerate their encryption keys and select a new dataset to store.
+
+3. NUM_STORAGE_PROOFS: Number of storage proofs required for a storage proof
+
+   claim to be successfully rewarded.
+
+4. RATIO_OF_FAKE_PROOFS: Ratio of fake proofs to real proofs that a storage
+
+   mining proof claim has to contain to be valid for a reward.
+
+5. NUM_STORAGE_SAMPLES: Number of samples required for a storage mining
+
+   proof.
+
+6. NUM_CHACHA_ROUNDS: Number of encryption rounds performed to generate
+
+   encrypted state.
+
+7. NUM_SLOTS_PER_TURN: Number of slots that define a single storage epoch or
+
+   a "turn" of the PoRep game.
+
+### Validator behavior
+
+1. Validators join the network and begin looking for archiver accounts at each
+
+   storage epoch/turn boundary.
+
+2. Every turn, Validators sign the PoH value at the boundary and use that signature
+
+   to randomly pick proofs to verify from each storage account found in the turn boundary.
+
+   This signed value is also submitted to the validator's storage account and will be used by
+
+   archivers at a later stage to cross-verify.
+
+3. Every `NUM_SLOTS_PER_TURN` slots the validator advertises the PoH value. This is value
+
+   is also served to Archivers via RPC interfaces.
+
+4. For a given turn N, all validations get locked out until turn N+3 \(a gap of 2 turn/epoch\).
+
+   At which point all validations during that turn are available for reward collection.
+
+5. Any incorrect validations will be marked during the turn in between.
+
+### Archiver behavior
+
+1. Since an archiver is somewhat of a light client and not downloading all the
+
+   ledger data, they have to rely on other validators and archivers for information.
+
+   Any given validator may or may not be malicious and give incorrect information, although
+
+   there are not any obvious attack vectors that this could accomplish besides having the
+
+   archiver do extra wasted work. For many of the operations there are a number of options
+
+   depending on how paranoid an archiver is:
+
+   - \(a\) archiver can ask a validator
+   - \(b\) archiver can ask multiple validators
+   - \(c\) archiver can ask other archivers
+   - \(d\) archiver can subscribe to the full transaction stream and generate
+
+     the information itself \(assuming the slot is recent enough\)
+
+   - \(e\) archiver can subscribe to an abbreviated transaction stream to
+
+     generate the information itself \(assuming the slot is recent enough\)
+
+2. An archiver obtains the PoH hash corresponding to the last turn with its slot.
+3. The archiver signs the PoH hash with its keypair. That signature is the
+
+   seed used to pick the segment to replicate and also the encryption key. The
+
+   archiver mods the signature with the slot to get which segment to
+
+   replicate.
+
+4. The archiver retrives the ledger by asking peer validators and
+
+   archivers. See 6.5.
+
+5. The archiver then encrypts that segment with the key with chacha algorithm
+
+   in CBC mode with `NUM_CHACHA_ROUNDS` of encryption.
+
+6. The archiver initializes a chacha rng with the a signed recent PoH value as
+
+   the seed.
+
+7. The archiver generates `NUM_STORAGE_SAMPLES` samples in the range of the
+
+   entry size and samples the encrypted segment with sha256 for 32-bytes at each
+
+   offset value. Sampling the state should be faster than generating the encrypted
+
+   segment.
+
+8. The archiver sends a PoRep proof transaction which contains its sha state
+
+   at the end of the sampling operation, its seed and the samples it used to the
+
+   current leader and it is put onto the ledger.
+
+9. During a given turn the archiver should submit many proofs for the same segment
+
+   and based on the `RATIO_OF_FAKE_PROOFS` some of those proofs must be fake.
+
+10. As the PoRep game enters the next turn, the archiver must submit a
+
+    transaction with the mask of which proofs were fake during the last turn. This
+
+    transaction will define the rewards for both archivers and validators.
+
+11. Finally for a turn N, as the PoRep game enters turn N + 3, archiver's proofs for
+
+    turn N will be counted towards their rewards.
+
+### The PoRep Game
+
+The Proof of Replication game has 4 primary stages. For each "turn" multiple PoRep games can be in progress but each in a different stage.
+
+The 4 stages of the PoRep Game are as follows:
+
+1. Proof submission stage
+   - Archivers: submit as many proofs as possible during this stage
+   - Validators: No-op
+2. Proof verification stage
+   - Archivers: No-op
+   - Validators: Select archivers and verify their proofs from the previous turn
+3. Proof challenge stage
+   - Archivers: Submit the proof mask with justifications \(for fake proofs submitted 2 turns ago\)
+   - Validators: No-op
+4. Reward collection stage
+   - Archivers: Collect rewards for 3 turns ago
+   - Validators: Collect rewards for 3 turns ago
+
+For each turn of the PoRep game, both Validators and Archivers evaluate each stage. The stages are run as separate transactions on the storage program.
+
+### Finding who has a given block of ledger
+
+1. Validators monitor the turns in the PoRep game and look at the rooted bank
+
+   at turn boundaries for any proofs.
+
+2. Validators maintain a map of ledger segments and corresponding archiver public keys.
+
+   The map is updated when a Validator processes an archiver's proofs for a segment.
+
+   The validator provides an RPC interface to access the this map. Using this API, clients
+
+   can map a segment to an archiver's network address \(correlating it via cluster_info table\).
+
+   The clients can then send repair requests to the archiver to retrieve segments.
+
+3. Validators would need to invalidate this list every N turns.
+
+## Sybil attacks
+
+For any random seed, we force everyone to use a signature that is derived from a PoH hash at the turn boundary. Everyone uses the same count, so the same PoH hash is signed by every participant. The signatures are then each cryptographically tied to the keypair, which prevents a leader from grinding on the resulting value for more than 1 identity.
+
+Since there are many more client identities then encryption identities, we need to split the reward for multiple clients, and prevent Sybil attacks from generating many clients to acquire the same block of data. To remain BFT we want to avoid a single human entity from storing all the replications of a single chunk of the ledger.
+
+Our solution to this is to force the clients to continue using the same identity. If the first round is used to acquire the same block for many client identities, the second round for the same client identities will force a redistribution of the signatures, and therefore PoRep identities and blocks. Thus to get a reward for archivers need to store the first block for free and the network can reward long lived client identities more than new ones.
+
+## Validator attacks
+
+- If a validator approves fake proofs, archiver can easily out them by
+
+  showing the initial state for the hash.
+
+- If a validator marks real proofs as fake, no on-chain computation can be done
+
+  to distinguish who is correct. Rewards would have to rely on the results from
+
+  multiple validators to catch bad actors and archivers from being denied rewards.
+
+- Validator stealing mining proof results for itself. The proofs are derived
+
+  from a signature from an archiver, since the validator does not know the
+
+  private key used to generate the encryption key, it cannot be the generator of
+
+  the proof.
+
+## Reward incentives
+
+Fake proofs are easy to generate but difficult to verify. For this reason, PoRep proof transactions generated by archivers may require a higher fee than a normal transaction to represent the computational cost required by validators.
+
+Some percentage of fake proofs are also necessary to receive a reward from storage mining.
+
+## Notes
+
+- We can reduce the costs of verification of PoRep by using PoH, and actually
+
+  make it feasible to verify a large number of proofs for a global dataset.
+
+- We can eliminate grinding by forcing everyone to sign the same PoH hash and
+
+  use the signatures as the seed
+
+- The game between validators and archivers is over random blocks and random
+
+  encryption identities and random data samples. The goal of randomization is
+
+  to prevent colluding groups from having overlap on data or validation.
+
+- Archiver clients fish for lazy validators by submitting fake proofs that
+
+  they can prove are fake.
+
+- To defend against Sybil client identities that try to store the same block we
+
+  force the clients to store for multiple rounds before receiving a reward.
+
+- Validators should also get rewarded for validating submitted storage proofs
+
+  as incentive for storing the ledger. They can only validate proofs if they
+
+  are storing that slice of the ledger.
+
+# Ledger Replication Not Implemented
+
+Replication behavior yet to be implemented.
+
+## Storage epoch
+
+The storage epoch should be the number of slots which results in around 100GB-1TB of ledger to be generated for archivers to store. Archivers will start storing ledger when a given fork has a high probability of not being rolled back.
+
+## Validator behavior
+
+1. Every NUM_KEY_ROTATION_TICKS it also validates samples received from
+
+   archivers. It signs the PoH hash at that point and uses the following
+
+   algorithm with the signature as the input:
+
+   - The low 5 bits of the first byte of the signature creates an index into
+
+     another starting byte of the signature.
+
+   - The validator then looks at the set of storage proofs where the byte of
+
+     the proof's sha state vector starting from the low byte matches exactly
+
+     with the chosen byte\(s\) of the signature.
+
+   - If the set of proofs is larger than the validator can handle, then it
+
+     increases to matching 2 bytes in the signature.
+
+   - Validator continues to increase the number of matching bytes until a
+
+     workable set is found.
+
+   - It then creates a mask of valid proofs and fake proofs and sends it to
+
+     the leader. This is a storage proof confirmation transaction.
+
+2. After a lockout period of NUM_SECONDS_STORAGE_LOCKOUT seconds, the
+
+   validator then submits a storage proof claim transaction which then causes the
+
+   distribution of the storage reward if no challenges were seen for the proof to
+
+   the validators and archivers party to the proofs.
+
+## Archiver behavior
+
+1. The archiver then generates another set of offsets which it submits a fake
+
+   proof with an incorrect sha state. It can be proven to be fake by providing the
+
+   seed for the hash result.
+
+   - A fake proof should consist of an archiver hash of a signature of a PoH
+
+     value. That way when the archiver reveals the fake proof, it can be
+
+     verified on chain.
+
+2. The archiver monitors the ledger, if it sees a fake proof integrated, it
+
+   creates a challenge transaction and submits it to the current leader. The
+
+   transacation proves the validator incorrectly validated a fake storage proof.
+
+   The archiver is rewarded and the validator's staking balance is slashed or
+
+   frozen.
+
+## Storage proof contract logic
+
+Each archiver and validator will have their own storage account. The validator's account would be separate from their gossip id similiar to their vote account. These should be implemented as two programs one which handles the validator as the keysigner and one for the archiver. In that way when the programs reference other accounts, they can check the program id to ensure it is a validator or archiver account they are referencing.
+
+### SubmitMiningProof
+
+```text
+SubmitMiningProof {
+    slot: u64,
+    sha_state: Hash,
+    signature: Signature,
+};
+keys = [archiver_keypair]
+```
+
+Archivers create these after mining their stored ledger data for a certain hash value. The slot is the end slot of the segment of ledger they are storing, the sha_state the result of the archiver using the hash function to sample their encrypted ledger segment. The signature is the signature that was created when they signed a PoH value for the current storage epoch. The list of proofs from the current storage epoch should be saved in the account state, and then transfered to a list of proofs for the previous epoch when the epoch passes. In a given storage epoch a given archiver should only submit proofs for one segment.
+
+The program should have a list of slots which are valid storage mining slots. This list should be maintained by keeping track of slots which are rooted slots in which a significant portion of the network has voted on with a high lockout value, maybe 32-votes old. Every SLOTS_PER_SEGMENT number of slots would be added to this set. The program should check that the slot is in this set. The set can be maintained by receiving a AdvertiseStorageRecentBlockHash and checking with its bank/Tower BFT state.
+
+The program should do a signature verify check on the signature, public key from the transaction submitter and the message of the previous storage epoch PoH value.
+
+### ProofValidation
+
+```text
+ProofValidation {
+   proof_mask: Vec<ProofStatus>,
+}
+keys = [validator_keypair, archiver_keypair(s) (unsigned)]
+```
+
+A validator will submit this transaction to indicate that a set of proofs for a given segment are valid/not-valid or skipped where the validator did not look at it. The keypairs for the archivers that it looked at should be referenced in the keys so the program logic can go to those accounts and see that the proofs are generated in the previous epoch. The sampling of the storage proofs should be verified ensuring that the correct proofs are skipped by the validator according to the logic outlined in the validator behavior of sampling.
+
+The included archiver keys will indicate the the storage samples which are being referenced; the length of the proof_mask should be verified against the set of storage proofs in the referenced archiver account\(s\), and should match with the number of proofs submitted in the previous storage epoch in the state of said archiver account.
+
+### ClaimStorageReward
+
+```text
+ClaimStorageReward {
+}
+keys = [validator_keypair or archiver_keypair, validator/archiver_keypairs (unsigned)]
+```
+
+Archivers and validators will use this transaction to get paid tokens from a program state where SubmitStorageProof, ProofValidation and ChallengeProofValidations are in a state where proofs have been submitted and validated and there are no ChallengeProofValidations referencing those proofs. For a validator, it should reference the archiver keypairs to which it has validated proofs in the relevant epoch. And for an archiver it should reference validator keypairs for which it has validated and wants to be rewarded.
+
+### ChallengeProofValidation
+
+```text
+ChallengeProofValidation {
+    proof_index: u64,
+    hash_seed_value: Vec<u8>,
+}
+keys = [archiver_keypair, validator_keypair]
+```
+
+This transaction is for catching lazy validators who are not doing the work to validate proofs. An archiver will submit this transaction when it sees a validator has approved a fake SubmitMiningProof transaction. Since the archiver is a light client not looking at the full chain, it will have to ask a validator or some set of validators for this information maybe via RPC call to obtain all ProofValidations for a certain segment in the previous storage epoch. The program will look in the validator account state see that a ProofValidation is submitted in the previous storage epoch and hash the hash_seed_value and see that the hash matches the SubmitMiningProof transaction and that the validator marked it as valid. If so, then it will save the challenge to the list of challenges that it has in its state.
+
+### AdvertiseStorageRecentBlockhash
+
+```text
+AdvertiseStorageRecentBlockhash {
+    hash: Hash,
+    slot: u64,
+}
+```
+
+Validators and archivers will submit this to indicate that a new storage epoch has passed and that the storage proofs which are current proofs should now be for the previous epoch. Other transactions should check to see that the epoch that they are referencing is accurate according to current chain state.

package/src/proposals/log_data.md

@@ -0,0 +1,146 @@
+# Program log binary data
+
+## Problem
+
+There is no support for logging binary data in Solidity.
+
+### Events in Solidity
+
+In Solidity, events can be reported. These look like structures with zero or
+more fields, and can be emitted with specific values. For example:
+
+```
+event PaymentReceived {
+    address sender;
+    uint amount;
+}
+
+contract c {
+    function pay() public payable {
+        emit PaymentReceived(msg.sender, msg.value);
+    }
+}
+```
+
+Events are write-only from a Solidity/VM perspective and are written to
+the blocks in the tx records.
+
+Some of these fields can be marked `indexed`, which affects how the data is
+encoded. All non-indexed fields are eth abi encoded into a variable length
+byte array. All indexed fields go into so-called topics.
+
+Topics are fixed length fields of 32 bytes. There are a maximum of 4 topics;
+if a type does not always fit into 32 bytes (e.g. string types), then the topic
+is keccak256 hashed.
+
+The first topic is a keccak256 hash of the event signature, in this case
+`keccak256('PaymentReceived(address,uint)')`. The four remaining are available
+for `indexed` fields. The event may be declared `anonymous`, in which case
+the first field is not a hash of the signature, and it is permitted to have
+4 indexed fields.
+
+### Listening to events in a client
+
+The reason for the distinction between topics/indexed and regular fields is
+that it easier to filter on topics.
+
+```
+const Web3 = require('web3');
+const url = 'ws://127.0.0.1:8546';
+const web3 = new Web3(url);
+
+var options = {
+    address: '0xfbBE8f06FAda977Ea1E177da391C370EFbEE3D25',
+    topics: [
+        '0xdf50c7bb3b25f812aedef81bc334454040e7b27e27de95a79451d663013b7e17',
+        //'0x0000000000000000000000000d8a3f5e71560982fb0eb5959ecf84412be6ae3e'
+      ]
+};
+
+var subscription = web3.eth.subscribe('logs', options, function(error, result){
+    if (!error) console.log('got result');
+    else console.log(error);
+}).on("data", function(log){
+    console.log('got data', log);
+}).on("changed", function(log){
+    console.log('changed');
+});
+```
+
+In order to decode the non-indexed fields (the data), the abi of the contract
+is needed. So, the topic is first used to discover what event was used, and
+then the data can be decoded.
+
+### Ethereum Tx in block
+
+The transaction calls event logs. Here is a tx with a single event, with 3
+topics and some data.
+
+```
+{
+  "tx": {
+    "nonce": "0x2",
+    "gasPrice": "0xf224d4a00",
+    "gas": "0xc350",
+    "to": "0x6B175474E89094C44Da98b954EedeAC495271d0F",
+    "value": "0x0",
+    "input": "0xa9059cbb000000000000000000000000a12431d0b9db640034b0cdfceef9cce161e62be40000000000000000000000000000000000000000000000a030dcebbd2f4c0000",
+    "hash": "0x98a67f0a35ebc0ac068acf0885d38419c632ffa4354e96641d6d5103a7681910",
+    "blockNumber": "0xc96431",
+    "from": "0x82f890D638478d211eF2208f3c1466B5Abf83551",
+    "transactionIndex": "0xe1"
+  },
+  "receipt": {
+    "gasUsed": "0x74d2",
+    "status": "0x1",
+    "logs": [
+      {
+        "address": "0x6B175474E89094C44Da98b954EedeAC495271d0F",
+        "topics": [
+          "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
+          "0x00000000000000000000000082f890d638478d211ef2208f3c1466b5abf83551",
+          "0x000000000000000000000000a12431d0b9db640034b0cdfceef9cce161e62be4"
+        ],
+        "data": "0x0000000000000000000000000000000000000000000000a030dcebbd2f4c0000"
+      }
+    ]
+  }
+}
+```
+
+### Further considerations
+
+In Ethereum, events are stored in blocks. Events mark certain state changes in
+smart contracts. This serves two purposes:
+
+ - Listen to events (i.e. state changes) as they happen by reading new blocks
+   as they are published
+ - Re-read historical events by reading old blocks
+
+So for example, smart contracts may emit changes as they happen but never the
+complete state, so the only way to recover the entire state of the contract
+is by re-reading all events from the chain. So an application will read events
+from block 1 or whatever block the application was deployed at and then use
+that state for local processing. This is a local cache and may re-populated
+from the chain at any point.
+
+## Proposed Solution
+
+Binary logging should be added to the program log. The program log should include the base64 encoded data (zero or more one permitted).
+
+So if we encoding the topics first, followed by the data then the event in the
+tx above would look like:
+```
+program data: 3fJSrRviyJtpwrBo/DeNqpUrpFjxKEWKPVaTfUjs8AAAAAAAAAAAAAAACC+JDWOEeNIR7yII88FGa1q/g1UQAAAAAAAAAAAAAAAKEkMdC522QANLDN/O75zOFh5ivk AAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgMNzrvS9MAAA=
+```
+
+This requires a new system call:
+
+```
+void sol_log_data(SolBytes *fields, uint64_t length);
+```
+
+### Considerations
+
+- Should there be text field in the program log so we can have a little bit of
+  metadata on the binary data, to make it more human readable