solana-docs 0.0.0

package/src/developing/on-chain-programs/overview.md

@@ -0,0 +1,218 @@
+---
+title: "Overview"
+---
+
+Developers can write and deploy their own programs to the Solana blockchain.
+
+The [Helloworld example](examples.md#helloworld) is a good starting place to see
+how a program is written, built, deployed, and interacted with on-chain.
+
+## Berkeley Packet Filter (BPF)
+
+Solana on-chain programs are compiled via the [LLVM compiler
+infrastructure](https://llvm.org/) to an [Executable and Linkable Format
+(ELF)](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format) containing
+a variation of the [Berkeley Packet Filter
+(BPF)](https://en.wikipedia.org/wiki/Berkeley_Packet_Filter) bytecode.
+
+Because Solana uses the LLVM compiler infrastructure, a program may be written
+in any programming language that can target the LLVM's BPF backend. Solana
+currently supports writing programs in Rust and C/C++.
+
+BPF provides an efficient [instruction
+set](https://github.com/iovisor/bpf-docs/blob/master/eBPF.md) that can be
+executed in an interpreted virtual machine or as efficient just-in-time compiled
+native instructions.
+
+## Memory map
+
+The virtual address memory map used by Solana BPF programs is fixed and laid out
+as follows
+
+- Program code starts at 0x100000000
+- Stack data starts at 0x200000000
+- Heap data starts at 0x300000000
+- Program input parameters start at 0x400000000
+
+The above virtual addresses are start addresses but programs are given access to
+a subset of the memory map. The program will panic if it attempts to read or
+write to a virtual address that it was not granted access to, and an
+`AccessViolation` error will be returned that contains the address and size of
+the attempted violation.
+
+## Stack
+
+BPF uses stack frames instead of a variable stack pointer. Each stack frame is
+4KB in size.
+
+If a program violates that stack frame size, the compiler will report the
+overrun as a warning.
+
+For example: `Error: Function _ZN16curve25519_dalek7edwards21EdwardsBasepointTable6create17h178b3d2411f7f082E Stack offset of -30728 exceeded max offset of -4096 by 26632 bytes, please minimize large stack variables`
+
+The message identifies which symbol is exceeding its stack frame but the name
+might be mangled if it is a Rust or C++ symbol. To demangle a Rust symbol use
+[rustfilt](https://github.com/luser/rustfilt). The above warning came from a
+Rust program, so the demangled symbol name is:
+
+```bash
+$ rustfilt _ZN16curve25519_dalek7edwards21EdwardsBasepointTable6create17h178b3d2411f7f082E
+curve25519_dalek::edwards::EdwardsBasepointTable::create
+```
+
+To demangle a C++ symbol use `c++filt` from binutils.
+
+The reason a warning is reported rather than an error is because some dependent
+crates may include functionality that violates the stack frame restrictions even
+if the program doesn't use that functionality. If the program violates the stack
+size at runtime, an `AccessViolation` error will be reported.
+
+BPF stack frames occupy a virtual address range starting at 0x200000000.
+
+## Call Depth
+
+Programs are constrained to run quickly, and to facilitate this, the program's
+call stack is limited to a max depth of 64 frames.
+
+## Heap
+
+Programs have access to a runtime heap either directly in C or via the Rust
+`alloc` APIs. To facilitate fast allocations, a simple 32KB bump heap is
+utilized. The heap does not support `free` or `realloc` so use it wisely.
+
+Internally, programs have access to the 32KB memory region starting at virtual
+address 0x300000000 and may implement a custom heap based on the the program's
+specific needs.
+
+- [Rust program heap usage](developing-rust.md#heap)
+- [C program heap usage](developing-c.md#heap)
+
+## Float Support
+
+Programs support a limited subset of Rust's float operations, if a program
+attempts to use a float operation that is not supported, the runtime will report
+an unresolved symbol error.
+
+Float operations are performed via software libraries, specifically LLVM's float
+builtins. Due to the software emulated they consume more compute units than
+integer operations. In general, fixed point operations are recommended where
+possible.
+
+The Solana Program Library math tests will report the performance of some math
+operations:
+https://github.com/solana-labs/solana-program-library/tree/master/libraries/math
+
+To run the test, sync the repo, and run:
+
+`$ cargo test-bpf -- --nocapture --test-threads=1`
+
+Recent results show the float operations take more instructions compared to
+integers equivalents. Fixed point implementations may vary but will also be
+less than the float equivalents:
+
+```
+         u64   f32
+Multipy    8   176
+Divide     9   219
+```
+
+## Static Writable Data
+
+Program shared objects do not support writable shared data. Programs are shared
+between multiple parallel executions using the same shared read-only code and
+data. This means that developers should not include any static writable or
+global variables in programs. In the future a copy-on-write mechanism could be
+added to support writable data.
+
+## Signed division
+
+The BPF instruction set does not support [signed
+division](https://www.kernel.org/doc/html/latest/bpf/bpf_design_QA.html#q-why-there-is-no-bpf-sdiv-for-signed-divide-operation).
+Adding a signed division instruction is a consideration.
+
+## Loaders
+
+Programs are deployed with and executed by runtime loaders, currently there are
+two supported loaders [BPF
+Loader](https://github.com/solana-labs/solana/blob/7ddf10e602d2ed87a9e3737aa8c32f1db9f909d8/sdk/program/src/bpf_loader.rs#L17)
+and [BPF loader
+deprecated](https://github.com/solana-labs/solana/blob/7ddf10e602d2ed87a9e3737aa8c32f1db9f909d8/sdk/program/src/bpf_loader_deprecated.rs#L14)
+
+Loaders may support different application binary interfaces so developers must
+write their programs for and deploy them to the same loader. If a program
+written for one loader is deployed to a different one the result is usually a
+`AccessViolation` error due to mismatched deserialization of the program's input
+parameters.
+
+For all practical purposes program should always be written to target the latest
+BPF loader and the latest loader is the default for the command-line interface
+and the javascript APIs.
+
+For language specific information about implementing a program for a particular
+loader see:
+
+- [Rust program entrypoints](developing-rust.md#program-entrypoint)
+- [C program entrypoints](developing-c.md#program-entrypoint)
+
+### Deployment
+
+BPF program deployment is the process of uploading a BPF shared object into a
+program account's data and marking the account executable. A client breaks the
+BPF shared object into smaller pieces and sends them as the instruction data of
+[`Write`](https://github.com/solana-labs/solana/blob/bc7133d7526a041d1aaee807b80922baa89b6f90/sdk/program/src/loader_instruction.rs#L13)
+instructions to the loader where loader writes that data into the program's
+account data. Once all the pieces are received the client sends a
+[`Finalize`](https://github.com/solana-labs/solana/blob/bc7133d7526a041d1aaee807b80922baa89b6f90/sdk/program/src/loader_instruction.rs#L30)
+instruction to the loader, the loader then validates that the BPF data is valid
+and marks the program account as _executable_. Once the program account is
+marked executable, subsequent transactions may issue instructions for that
+program to process.
+
+When an instruction is directed at an executable BPF program the loader
+configures the program's execution environment, serializes the program's input
+parameters, calls the program's entrypoint, and reports any errors encountered.
+
+For further information see [deploying](deploying.md)
+
+### Input Parameter Serialization
+
+BPF loaders serialize the program input parameters into a byte array that is
+then passed to the program's entrypoint, where the program is responsible for
+deserializing it on-chain. One of the changes between the deprecated loader and
+the current loader is that the input parameters are serialized in a way that
+results in various parameters falling on aligned offsets within the aligned byte
+array. This allows deserialization implementations to directly reference the
+byte array and provide aligned pointers to the program.
+
+For language specific information about serialization see:
+
+- [Rust program parameter
+  deserialization](developing-rust.md#parameter-deserialization)
+- [C program parameter
+  deserialization](developing-c.md#parameter-deserialization)
+
+The latest loader serializes the program input parameters as follows (all
+encoding is little endian):
+
+- 8 byte unsigned number of accounts
+- For each account
+  - 1 byte indicating if this is a duplicate account, if not a duplicate then
+    the value is 0xff, otherwise the value is the index of the account it is a
+    duplicate of.
+  - If duplicate: 7 bytes of padding
+  - If not duplicate:
+    - 1 byte boolean, true if account is a signer
+    - 1 byte boolean, true if account is writable
+    - 1 byte boolean, true if account is executable
+    - 4 bytes of padding
+    - 32 bytes of the account public key
+    - 32 bytes of the account's owner public key
+    - 8 byte unsigned number of lamports owned by the account
+    - 8 bytes unsigned number of bytes of account data
+    - x bytes of account data
+    - 10k bytes of padding, used for realloc
+    - enough padding to align the offset to 8 bytes.
+    - 8 bytes rent epoch
+- 8 bytes of unsigned number of instruction data
+- x bytes of instruction data
+- 32 bytes of the program id

package/src/developing/plugins/accountsdb_plugin.md

@@ -0,0 +1,360 @@
+---
+title: Plugins
+---
+
+## Overview
+
+Validators under heavy RPC loads, such as when serving getProgramAccounts calls,
+can fall behind the network. To solve this problem, the validator has been
+enhanced to support a plugin mechanism through which the information about
+accounts and slots can be transmitted to external data stores such as relational
+databases, NoSQL databases or Kafka. RPC services then can be developed to
+consume data from these external data stores with the possibility of more
+flexible and targeted optimizations such as caching and indexing. This allows
+the validator to focus on processing transactions without being slowed down by
+busy RPC requests.
+
+This document describes the interfaces of the plugin and the referential plugin
+implementation for the PostgreSQL database.
+
+[crates.io]: https://crates.io/search?q=solana-
+[docs.rs]: https://docs.rs/releases/search?query=solana-
+
+### Important Crates:
+
+- [`solana-accountsdb-plugin-interface`] — This crate defines the plugin
+interfaces.
+
+- [`solana-accountsdb-plugin-postgres`] — The crate for the referential
+plugin implementation for the PostgreSQL database.
+
+[`solana-accountsdb-plugin-interface`]: https://docs.rs/solana-accountsdb-plugin-interface
+[`solana-accountsdb-plugin-postgres`]: https://docs.rs/solana-accountsdb-plugin-postgres
+
+
+## The Plugin Interface
+
+The Plugin interface is declared in [`solana-accountsdb-plugin-interface`]. It
+is defined by the trait `AccountsDbPlugin`. The plugin should implement the
+trait and expose a "C" function `_create_plugin` to return the pointer to this
+trait. For example, in the referential implementation, the following code
+instantiates the PostgreSQL plugin `AccountsDbPluginPostgres ` and returns its
+pointer.
+
+```
+#[no_mangle]
+#[allow(improper_ctypes_definitions)]
+/// # Safety
+///
+/// This function returns the AccountsDbPluginPostgres pointer as trait AccountsDbPlugin.
+pub unsafe extern "C" fn _create_plugin() -> *mut dyn AccountsDbPlugin {
+    let plugin = AccountsDbPluginPostgres::new();
+    let plugin: Box<dyn AccountsDbPlugin> = Box::new(plugin);
+    Box::into_raw(plugin)
+}
+```
+
+A plugin implementation can implement the `on_load` method to initialize itself.
+This function is invoked after a plugin is dynamically loaded into the validator
+when it starts. The configuration of the plugin is controlled by a configuration
+file in JSON format. The JSON file must have a field `libpath` that points
+to the full path name of the shared library implementing the plugin, and may
+have other configuration information, like connection parameters for the external
+database. The plugin configuration file is specified by the validator's CLI
+parameter `--accountsdb-plugin-config` and the file must be readable to the
+validator process.
+
+Please see the [config file](#config) for the referential
+PostgreSQL plugin below for an example.
+
+The plugin can implement the `on_unload` method to do any cleanup before the
+plugin is unloaded when the validator is gracefully shutdown.
+
+The following method is used for notifying on an account update:
+
+```
+    fn update_account(
+        &mut self,
+        account: ReplicaAccountInfoVersions,
+        slot: u64,
+        is_startup: bool,
+    ) -> Result<()>
+```
+
+The `ReplicaAccountInfoVersions` struct contains the metadata and data of the account
+streamed. The `slot` points to the slot the account is being updated at. When
+`is_startup` is true, it indicates the account is loaded from snapshots when
+the validator starts up. When `is_startup` is false, the account is updated
+when processing a transaction.
+
+
+The following method is called when all accounts have been notified when the
+validator restores the AccountsDb from snapshots at startup.
+
+```
+fn notify_end_of_startup(&mut self) -> Result<()>
+```
+
+When `update_account` is called during processing transactions, the plugin
+should process the notification as fast as possible because any delay may
+cause the validator to fall behind the network. Persistence to external data
+store is best to be done asynchronously.
+
+The following method is used for notifying slot status changes:
+
+```
+    fn update_slot_status(
+        &mut self,
+        slot: u64,
+        parent: Option<u64>,
+        status: SlotStatus,
+    ) -> Result<()>
+```
+
+To ensure data consistency, the plugin implementation can choose to abort
+the validator in case of error persisting to external stores. When the
+validator restarts the account data will be re-transmitted.
+
+For more details, please refer to the Rust documentation in
+[`solana-accountsdb-plugin-interface`].
+
+## Example PostgreSQL Plugin
+
+The [`solana-accountsdb-plugin-postgres`] crate implements a plugin storing
+account data to a PostgreSQL database to illustrate how a plugin can be
+developed.
+
+<a name="config">
+### Configuration File Format
+</a>
+
+The plugin is configured using the input configuration file. An example
+configuration file looks like the following:
+
+
+```
+{
+	"libpath": "/solana/target/release/libsolana_accountsdb_plugin_postgres.so",
+	"host": "postgres-server",
+	"user": "solana",
+	"port": 5433,
+	"threads": 20,
+	"batch_size": 20,
+	"panic_on_db_errors": true,
+	"accounts_selector" : {
+		"accounts" : ["*"]
+	}
+}
+```
+
+The `host`, `user`, and `port` control the PostgreSQL configuration
+information. For more advanced connection options, please use the
+`connection_str` field. Please see [Rust postgres configuration]
+(https://docs.rs/postgres/0.19.2/postgres/config/struct.Config.html).
+
+To improve the throughput to the database, the plugin supports connection pooling
+using multiple threads, each maintaining a connection to the PostgreSQL database.
+The count of the threads is controlled by the `threads` field. A higher thread
+count usually offers better performance.
+
+To further improve performance when saving large numbers of accounts at
+startup, the plugin uses bulk inserts. The batch size is controlled by the
+`batch_size` parameter. This can help reduce the round trips to the database.
+
+The `panic_on_db_errors` can be used to panic the validator in case of database
+errors to ensure data consistency.
+
+### Account Selection
+
+The `accounts_selector` can be used to filter the accounts that should be persisted.
+
+For example, one can use the following to persist only the accounts with particular
+Base58-encoded Pubkeys,
+
+```
+    "accounts_selector" : {
+         "accounts" : ["pubkey-1", "pubkey-2", ..., "pubkey-n"],
+    }
+```
+
+Or use the following to select accounts with certain program owners:
+
+```
+    "accounts_selector" : {
+         "owners" : ["pubkey-owner-1", "pubkey-owner-2", ..., "pubkey-owner-m"],
+    }
+```
+
+To select all accounts, use the wildcard character (*):
+
+```
+    "accounts_selector" : {
+         "accounts" : ["*"],
+    }
+```
+
+
+### Database Setup
+
+#### Install PostgreSQL Server
+
+Please follow [PostgreSQL Ubuntu Installation](https://www.postgresql.org/download/linux/ubuntu/)
+on instructions to install the PostgreSQL database server. For example, to
+install postgresql-14,
+
+```
+sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
+wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
+sudo apt-get update
+sudo apt-get -y install postgresql-14
+```
+#### Control the Database Access
+
+Modify the pg_hba.conf as necessary to grant the plugin to access the database.
+For example, in /etc/postgresql/14/main/pg_hba.conf, the following entry allows
+nodes with IPs in the CIDR 10.138.0.0/24 to access all databases. The validator
+runs in a node with an ip in the specified range.
+
+```
+host    all             all             10.138.0.0/24           trust
+```
+
+It is recommended to run the database server on a separate node from the validator for
+better performance.
+
+#### Configure the Database Performance Parameters
+
+Please refer to the [PostgreSQL Server Configuration](https://www.postgresql.org/docs/14/runtime-config.html)
+for configuration details. The referential implementation uses the following
+configurations for better database performance in the /etc/postgresql/14/main/postgresql.conf
+which are different from the default postgresql-14 installation.
+
+```
+max_connections = 200                  # (change requires restart)
+shared_buffers = 1GB                   # min 128kB
+effective_io_concurrency = 1000        # 1-1000; 0 disables prefetching
+wal_level = minimal                    # minimal, replica, or logical
+fsync = off                            # flush data to disk for crash safety
+synchronous_commit = off               # synchronization level;
+full_page_writes = off                 # recover from partial page writes
+max_wal_senders = 0                    # max number of walsender processes
+```
+
+The sample [postgresql.conf](https://github.com/solana-labs/solana/blob/7ac43b16d2c766df61ae0a06d7aaf14ba61996ac/accountsdb-plugin-postgres/scripts/postgresql.conf)
+can be used for reference.
+
+#### Create the Database Instance and the Role
+
+Start the server:
+
+```
+sudo systemctl start postgresql@14-main
+```
+
+Create the database. For example, the following creates a database named 'solana':
+
+```
+sudo -u postgres createdb solana -p 5433
+```
+
+Create the database user. For example, the following creates a regular user named 'solana':
+
+```
+sudo -u postgres createuser -p 5433 solana
+```
+
+Verify the database is working using psql. For example, assuming the node running
+PostgreSQL has the ip 10.138.0.9, the following command will land in a shell where
+SQL commands can be entered:
+
+```
+psql -U solana -p 5433 -h 10.138.0.9 -w -d solana
+```
+
+#### Create the Schema Objects
+
+Use the [create_schema.sql](https://github.com/solana-labs/solana/blob/7ac43b16d2c766df61ae0a06d7aaf14ba61996ac/accountsdb-plugin-postgres/scripts/create_schema.sql)
+to create the objects for storing accounts and slots.
+
+Download the script from github:
+
+```
+wget https://raw.githubusercontent.com/solana-labs/solana/7ac43b16d2c766df61ae0a06d7aaf14ba61996ac/accountsdb-plugin-postgres/scripts/create_schema.sql
+```
+
+Then run the script:
+
+```
+psql -U solana -p 5433 -h 10.138.0.9 -w -d solana -f create_schema.sql
+```
+
+After this, start the validator with the plugin by using the `--accountsdb-plugin-config`
+argument mentioned above.
+
+#### Destroy the Schema Objects
+
+To destroy the database objects, created by `create_schema.sql`, use
+[drop_schema.sql](https://github.com/solana-labs/solana/blob/7ac43b16d2c766df61ae0a06d7aaf14ba61996ac/accountsdb-plugin-postgres/scripts/drop_schema.sql).
+For example,
+
+```
+psql -U solana -p 5433 -h 10.138.0.9 -w -d solana -f drop_schema.sql
+```
+
+### Capture Historical Account Data
+
+The account historical data is captured using a database trigger as shown in
+`create_schema.sql`,
+
+```
+CREATE FUNCTION audit_account_update() RETURNS trigger AS $audit_account_update$
+    BEGIN
+		INSERT INTO account_audit (pubkey, owner, lamports, slot, executable, rent_epoch, data, write_version, updated_on)
+            VALUES (OLD.pubkey, OLD.owner, OLD.lamports, OLD.slot,
+                    OLD.executable, OLD.rent_epoch, OLD.data, OLD.write_version, OLD.updated_on);
+        RETURN NEW;
+    END;
+
+$audit_account_update$ LANGUAGE plpgsql;
+
+CREATE TRIGGER account_update_trigger AFTER UPDATE OR DELETE ON account
+    FOR EACH ROW EXECUTE PROCEDURE audit_account_update();
+```
+
+The historical data is stored in the account_audit table.
+
+The trigger can be dropped to disable this feature, for example,
+
+
+```
+DROP TRIGGER account_update_trigger ON account;
+```
+
+Over time, the account_audit can accumulate large amount of data. You may choose to
+limit that by deleting older historical data.
+
+
+For example, the following SQL statement can be used to keep up to 1000 of the most
+recent records for an account:
+
+```
+delete from account_audit a2 where (pubkey, write_version) in
+    (select pubkey, write_version from
+        (select a.pubkey, a.updated_on, a.slot, a.write_version, a.lamports,
+            rank() OVER ( partition by pubkey order by write_version desc) as rnk
+            from account_audit a) ranked
+            where ranked.rnk > 1000)
+```
+
+### Performance Considerations
+
+When a validator lacks sufficient compute power, the overhead of saving the
+account data can cause it to fall behind the network especially when all
+accounts or a large number of accounts are selected. The node hosting the
+PostgreSQL database need to be powerful enough to handle the database loads
+as well. It has been found using GCP n2-standard-64 machine type for the
+validator and n2-highmem-32 for the PostgreSQL node is adequate for handling
+transmiting all accounts while keeping up with the network. In addition, it is
+best to keep the validator and the PostgreSQL in the same local network to
+reduce latency. You may need to size the validator and database nodes
+differently if serving other loads.