@blinklabs/dingo 0.19.0 → 0.21.0

package/README.md

@@ -13,7 +13,15 @@
 
 ⚠️ This is a work in progress and is currently under heavy development
 
-**Note:** On Windows systems, named pipes are used instead of Unix sockets for node-to-client communication.
+A high-performance Cardano blockchain node implementation in Go by Blink Labs. Dingo provides:
+- Full chain synchronization and validation via Ouroboros consensus protocol
+- UTxO tracking with 41 UTXO validation rules and Plutus smart contract execution
+- Client connectivity for wallets and applications
+- Pluggable storage backends (Badger, SQLite, PostgreSQL, GCS, S3)
+- Peer governance with dynamic peer selection and topology support
+- Chain rollback support for handling forks with automatic state restoration
+
+Note: On Windows systems, named pipes are used instead of Unix sockets for node-to-client communication.
 
 <div align="center">
   <img src="./.github/dingo-20241210.png" alt="dingo screenshot" width="640">
@@ -77,6 +85,7 @@ Dingo supports pluggable storage backends for both blob storage (blocks, transac
 
 **Metadata Storage Plugins:**
 - `sqlite` - SQLite relational database (default)
+- `postgres` - PostgreSQL relational database
 
 ### Plugin Selection
 
@@ -112,7 +121,6 @@ Each plugin supports specific configuration options. See `dingo.yaml.example` fo
 - `bucket` - GCS bucket name
 - `project-id` - Google Cloud project ID
 - `prefix` - Path prefix within bucket
-- `credentials-file` - Path to service account credentials file (optional - uses Application Default Credentials if not provided)
 
 **AWS S3 Options:**
 - `bucket` - S3 bucket name
@@ -124,6 +132,13 @@ Each plugin supports specific configuration options. See `dingo.yaml.example` fo
 **SQLite Options:**
 - `data-dir` - Path to SQLite database file
 
+**PostgreSQL Options:**
+- `host` - PostgreSQL server hostname
+- `port` - PostgreSQL server port
+- `username` - Database user
+- `password` - Database password
+- `database` - Database name
+
 ### Listing Available Plugins
 
 You can see all available plugins and their descriptions:
@@ -165,10 +180,12 @@ testing, so success/failure reports are very welcome and encouraged!
       - [x] LocalTxMonitor
       - [x] LocalTxSubmission
       - [x] LocalStateQuery
-    - [ ] Peer governor
+    - [x] Peer governor
       - [x] Topology config
-      - [ ] Peer churn
-      - [ ] Ledger peers
+      - [x] Peer churn (full PeerChurnEvent with gossip/public root churn, bootstrap events)
+      - [x] Ledger peers
+      - [x] Peer sharing
+      - [x] Denied peers tracking
     - [x] Connection manager
       - [x] Inbound connections
         - [x] Node-to-client over TCP
@@ -179,12 +196,14 @@ testing, so success/failure reports are very welcome and encouraged!
 - [ ] Ledger
   - [x] Blocks
     - [x] Block storage
-    - [ ] Chain selection
+    - [x] Chain selection (density comparison, VRF tie-breaker, ChainForkEvent)
   - [x] UTxO tracking
   - [x] Protocol parameters
+  - [x] Genesis validation
   - [ ] Certificates
     - [x] Pool registration
     - [x] Stake registration/delegation
+    - [x] Account registration checks
     - [ ] Governance
   - [ ] Transaction validation
     - [ ] Phase 1 validation
@@ -202,6 +221,14 @@ testing, so success/failure reports are very welcome and encouraged!
   - [x] Validation of transaction on add
   - [x] Consumer tracking
   - [x] Transaction purging on chain update
+- [x] Database Recovery
+  - [x] Chain rollback support (SQLite and PostgreSQL plugins)
+  - [x] State restoration on rollback
+  - [x] WAL mode for crash recovery
+  - [x] Automatic rollback on transaction error
+- [x] Plutus Validation
+  - [x] Plutus V3 smart contract validation
+  - [ ] Plutus V1/V2 smart contract validation
 
 Additional planned features can be found in our issue tracker and project boards.
 

package/dingo.yaml.example

@@ -32,8 +32,6 @@ database:
       project-id: ""
       # Path prefix within the bucket
       prefix: ""
-      # Path to service account credentials file (optional - uses Application Default Credentials if not set)
-      credentials-file: ""
     s3:
       # AWS S3 bucket name
       bucket: ""
@@ -48,12 +46,50 @@ database:
 
   # Metadata storage plugin configuration
   metadata:
-    # Plugin to use for metadata storage (sqlite)
+    # Plugin to use for metadata storage (sqlite, postgres, mysql)
     plugin: "sqlite"
     # Configuration options for each plugin
     sqlite:
-      # Data directory for SQLite database file
+      # Path to SQLite database file
       data-dir: ".dingo/metadata.db"
+    postgres:
+      # NOTE: These are example values for local development only.
+      # Do not use these credentials in production environments.
+      # Postgres host
+      host: "localhost"
+      # Postgres port
+      port: 5432
+      # Postgres user
+      user: "postgres"
+      # Postgres password (required - no default)
+      password: ""
+      # Postgres database name
+      database: "postgres"
+      # Postgres sslmode
+      ssl-mode: "disable"
+      # Postgres TimeZone
+      timezone: "UTC"
+      # Full Postgres DSN (overrides other options when set)
+      dsn: ""
+    mysql:
+      # NOTE: These are example values for local development only.
+      # Do not use these credentials in production environments.
+      # MySQL host
+      host: "localhost"
+      # MySQL port
+      port: 3306
+      # MySQL user
+      user: "root"
+      # MySQL password (required - no default)
+      password: ""
+      # MySQL database name
+      database: "mysql"
+      # MySQL TLS mode (mapped to tls= in DSN)
+      ssl-mode: ""
+      # MySQL time zone location
+      timezone: "UTC"
+      # Full MySQL DSN (overrides other options when set)
+      dsn: ""
 
 # Path to the UNIX domain socket file used by the server
 socketPath: "dingo.socket"
@@ -100,9 +136,44 @@ intersectTip: false
 # Default: 1048576 (1 MB)
 mempoolCapacity: 1048576
 
-# Enable development mode which prevents outbound connections
-# Default: false
-devMode: false
+# Operational mode: "serve" (default), "load", or "dev"
+# - serve: Full node with network connectivity (default)
+# - load: Batch import from ImmutableDB (requires immutableDbPath)
+# - dev: Development mode (forge blocks, disable outbound, skip topology)
+# Note: CLI commands (serve, load) take priority over this setting
+#
+# Can be overridden with the DINGO_RUN_MODE environment variable
+runMode: "serve"
+
+# Path to ImmutableDB for batch import (used when runMode is "load")
+# Can also be provided as argument to 'dingo load' command
+#
+# Can be overridden with the DINGO_IMMUTABLE_DB_PATH environment variable
+immutableDbPath: ""
 
 # Validate historical blocks during ledger processing (default: false)
 validateHistorical: false
+
+# Peer targets - target number of peers in each state
+# These are goals the system works toward, not hard limits.
+# Use 0 for default values, -1 for unlimited
+# Default: known=150, established=50, active=20
+#
+# Target number of known (cold) peers
+targetNumberOfKnownPeers: 0
+# Target number of established (warm) peers
+targetNumberOfEstablishedPeers: 0
+# Target number of active (hot) peers
+targetNumberOfActivePeers: 0
+
+# Per-source quotas for active peers
+# These specify how active peer slots are distributed by source.
+# Use 0 for default values.
+# Default: topology=3, gossip=12, ledger=5
+#
+# Active peer slots for topology sources (local + public roots)
+activePeersTopologyQuota: 0
+# Active peer slots for gossip sources
+activePeersGossipQuota: 0
+# Active peer slots for ledger sources
+activePeersLedgerQuota: 0

package/generate_benchmarks.sh

@@ -281,10 +281,10 @@ $benchmark"
     done < <(list_previous_benchmarks)
 
     # Count items (remove empty lines and count)
-    faster_count=$(echo "$faster_benchmarks" | grep -c "^." || echo "0")
-    slower_count=$(echo "$slower_benchmarks" | grep -c "^." || echo "0")
-    new_count=$(echo "$new_benchmarks" | grep -c "^." || echo "0")
-    removed_count=$(echo "$removed_benchmarks" | grep -c "^." || echo "0")
+    faster_count=$(echo "$faster_benchmarks" | sed '/^$/d' | wc -l)
+    slower_count=$(echo "$slower_benchmarks" | sed '/^$/d' | wc -l)
+    new_count=$(echo "$new_benchmarks" | sed '/^$/d' | wc -l)
+    removed_count=$(echo "$removed_benchmarks" | sed '/^$/d' | wc -l)
 
     echo ""
     echo "Performance Changes Summary:"
@@ -362,10 +362,10 @@ $benchmark"
             done < <(list_previous_benchmarks)
 
             # Count items
-            faster_count=$(echo "$faster_benchmarks" | grep -c "^." || echo "0")
-            slower_count=$(echo "$slower_benchmarks" | grep -c "^." || echo "0")
-            new_count=$(echo "$new_benchmarks" | grep -c "^." || echo "0")
-            removed_count=$(echo "$removed_benchmarks" | grep -c "^." || echo "0")
+            faster_count=$(echo "$faster_benchmarks" | sed '/^$/d' | wc -l)
+            slower_count=$(echo "$slower_benchmarks" | sed '/^$/d' | wc -l)
+            new_count=$(echo "$new_benchmarks" | sed '/^$/d' | wc -l)
+            removed_count=$(echo "$removed_benchmarks" | sed '/^$/d' | wc -l)
 
             echo "### Summary"
             echo "- **Faster benchmarks**: $faster_count"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@blinklabs/dingo",
-  "version": "0.19.0",
+  "version": "0.21.0",
   "description": "Dingo is a Cardano blockchain data node",
   "main": "index.js",
   "bin": {

package/PLUGIN_DEVELOPMENT.md

@@ -1,306 +0,0 @@
-# Plugin Development Guide
-
-This guide explains how to develop plugins for Dingo's storage system.
-
-## Overview
-
-Dingo supports pluggable storage backends through a registration-based plugin system. Plugins can extend the system with new blob storage (blocks, transactions) and metadata storage (indexes, state) implementations.
-
-## Plugin Types
-
-### Blob Storage Plugins
-Store blockchain data (blocks, transactions, etc.). Examples:
-- `badger` - Local BadgerDB key-value store
-- `gcs` - Google Cloud Storage
-- `s3` - AWS S3
-
-#### Iterator lifetime note
-Blob plugins expose iterators via `NewIterator(txn, opts)`. Items returned by the
-iterator's `Item()` must only be accessed while the transaction used to create
-the iterator is still active — implementations may validate transaction state at
-access time and will return errors if the transaction has been committed or
-rolled back. See `database/types/types.go` `BlobIterator` for details.
-
-### Metadata Storage Plugins
-Store metadata and indexes. Examples:
-- `sqlite` - SQLite relational database
-
-## Plugin Interface
-
-All plugins must implement the `plugin.Plugin` interface:
-
-```go
-type Plugin interface {
-    Start() error
-    Stop() error
-}
-```
-
-## Plugin Registration
-
-Plugins register themselves during package initialization using the `plugin.Register()` function:
-
-```go
-func init() {
-    plugin.Register(plugin.PluginEntry{
-        Type:               plugin.PluginTypeBlob, // or PluginTypeMetadata
-        Name:               "myplugin",
-        Description:        "My custom storage plugin",
-        NewFromOptionsFunc: NewFromCmdlineOptions,
-        Options: []plugin.PluginOption{
-            // Plugin-specific options
-        },
-    })
-}
-```
-
-## Plugin Options
-
-Plugins define configuration options using the `PluginOption` struct:
-
-```go
-plugin.PluginOption{
-    Name:         "data-dir",           // Option name
-    Type:         plugin.PluginOptionTypeString, // Data type
-    Description:  "Data directory path", // Help text
-    DefaultValue: "/tmp/data",         // Default value
-    Dest:         &cmdlineOptions.dataDir, // Destination variable
-}
-```
-
-Supported option types:
-- `PluginOptionTypeString`
-- `PluginOptionTypeBool`
-- `PluginOptionTypeInt`
-- `PluginOptionTypeUint`
-
-## Environment Variables
-
-Plugins automatically support environment variables with the pattern:
-`DINGO_DATABASE_{TYPE}_{PLUGIN}_{OPTION}`
-
-Examples:
-- `DINGO_DATABASE_BLOB_BADGER_DATA_DIR=/data`
-- `DINGO_DATABASE_METADATA_SQLITE_DATA_DIR=/metadata.db`
-
-## YAML Configuration
-
-Plugins can be configured in `dingo.yaml`:
-
-```yaml
-database:
-  blob:
-    plugin: "myplugin"
-    myplugin:
-      option1: "value1"
-      option2: 42
-  metadata:
-    plugin: "sqlite"
-    sqlite:
-      data-dir: "/data/metadata.db"
-```
-
-## Configuration Precedence
-
-1. Command-line flags (highest priority)
-2. Environment variables
-3. YAML configuration
-4. Default values (lowest priority)
-
-## Command Line Options
-
-Plugins support command-line flags with the pattern:
-`--{type}-{plugin}-{option}`
-
-Examples:
-- `--blob-badger-data-dir /data`
-- `--metadata-sqlite-data-dir /metadata.db`
-
-## Plugin Development Steps
-
-### 1. Create Plugin Structure
-
-```text
-database/plugin/{type}/{name}/
-├── plugin.go      # Registration and options
-├── options.go     # Option functions
-├── database.go    # Core implementation
-└── options_test.go # Unit tests
-```
-
-### 2. Implement Core Plugin
-
-Create the main plugin struct that implements `plugin.Plugin`:
-
-```go
-type MyPlugin struct {
-    // Fields
-}
-
-func (p *MyPlugin) Start() error {
-    // Initialize resources
-    return nil
-}
-
-func (p *MyPlugin) Stop() error {
-    // Clean up resources
-    return nil
-}
-```
-
-### 3. Define Options
-
-Create option functions following the pattern:
-
-```go
-func WithOptionName(value Type) OptionFunc {
-    return func(p *MyPlugin) {
-        p.field = value
-    }
-}
-```
-
-### 4. Implement Constructors
-
-Provide both options-based and legacy constructors:
-
-```go
-func NewWithOptions(opts ...OptionFunc) (*MyPlugin, error) {
-    p := &MyPlugin{}
-    for _, opt := range opts {
-        opt(p)
-    }
-    return p, nil
-}
-
-func New(legacyParam1, legacyParam2) (*MyPlugin, error) {
-    // For backward compatibility
-    return NewWithOptions(
-        WithOption1(legacyParam1),
-        WithOption2(legacyParam2),
-    )
-}
-```
-
-### 5. Register Plugin
-
-In `plugin.go`, register during initialization:
-
-```go
-var cmdlineOptions struct {
-    option1 string
-    option2 int
-}
-
-func init() {
-    plugin.Register(plugin.PluginEntry{
-        Type: plugin.PluginTypeBlob,
-        Name: "myplugin",
-        Description: "My custom plugin",
-        NewFromOptionsFunc: NewFromCmdlineOptions,
-        Options: []plugin.PluginOption{
-            {
-                Name: "option1",
-                Type: plugin.PluginOptionTypeString,
-                Description: "First option",
-                DefaultValue: "default",
-                Dest: &cmdlineOptions.option1,
-            },
-            // More options...
-        },
-    })
-}
-
-func NewFromCmdlineOptions() plugin.Plugin {
-    p, err := NewWithOptions(
-        WithOption1(cmdlineOptions.option1),
-        WithOption2(cmdlineOptions.option2),
-    )
-    if err != nil {
-        panic(err)
-    }
-    return p
-}
-```
-
-### 6. Add Tests
-
-Create comprehensive tests:
-
-```go
-func TestOptions(t *testing.T) {
-    // Test option functions
-}
-
-func TestLifecycle(t *testing.T) {
-    p, err := NewWithOptions(WithOption1("test"))
-    // Test Start/Stop
-}
-```
-
-### 7. Update Imports
-
-Add your plugin to the import list in the appropriate store file:
-- `database/plugin/blob/blob.go` for blob plugins
-- `database/plugin/metadata/metadata.go` for metadata plugins
-
-## Example: Complete Plugin
-
-See the existing plugins for complete examples:
-- `database/plugin/blob/badger/` - BadgerDB implementation
-- `database/plugin/metadata/sqlite/` - SQLite implementation
-- `database/plugin/blob/gcs/` - Google Cloud Storage implementation
-- `database/plugin/blob/aws/` - AWS S3 implementation
-
-## Best Practices
-
-1. **Error Handling**: Always return descriptive errors
-2. **Resource Management**: Properly implement Start/Stop for resource lifecycle
-3. **Thread Safety**: Ensure plugins are safe for concurrent use
-4. **Configuration Validation**: Validate configuration during construction
-5. **Backward Compatibility**: Maintain compatibility with existing deployments
-6. **Documentation**: Document all options and their effects
-7. **Testing**: Provide comprehensive unit and integration tests
-
-## Testing Your Plugin
-
-### Unit Tests
-Test individual components and option functions.
-
-### Integration Tests
-Test the complete plugin lifecycle and interaction with the plugin system.
-
-### CLI Testing
-Use the CLI to test plugin listing and selection:
-
-```bash
-./dingo --blob list
-./dingo --metadata list
-```
-
-### Configuration Testing
-Test environment variables and YAML configuration:
-
-```bash
-DINGO_DATABASE_BLOB_MYPLUGIN_OPTION1=value ./dingo --blob myplugin
-```
-
-## Programmatic Option Overrides (for tests)
-
-When writing tests or programmatically constructing database instances you can override plugin options
-without importing plugin implementation packages directly by using the plugin registry helper:
-
-```go
-// Set data-dir for the blob plugin to a per-test temp directory
-plugin.SetPluginOption(plugin.PluginTypeBlob, "badger", "data-dir", t.TempDir())
-
-// Set data-dir for the metadata plugin
-plugin.SetPluginOption(plugin.PluginTypeMetadata, "sqlite", "data-dir", t.TempDir())
-```
-
-The helper sets the plugin option's destination variable in the registry before plugin instantiation.
-If the requested option is not defined by the targeted plugin the call is non-fatal and returns nil,
-allowing tests to run regardless of which plugin implementation is selected.
-
-Using `t.TempDir()` guarantees each test uses its own on-disk path and prevents concurrent tests from
-colliding on shared directories (for example the default `.dingo` Badger directory).