typenative 0.1.3 → 0.2.0

package/govendor/github.com/ncruces/go-sqlite3/vfs/README.md

@@ -0,0 +1,133 @@
+# Go SQLite VFS API
+
+This package implements the SQLite [OS Interface](https://sqlite.org/vfs.html) (aka VFS).
+
+It replaces the default SQLite VFS with a **pure Go** implementation,
+and exposes [interfaces](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#VFS)
+that should allow you to implement your own [custom VFSes](#custom-vfses).
+
+See the [support matrix](https://github.com/ncruces/go-sqlite3/wiki/Support-matrix)
+for the list of supported OS and CPU architectures.
+
+Since this is a from scratch reimplementation,
+there are naturally some ways it deviates from the original.
+It's also not as battle tested as the original.
+
+The main differences to be aware of are
+[file locking](#file-locking) and
+[WAL mode](#write-ahead-logging) support.
+
+### File Locking
+
+POSIX advisory locks,
+which SQLite uses on [Unix](https://github.com/sqlite/sqlite/blob/41fda52/src/os_unix.c#L13-L14),
+are [broken by design](https://github.com/sqlite/sqlite/blob/41fda52/src/os_unix.c#L1188-L1276).
+Instead, on Linux and macOS, this package uses
+[OFD locks](https://sourceware.org/glibc/manual/2.25/html_node/Open-File-Description-Locks.html)
+to synchronize access to database files.
+OFD locks require [Linux 3.15](https://lwn.net/Articles/586904/).
+
+This package can also use
+[BSD locks](https://man.freebsd.org/cgi/man.cgi?query=flock&sektion=2),
+albeit with reduced concurrency (`BEGIN IMMEDIATE` behaves like `BEGIN EXCLUSIVE`,
+[docs](https://sqlite.org/lang_transaction.html#immediate)).
+BSD locks are the default on BSD and illumos,
+but you can opt into them with the `sqlite3_flock` build tag.
+
+On Windows, this package uses `LockFileEx` and `UnlockFileEx`,
+like SQLite.
+
+You can also opt into a cross-platform locking implementation
+with the `sqlite3_dotlk` build tag.
+
+Otherwise, file locking is not supported, and you must use
+[`nolock=1`](https://sqlite.org/uri.html#urinolock)
+(or [`immutable=1`](https://sqlite.org/uri.html#uriimmutable))
+to open database files.
+To use the [`database/sql`](https://pkg.go.dev/database/sql) driver
+with `nolock=1` you must disable connection pooling by calling
+[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
+
+You can use [`vfs.SupportsFileLocking`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsFileLocking)
+to check if your build supports file locking.
+
+### Write-Ahead Logging
+
+On Unix, this package uses `mmap` to implement
+[shared-memory for the WAL-index](https://sqlite.org/wal.html#implementation_of_shared_memory_for_the_wal_index),
+like SQLite.
+
+On Windows, this package uses `MapViewOfFile`, like SQLite.
+
+You can also opt into a cross-platform, in-process, memory sharing implementation
+with the `sqlite3_dotlk` build tag.
+This implementation [shares memory through copying](https://github.com/ncruces/go-sqlite3/wiki/Sharing-memory-through-copying),
+and is best used with moderate concurrency. Call
+[`db.SetMaxOpenConns(max(2, runtime.GOMAXPROCS(0)))`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns)
+to avoid excessive concurrency.
+
+Otherwise, [WAL support is limited](https://sqlite.org/wal.html#noshm),
+and `EXCLUSIVE` locking mode must be set to create, read, and write WAL databases.
+To use `EXCLUSIVE` locking mode with the
+[`database/sql`](https://pkg.go.dev/database/sql) driver
+you must disable connection pooling by calling
+[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
+
+You can use [`vfs.SupportsSharedMemory`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsSharedMemory)
+to check if your build supports shared memory.
+
+### Blocking Locks
+
+On macOS, this package implements
+[Wal-mode blocking locks](https://sqlite.org/src/doc/tip/doc/wal-lock.md).
+
+### Batch-Atomic Write
+
+On Linux, this package may support
+[batch-atomic writes](https://sqlite.org/cgi/src/technote/714)
+on the F2FS filesystem.
+
+### Checksums
+
+This package can be [configured](https://pkg.go.dev/github.com/ncruces/go-sqlite3#Conn.EnableChecksums)
+to add an 8-byte checksum to the end of every page in an SQLite database.
+The checksum is added as each page is written
+and verified as each page is read.\
+The checksum is intended to help detect database corruption
+caused by random bit-flips in the mass storage device.
+
+The implementation is compatible with SQLite's
+[Checksum VFS Shim](https://sqlite.org/cksumvfs.html).
+
+### Build Tags
+
+The VFS can be customized with a few build tags:
+- `sqlite3_flock` forces the use of BSD locks.
+- `sqlite3_dotlk` forces the use of dot-file locks.
+
+> [!IMPORTANT]
+> The default configuration of this package is compatible with the standard
+> [Unix and Windows SQLite VFSes](https://sqlite.org/vfs.html#multiple_vfses);
+> `sqlite3_flock` builds are compatible with the
+> [`unix-flock` VFS](https://sqlite.org/compile.html#enable_locking_style);
+> `sqlite3_dotlk` builds are compatible with the
+> [`unix-dotfile` VFS](https://sqlite.org/compile.html#enable_locking_style).
+
+> [!CAUTION]
+> Concurrently accessing databases using incompatible VFSes
+> will eventually corrupt data.
+
+### Custom VFSes
+
+- [`github.com/ncruces/go-sqlite3/vfs/memdb`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/memdb)
+  implements an in-memory VFS.
+- [`github.com/ncruces/go-sqlite3/vfs/mvcc`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/mvcc)
+  implements an in-memory MVCC VFS.
+- [`github.com/ncruces/go-sqlite3/vfs/readervfs`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/readervfs)
+  implements a VFS for immutable databases.
+- [`github.com/ncruces/go-sqlite3/vfs/adiantum`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/adiantum)
+  wraps a VFS to offer encryption at rest.
+- [`github.com/ncruces/go-sqlite3/vfs/xts`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/xts)
+  wraps a VFS to offer encryption at rest.
+- [`github.com/ncruces/litestream`](https://pkg.go.dev/github.com/ncruces/litestream)
+  implements Litestream [lightweight read-replicas](https://fly.io/blog/litestream-revamped/#lightweight-read-replicas).

package/govendor/github.com/ncruces/go-sqlite3/vfs/api.go

@@ -0,0 +1,239 @@
+// Package vfs wraps the C SQLite VFS API.
+package vfs
+
+import (
+	"io"
+
+	"github.com/ncruces/go-sqlite3/internal/sqlite3_wrap"
+)
+
+// A VFS defines the interface between the SQLite core and the underlying operating system.
+//
+// Use [SystemError], sqlite3.ErrorCode, or sqlite3.ExtendedErrorCode
+// to return specific error codes to SQLite.
+//
+// https://sqlite.org/c3ref/vfs.html
+type VFS interface {
+	Open(name string, flags OpenFlag) (File, OpenFlag, error)
+	Delete(name string, syncDir bool) error
+	Access(name string, flags AccessFlag) (bool, error)
+	FullPathname(name string) (string, error)
+}
+
+// VFSFilename extends VFS with the ability to use Filename
+// objects for opening files.
+//
+// https://sqlite.org/c3ref/filename.html
+type VFSFilename interface {
+	VFS
+	OpenFilename(name *Filename, flags OpenFlag) (File, OpenFlag, error)
+}
+
+// A File represents an open file in the OS interface layer.
+//
+// Use [SystemError], sqlite3.ErrorCode, or sqlite3.ExtendedErrorCode
+// to return specific error codes to SQLite.
+// In particular, sqlite3.BUSY is needed to correctly implement lock methods.
+//
+// https://sqlite.org/c3ref/io_methods.html
+type File interface {
+	io.Closer
+	io.ReaderAt
+	io.WriterAt
+	Truncate(size int64) error
+	Sync(flags SyncFlag) error
+	Size() (int64, error)
+	Lock(lock LockLevel) error
+	Unlock(lock LockLevel) error
+	CheckReservedLock() (bool, error)
+	SectorSize() int
+	DeviceCharacteristics() DeviceCharacteristic
+}
+
+// FileUnwrap should be implemented by a File
+// that wraps another File implementation.
+type FileUnwrap interface {
+	File
+	Unwrap() File
+}
+
+// FileLockState extends File to implement the
+// SQLITE_FCNTL_LOCKSTATE file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntllockstate
+type FileLockState interface {
+	File
+	LockState() LockLevel
+}
+
+// FilePersistWAL extends File to implement the
+// SQLITE_FCNTL_PERSIST_WAL file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal
+type FilePersistWAL interface {
+	File
+	PersistWAL() bool
+	SetPersistWAL(bool)
+}
+
+// FilePowersafeOverwrite extends File to implement the
+// SQLITE_FCNTL_POWERSAFE_OVERWRITE file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpowersafeoverwrite
+type FilePowersafeOverwrite interface {
+	File
+	PowersafeOverwrite() bool
+	SetPowersafeOverwrite(bool)
+}
+
+// FileChunkSize extends File to implement the
+// SQLITE_FCNTL_CHUNK_SIZE file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlchunksize
+type FileChunkSize interface {
+	File
+	ChunkSize(size int)
+}
+
+// FileSizeHint extends File to implement the
+// SQLITE_FCNTL_SIZE_HINT file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlsizehint
+type FileSizeHint interface {
+	File
+	SizeHint(size int64) error
+}
+
+// FileHasMoved extends File to implement the
+// SQLITE_FCNTL_HAS_MOVED file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlhasmoved
+type FileHasMoved interface {
+	File
+	HasMoved() (bool, error)
+}
+
+// FileOverwrite extends File to implement the
+// SQLITE_FCNTL_OVERWRITE file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntloverwrite
+type FileOverwrite interface {
+	File
+	Overwrite() error
+}
+
+// FileSync extends File to implement the
+// SQLITE_FCNTL_SYNC file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlsync
+type FileSync interface {
+	File
+	SyncSuper(super string) error
+}
+
+// FileCommitPhaseTwo extends File to implement the
+// SQLITE_FCNTL_COMMIT_PHASETWO file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlcommitphasetwo
+type FileCommitPhaseTwo interface {
+	File
+	CommitPhaseTwo() error
+}
+
+// FileBatchAtomicWrite extends File to implement the
+// SQLITE_FCNTL_BEGIN_ATOMIC_WRITE, SQLITE_FCNTL_COMMIT_ATOMIC_WRITE
+// and SQLITE_FCNTL_ROLLBACK_ATOMIC_WRITE file control opcodes.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlbeginatomicwrite
+type FileBatchAtomicWrite interface {
+	File
+	BeginAtomicWrite() error
+	CommitAtomicWrite() error
+	RollbackAtomicWrite() error
+}
+
+// FileCheckpoint extends File to implement the
+// SQLITE_FCNTL_CKPT_START and SQLITE_FCNTL_CKPT_DONE
+// file control opcodes.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlckptstart
+type FileCheckpoint interface {
+	File
+	CheckpointStart()
+	CheckpointDone()
+}
+
+// FilePragma extends File to implement the
+// SQLITE_FCNTL_PRAGMA file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpragma
+type FilePragma interface {
+	File
+	Pragma(name, value string) (string, error)
+}
+
+// FileBusyHandler extends File to implement the
+// SQLITE_FCNTL_BUSYHANDLER file control opcode.
+//
+// https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlbusyhandler
+type FileBusyHandler interface {
+	File
+	BusyHandler(func() bool)
+}
+
+// FileSharedMemory extends File to possibly implement
+// shared-memory for the WAL-index.
+// The same shared-memory instance must be returned
+// for the entire life of the file.
+// It's OK for SharedMemory to return nil.
+type FileSharedMemory interface {
+	File
+	SharedMemory() SharedMemory
+}
+
+// SharedMemory is a shared-memory WAL-index implementation.
+// Use [NewSharedMemory] to create a shared-memory.
+type SharedMemory interface {
+	shmMap(*sqlite3_wrap.Wrapper, int32, int32, bool) (ptr_t, error)
+	shmLock(int32, int32, _ShmFlag) error
+	shmUnmap(bool)
+	shmBarrier()
+	io.Closer
+}
+
+type blockingSharedMemory interface {
+	SharedMemory
+	shmEnableBlocking(block bool)
+}
+
+// FileMemoryMapper extends File to possibly implement
+// memory mapped files.
+// The same memory mapper instance must be returned
+// for the entire life of the file.
+// It's OK for MemoryMapper to return nil.
+type FileMemoryMapper interface {
+	File
+	MemoryMapper() MemoryMapper
+}
+
+// MemoryMapper is a file mapper implementation.
+// Use [NewMemoryMapper] to create a mapper.
+type MemoryMapper interface {
+	mmapSize(*sqlite3_wrap.Wrapper, ptr_t)
+	fetch(*sqlite3_wrap.Wrapper, int64, int32, ptr_t) error
+	io.Closer
+}
+
+// FileControl makes it easy to forward all fileControl methods,
+// which we want to do for the checksum VFS.
+// However, this is not a safe default, and other VFSes
+// should explicitly wrap the methods they want to wrap.
+type fileControl interface {
+	File
+	fileControl(*sqlite3_wrap.Wrapper, _FcntlOpcode, ptr_t) _ErrorCode
+}
+
+type filePDB interface {
+	File
+	SetDB(any)
+}

package/govendor/github.com/ncruces/go-sqlite3/vfs/cksm.go

@@ -0,0 +1,143 @@
+package vfs
+
+import (
+	"bytes"
+	_ "embed"
+	"encoding/binary"
+
+	"github.com/ncruces/go-sqlite3/internal/errutil"
+	"github.com/ncruces/go-sqlite3/internal/sqlite3_wrap"
+	"github.com/ncruces/go-sqlite3/internal/util"
+)
+
+func cksmWrapFile(file File, flags OpenFlag) File {
+	// Checksum only main databases.
+	if flags&OPEN_MAIN_DB == 0 {
+		return file
+	}
+	return &cksmFile{File: file}
+}
+
+type cksmFile struct {
+	File
+	verifyCksm  bool
+	computeCksm bool
+}
+
+func (c *cksmFile) ReadAt(p []byte, off int64) (n int, err error) {
+	n, err = c.File.ReadAt(p, off)
+	p = p[:n]
+
+	if isHeader(p, off) {
+		c.init((*[100]byte)(p))
+	}
+
+	// Verify checksums.
+	if c.verifyCksm && util.ValidPageSize(len(p)) {
+		cksm1 := cksmCompute(p[:len(p)-8])
+		cksm2 := *(*[8]byte)(p[len(p)-8:])
+		if cksm1 != cksm2 {
+			return 0, _IOERR_DATA
+		}
+	}
+	return n, err
+}
+
+func (c *cksmFile) WriteAt(p []byte, off int64) (n int, err error) {
+	if isHeader(p, off) {
+		c.init((*[100]byte)(p))
+	}
+
+	// Compute checksums.
+	if c.computeCksm && util.ValidPageSize(len(p)) {
+		*(*[8]byte)(p[len(p)-8:]) = cksmCompute(p[:len(p)-8])
+	}
+
+	return c.File.WriteAt(p, off)
+}
+
+func (c *cksmFile) Pragma(name, value string) (string, error) {
+	switch name {
+	case "checksum_verification":
+		b, ok := util.ParseBool(value)
+		if ok {
+			c.verifyCksm = b && c.computeCksm
+		}
+		if !c.verifyCksm {
+			return "0", nil
+		}
+		return "1", nil
+
+	case "page_size":
+		if c.computeCksm && value != "" {
+			// Do not allow page size changes on a checksum database.
+			return "", nil
+		}
+	}
+	return "", _NOTFOUND
+}
+
+func (c *cksmFile) DeviceCharacteristics() DeviceCharacteristic {
+	ret := c.File.DeviceCharacteristics()
+	if c.verifyCksm {
+		ret &^= IOCAP_SUBPAGE_READ
+	}
+	return ret
+}
+
+func (c *cksmFile) fileControl(wrp *sqlite3_wrap.Wrapper, op _FcntlOpcode, pArg ptr_t) _ErrorCode {
+	if op == _FCNTL_PRAGMA {
+		rc := vfsFileControlImpl(wrp, c, op, pArg)
+		if rc != _NOTFOUND {
+			return rc
+		}
+	}
+	return vfsFileControlImpl(wrp, c.File, op, pArg)
+}
+
+func (c *cksmFile) init(header *[100]byte) {
+	if r := header[20] == 8; r != c.computeCksm {
+		c.computeCksm = r
+		c.verifyCksm = r
+	}
+}
+
+func (c *cksmFile) SharedMemory() SharedMemory {
+	if f, ok := c.File.(FileSharedMemory); ok {
+		return f.SharedMemory()
+	}
+	return nil
+}
+
+func (c *cksmFile) MemoryMapper() MemoryMapper {
+	if c.verifyCksm {
+		return nil
+	}
+	if f, ok := c.File.(FileMemoryMapper); ok {
+		return f.MemoryMapper()
+	}
+	return nil
+}
+
+func (c *cksmFile) Unwrap() File {
+	return c.File
+}
+
+func isHeader(p []byte, off int64) bool {
+	return off == 0 && len(p) >= 100 && bytes.HasPrefix(p, []byte("SQLite format 3\000"))
+}
+
+func cksmCompute(a []byte) (cksm [8]byte) {
+	var s1, s2 uint32
+	for len(a) >= 8 {
+		s1 += binary.LittleEndian.Uint32(a[0:4]) + s2
+		s2 += binary.LittleEndian.Uint32(a[4:8]) + s1
+		a = a[8:]
+	}
+	if len(a) != 0 {
+		panic(errutil.AssertErr())
+	}
+	binary.LittleEndian.PutUint32(cksm[0:4], s1)
+	binary.LittleEndian.PutUint32(cksm[4:8], s2)
+	return
+}