@lossless.org/objectstorage 10.7.1 โ 11.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.smartconfig.json +12 -10
- package/dist_rust/ruststorage_linux_amd64 +0 -0
- package/dist_rust/ruststorage_linux_amd64.tsrust-build.json +5 -5
- package/dist_rust/ruststorage_linux_amd64_musl +0 -0
- package/dist_rust/ruststorage_linux_amd64_musl.tsrust-build.json +5 -5
- package/dist_rust/ruststorage_linux_arm64 +0 -0
- package/dist_rust/ruststorage_linux_arm64.tsrust-build.json +5 -5
- package/dist_rust/ruststorage_linux_arm64_musl +0 -0
- package/dist_rust/ruststorage_linux_arm64_musl.tsrust-build.json +5 -5
- package/dist_rust/ruststorage_macos_amd64 +0 -0
- package/dist_rust/ruststorage_macos_amd64.tsrust-build.json +5 -5
- package/dist_rust/ruststorage_macos_arm64 +0 -0
- package/dist_rust/ruststorage_macos_arm64.tsrust-build.json +5 -5
- package/dist_ts/00_commitinfo_data.js +1 -1
- package/dist_ts/index.d.ts +125 -0
- package/dist_ts/index.js +147 -8
- package/package.json +9 -12
- package/readme.md +526 -28
- package/third-party-notices.md +1 -1
- package/third_party_licenses/inventory.json +2 -2
- package/ts/00_commitinfo_data.ts +1 -1
- package/ts/index.ts +311 -13
package/readme.md
CHANGED
|
@@ -69,13 +69,13 @@ described below.
|
|
|
69
69
|
### Clustering Features
|
|
70
70
|
|
|
71
71
|
- ๐ **Erasure coding** โ Reed-Solomon (configurable k data + m parity shards) for storage efficiency and fault tolerance
|
|
72
|
-
- ๐ **QUIC transport** โ multiplexed
|
|
72
|
+
- ๐ **QUIC transport** โ multiplexed inter-node communication via `quinn` with zero head-of-line blocking. Encrypted but not authenticated: run it only on a trusted network (see Cluster transport trust).
|
|
73
73
|
- ๐ฝ **Multi-drive awareness** โ each node manages multiple independent storage paths with health monitoring
|
|
74
74
|
- ๐ฉบ **Cluster health introspection** โ query native node, drive, quorum, and healing status for product dashboards
|
|
75
75
|
- ๐ค **Cluster membership** โ static seed config + runtime join, heartbeat-based failure detection
|
|
76
76
|
- โ๏ธ **Quorum writes** โ data is only acknowledged after k+1 shards are persisted
|
|
77
77
|
- ๐ **Quorum reads** โ reconstruct from any k available shards, local-first fast path
|
|
78
|
-
- ๐ฉน **Self-healing** โ background scanner
|
|
78
|
+
- ๐ฉน **Self-healing** โ background scanner rebuilds missing shards on the drives their manifest names
|
|
79
79
|
|
|
80
80
|
## Installation
|
|
81
81
|
|
|
@@ -114,7 +114,7 @@ glibc LGPL terms, Cargo licenses and native distribution inventory.
|
|
|
114
114
|
## Native builds and verification
|
|
115
115
|
|
|
116
116
|
`pnpm build` builds the current host architecture (GNU and musl on Linux), then
|
|
117
|
-
builds the TypeScript facade. Test preparation uses `
|
|
117
|
+
builds the TypeScript facade. Test preparation uses `rsrepo --configured-targets`
|
|
118
118
|
to cover all configured host-OS binaries in the packaging checks. Rust 1.95.0 is
|
|
119
119
|
pinned in `rust-toolchain.toml`. Mac builders
|
|
120
120
|
need Xcode command-line tools, both Darwin Rust targets and Rosetta 2 for Intel
|
|
@@ -126,7 +126,7 @@ with aliases from their full `*-unknown-linux-musl-gcc` names to the names above
|
|
|
126
126
|
Rust supplies its pinned, patched musl runtime. Native runtime notices are included
|
|
127
127
|
in the package; retain them when distributing standalone binaries.
|
|
128
128
|
|
|
129
|
-
Releases use `
|
|
129
|
+
Releases use `rsrepo matrix build` to assemble all six binaries from one clean
|
|
130
130
|
Git commit. Configure these environment variables on the release coordinator:
|
|
131
131
|
|
|
132
132
|
```sh
|
|
@@ -184,6 +184,10 @@ Cluster mode requires the control root and every drive root to exist before
|
|
|
184
184
|
startup. The process user must own them, no path component may be a symlink,
|
|
185
185
|
and group/world write permission must be disabled.
|
|
186
186
|
|
|
187
|
+
The cluster transport does not authenticate nodes: run cluster mode only on a
|
|
188
|
+
network where every host that can reach `quicPort` is a cluster node (see
|
|
189
|
+
[Cluster transport trust](#cluster-transport-trust)).
|
|
190
|
+
|
|
187
191
|
```typescript
|
|
188
192
|
import { ObjectStorage } from '@lossless.org/objectstorage';
|
|
189
193
|
|
|
@@ -211,6 +215,21 @@ const storage = await ObjectStorage.createAndStart({
|
|
|
211
215
|
|
|
212
216
|
Objects are automatically split into chunks (default 4 MB), erasure-coded into 6 shards (4 data + 2 parity), and distributed across drives/nodes. Any 4 of 6 shards can reconstruct the original data.
|
|
213
217
|
|
|
218
|
+
Each drive root counts as one independent drive, so startup refuses two drive
|
|
219
|
+
roots on the same device (`st_dev`). Only the device number is checked:
|
|
220
|
+
partitions of one disk, LVM volumes, btrfs subvolumes and datasets of one ZFS
|
|
221
|
+
pool report distinct devices, are not detected, and keeping them apart is the
|
|
222
|
+
operator's responsibility. The control root may share a device with a drive.
|
|
223
|
+
For dev and test clusters whose drive roots are directories on one filesystem,
|
|
224
|
+
opt in explicitly; startup then logs which roots share a device:
|
|
225
|
+
|
|
226
|
+
```typescript
|
|
227
|
+
drives: {
|
|
228
|
+
paths: ['/srv/objectstorage-dev/drive-1', '/srv/objectstorage-dev/drive-2', '/srv/objectstorage-dev/drive-3'],
|
|
229
|
+
allowSharedDevice: true, // dev/test only: these roots are not independent drives
|
|
230
|
+
},
|
|
231
|
+
```
|
|
232
|
+
|
|
214
233
|
## Configuration
|
|
215
234
|
|
|
216
235
|
All config fields are optional โ sensible defaults are applied automatically.
|
|
@@ -263,6 +282,7 @@ const config: IObjectStorageConfig = {
|
|
|
263
282
|
multipart: {
|
|
264
283
|
expirationDays: 7, // Must be greater than zero
|
|
265
284
|
cleanupIntervalMinutes: 60, // Must be greater than zero
|
|
285
|
+
maxUploadsPerPrefixScope: 100, // Open uploads per prefix credential scope; must be non-zero
|
|
266
286
|
},
|
|
267
287
|
cluster: { // Optional โ omit for standalone mode
|
|
268
288
|
enabled: true,
|
|
@@ -272,13 +292,15 @@ const config: IObjectStorageConfig = {
|
|
|
272
292
|
erasure: {
|
|
273
293
|
dataShards: 4, // Default: 4
|
|
274
294
|
parityShards: 2, // Default: 2
|
|
275
|
-
chunkSizeBytes: 4194304, // Default: 4 MB
|
|
295
|
+
chunkSizeBytes: 4194304, // Default: 4 MB; between 1 and 64 MiB
|
|
276
296
|
},
|
|
277
297
|
drives: {
|
|
278
298
|
paths: ['/mnt/disk1', '/mnt/disk2'],
|
|
299
|
+
allowSharedDevice: false, // Default: false; true only for dev/test roots on one device
|
|
279
300
|
},
|
|
280
301
|
heartbeatIntervalMs: 5000, // Default: 5000
|
|
281
302
|
heartbeatTimeoutMs: 30000, // Default: 30000
|
|
303
|
+
getDeadlineMs: 21600000, // Default: 6 h; 60000 (1 min) to 604800000 (7 d)
|
|
282
304
|
},
|
|
283
305
|
};
|
|
284
306
|
|
|
@@ -460,6 +482,7 @@ interface IStorageCredential {
|
|
|
460
482
|
- Requests that were already authenticated before the replacement keep running; auth is evaluated when each request starts.
|
|
461
483
|
- No restart is required, and runtime-created credentials survive restart unless `storage.cleanSlate` clears the bounded storage contents; the owned root, maintenance-lock inode, and any durable provider/pool identity are preserved.
|
|
462
484
|
- Replacement input must contain at least one credential, each `accessKeyId` and `secretAccessKey` must be non-empty, and `accessKeyId` values must be unique.
|
|
485
|
+
- A new `accessKeyId` must not be held by a live or revoked prefix credential (`ECREDENTIAL_CONFLICT`); the same check applies to bucket tenant create, ensure and rotate. Prefix credentials are not part of `listCredentials()`; see [Prefix Credentials](#prefix-credentials).
|
|
463
486
|
|
|
464
487
|
## Bucket Tenants
|
|
465
488
|
|
|
@@ -499,6 +522,207 @@ const tenants = await storage.listBucketTenants();
|
|
|
499
522
|
- Tenant credentials can list, read, write, and delete objects in their assigned bucket, but cannot list all buckets, access other buckets, copy from other buckets, delete buckets, or mutate bucket policies.
|
|
500
523
|
- Bucket tenant APIs require `auth.enabled: true`.
|
|
501
524
|
|
|
525
|
+
## Prefix Credentials
|
|
526
|
+
|
|
527
|
+
A prefix credential is an ordinary SigV4 key that reaches only the keys below one
|
|
528
|
+
prefix of one bucket, and only until it expires. It is meant for handing a
|
|
529
|
+
short-lived key to a guest (a VM, a job, a mounted drive) that must see one
|
|
530
|
+
folder and nothing else.
|
|
531
|
+
|
|
532
|
+
```typescript
|
|
533
|
+
const credential = await storage.createPrefixCredential({
|
|
534
|
+
bucketName: 'drives',
|
|
535
|
+
prefix: 'drives/user-42/',
|
|
536
|
+
ttlSeconds: 3600, // 1 to 43200 (12 h)
|
|
537
|
+
});
|
|
538
|
+
// { accessKeyId, secretAccessKey, bucketName, prefix, region, createdAt, expiresAt }
|
|
539
|
+
|
|
540
|
+
const client = new S3Client({
|
|
541
|
+
endpoint: 'http://localhost:3000',
|
|
542
|
+
region: credential.region,
|
|
543
|
+
credentials: {
|
|
544
|
+
accessKeyId: credential.accessKeyId,
|
|
545
|
+
secretAccessKey: credential.secretAccessKey,
|
|
546
|
+
},
|
|
547
|
+
forcePathStyle: true,
|
|
548
|
+
});
|
|
549
|
+
|
|
550
|
+
await storage.revokePrefixCredential({ accessKeyId: credential.accessKeyId }); // { revoked: true }
|
|
551
|
+
const live = await storage.listPrefixCredentials({ bucketName: 'drives' }); // metadata, no secrets
|
|
552
|
+
```
|
|
553
|
+
|
|
554
|
+
```typescript
|
|
555
|
+
interface IPrefixCredentialInput {
|
|
556
|
+
bucketName: string;
|
|
557
|
+
prefix: string;
|
|
558
|
+
ttlSeconds?: number; // first node: lifetime in seconds
|
|
559
|
+
expiresAt?: number; // further nodes: the first node's expiresAt (epoch ms)
|
|
560
|
+
accessKeyId?: string; // supplied together with secretAccessKey and expiresAt only
|
|
561
|
+
secretAccessKey?: string;
|
|
562
|
+
region?: string; // default: server.region
|
|
563
|
+
}
|
|
564
|
+
```
|
|
565
|
+
|
|
566
|
+
**Input rules.**
|
|
567
|
+
- Exactly one of `ttlSeconds` (an integer, 1 to 43 200) and `expiresAt` (an
|
|
568
|
+
integer epoch-ms time after now and at most 43 200 s ahead).
|
|
569
|
+
- Key material is generated when omitted (`SP` plus 20 uppercase hex
|
|
570
|
+
characters, a 64-hex-character secret). Supplied material needs `expiresAt`,
|
|
571
|
+
and both `accessKeyId` and `secretAccessKey`: an ID of 16 to 128 characters
|
|
572
|
+
of `A-Z0-9`, and a secret of 32 to 128 printable ASCII characters without
|
|
573
|
+
spaces.
|
|
574
|
+
- The prefix is validated, never normalised. It is 1 to 1023 bytes of
|
|
575
|
+
`A-Z a-z 0-9 - . _ ~ /` (RFC 3986 unreserved characters plus `/`, no
|
|
576
|
+
percent-encoding). It ends with `/` and does not start with `/`, and has no
|
|
577
|
+
empty, `.` or `..` segment.
|
|
578
|
+
- The bucket must exist. The access key ID must not belong to a static or bucket
|
|
579
|
+
tenant credential, and it must not have been revoked.
|
|
580
|
+
- The secret appears only in the create result. It is never listed, logged, or
|
|
581
|
+
stored in `storage.config`.
|
|
582
|
+
- Errors are `ObjectStorageCredentialError` with a `code`:
|
|
583
|
+
|
|
584
|
+
| Code | Meaning |
|
|
585
|
+
|---|---|
|
|
586
|
+
| `ECREDENTIAL_AUTH_DISABLED` | `auth.enabled` is false |
|
|
587
|
+
| `ECREDENTIAL_INVALID_PREFIX` | the prefix fails the grammar |
|
|
588
|
+
| `ECREDENTIAL_INVALID_TTL` | not exactly one of `ttlSeconds` and `expiresAt`, or `ttlSeconds` out of range |
|
|
589
|
+
| `ECREDENTIAL_INVALID_EXPIRY` | `expiresAt` is not in the future, or is more than 12 h ahead |
|
|
590
|
+
| `ECREDENTIAL_INVALID_KEY` | the key material or the region is malformed, or only part of it was supplied |
|
|
591
|
+
| `ECREDENTIAL_NO_SUCH_BUCKET` | the bucket does not exist |
|
|
592
|
+
| `ECREDENTIAL_BUCKET_UNAVAILABLE` | the bucket is a sealed or retired migration source |
|
|
593
|
+
| `ECREDENTIAL_CONFLICT` | the ID is held by another credential, or a replay differs from the stored record |
|
|
594
|
+
| `ECREDENTIAL_REVOKED` | the ID was revoked |
|
|
595
|
+
| `ECREDENTIAL_NOT_PREFIX` | a revoke named a static or bucket tenant credential |
|
|
596
|
+
| `ECREDENTIAL_LIMIT` | 10 000 live credentials and revocations are stored |
|
|
597
|
+
| `ECREDENTIAL_PERSISTENCE` | the store could not be written |
|
|
598
|
+
| `ECREDENTIAL_STORE_UNAVAILABLE` | the store file is unreadable (see below) |
|
|
599
|
+
|
|
600
|
+
**Access.** A key is covered when its bytes start with the prefix; a key equal
|
|
601
|
+
to the prefix (its folder marker) is covered.
|
|
602
|
+
- **Objects.** GetObject, HeadObject, PutObject, DeleteObject and the multipart
|
|
603
|
+
operations work on covered keys.
|
|
604
|
+
- Writes also refuse keys with `.`, `..` or empty segments after the prefix,
|
|
605
|
+
and keys with control characters.
|
|
606
|
+
- CopyObject and UploadPartCopy need the source to be covered too.
|
|
607
|
+
- Object requests accept only the query parameters the operation uses:
|
|
608
|
+
`uploads`, `uploadId`, `partNumber`, `max-parts` and `part-number-marker`,
|
|
609
|
+
the six `response-*` header overrides on GetObject and HeadObject, plus
|
|
610
|
+
the `x-id` operation label that AWS SDKs append. Any other object query
|
|
611
|
+
(`?acl`, `?tagging`, `?versionId`, `?retention`, โฆ) is refused with
|
|
612
|
+
`AccessDenied` (403), also a name S3 does not define (`?v=1`), and so is
|
|
613
|
+
any repeated query parameter. A prefix
|
|
614
|
+
credential never names a version: `?versionId` is refused even as `null`,
|
|
615
|
+
and so is a copy source with a `?versionId=` suffix.
|
|
616
|
+
- Requests that pass these checks still go through the router's allowlist
|
|
617
|
+
(see [Supported S3 Operations](#supported-s3-operations)), so a parameter
|
|
618
|
+
the operation does not implement gets `NotImplemented` (501).
|
|
619
|
+
- **DeleteObjects.** Every member is checked on its own: covered keys are
|
|
620
|
+
deleted, and the others get a per-key `AccessDenied`.
|
|
621
|
+
- **Listings.** ListObjects V1, ListObjectsV2, ListObjectVersions and
|
|
622
|
+
ListMultipartUploads need a `prefix` parameter that is covered. A missing or
|
|
623
|
+
wider prefix is refused, never clamped.
|
|
624
|
+
- **Bucket level.**
|
|
625
|
+
- ListBuckets returns only the credential's bucket.
|
|
626
|
+
- HeadBucket and `GET /{bucket}?location` work on that bucket.
|
|
627
|
+
- Any other bucket-level query (`?acl`, `?versioning`, `?lifecycle`, โฆ) is
|
|
628
|
+
refused with `AccessDenied`, and so is any repeated query parameter.
|
|
629
|
+
- Creating or deleting buckets and bucket policy operations are refused.
|
|
630
|
+
- **Policies.** The scope check runs before bucket policies, so a policy Allow
|
|
631
|
+
(including `Principal: "*"`) never widens a prefix credential. A policy Deny
|
|
632
|
+
still applies.
|
|
633
|
+
|
|
634
|
+
**Expiry and deadlines.**
|
|
635
|
+
- A credential authenticates while the engine clock is before `expiresAt`.
|
|
636
|
+
After that, or after revocation, requests get `InvalidAccessKeyId` (403).
|
|
637
|
+
- Requests that were admitted end at `expiresAt` as well: a trickled upload
|
|
638
|
+
fails with nothing stored, and a slowly read download is cut off.
|
|
639
|
+
- Revocation affects only requests admitted after it. Requests already running
|
|
640
|
+
continue until they finish or until `expiresAt`.
|
|
641
|
+
- There is no extend operation. To refresh, create a new key and revoke the old
|
|
642
|
+
one.
|
|
643
|
+
- **Multipart cap.** All prefix credentials of one scope (the same bucket and
|
|
644
|
+
prefix) together hold at most `multipart.maxUploadsPerPrefixScope`
|
|
645
|
+
(default 100, must be non-zero) open multipart uploads. The next
|
|
646
|
+
CreateMultipartUpload of that scope gets `AccessDenied` (403) until one of
|
|
647
|
+
them ends: completed or aborted by any credential, or removed by the
|
|
648
|
+
multipart expiry cleanup. Revoking, expiring or replacing a credential
|
|
649
|
+
frees none of its scope's slots, so issuing new keys does not raise the cap.
|
|
650
|
+
The count is held in memory and resets when the engine restarts. The
|
|
651
|
+
engine-wide limit of 100 000 uploads still applies.
|
|
652
|
+
|
|
653
|
+
**Bucket deletion.** Deletion attempts revoke first; callers re-issue.
|
|
654
|
+
- Every path that deletes a bucket revokes the bucket's prefix credentials
|
|
655
|
+
before it removes anything: S3 DeleteBucket, `deleteBucket()`,
|
|
656
|
+
`deleteBucketTenant()` without an `accessKeyId`, and migration source
|
|
657
|
+
cleanup. A recreated bucket of the same name is not reachable with them.
|
|
658
|
+
- S3 DeleteBucket checks that the bucket is empty before it revokes, so a
|
|
659
|
+
`BucketNotEmpty` (409) keeps the credentials.
|
|
660
|
+
- If the revocation cannot be stored, the deletion is refused.
|
|
661
|
+
- A deletion that fails or is refused after the revocation (for example a
|
|
662
|
+
stale resource fence, `EFENCE_STALE`) returns that error, never a success,
|
|
663
|
+
and leaves the credentials revoked. Callers that keep the bucket create new
|
|
664
|
+
prefix credentials.
|
|
665
|
+
|
|
666
|
+
**Persistence.**
|
|
667
|
+
- Credentials and revocations live in `.smartstorage/prefix-credentials.json`,
|
|
668
|
+
under the same root as `credentials.json`. The engine writes the file
|
|
669
|
+
atomically with mode `0600`, and `storage.cleanSlate` clears it.
|
|
670
|
+
- A sweep every 60 seconds drops expired entries.
|
|
671
|
+
- Engines older than prefix credentials never read the file, so a rollback
|
|
672
|
+
disables every prefix credential rather than widening it.
|
|
673
|
+
- **Fail closed.** If the file is unreadable, larger than 32 MiB, malformed,
|
|
674
|
+
or inconsistent (duplicate IDs, an ID held by a static credential, a record
|
|
675
|
+
that fails validation), the store is unavailable:
|
|
676
|
+
- every prefix credential is refused;
|
|
677
|
+
- create, revoke and list fail with `ECREDENTIAL_STORE_UNAVAILABLE`;
|
|
678
|
+
- no new static or tenant access key ID can be introduced, because the ID
|
|
679
|
+
collision check cannot run: `replaceCredentials()`,
|
|
680
|
+
`createBucketTenant()`, `ensureBucketTenant()` and
|
|
681
|
+
`rotateBucketTenantCredentials()` throw `ObjectStorageCredentialError`
|
|
682
|
+
with code `ECREDENTIAL_STORE_UNAVAILABLE` when they would add an access
|
|
683
|
+
key ID that is not in the current set;
|
|
684
|
+
- bucket deletion is refused, because the bucket's credentials cannot be
|
|
685
|
+
revoked: `deleteBucket()` and `deleteBucketTenant()` without an
|
|
686
|
+
`accessKeyId` throw `ObjectStorageCredentialError` with code
|
|
687
|
+
`ECREDENTIAL_STORE_UNAVAILABLE`, and S3 DeleteBucket returns
|
|
688
|
+
`InvalidRequest` (400) with a message starting
|
|
689
|
+
`ECREDENTIAL_STORE_UNAVAILABLE:`;
|
|
690
|
+
- static and tenant credentials keep working;
|
|
691
|
+
- `getHealth().auth.prefixCredentialStoreAvailable` is `false`;
|
|
692
|
+
- the engine logs the condition without secrets and leaves the file as found.
|
|
693
|
+
|
|
694
|
+
To recover, repair or delete the file and restart. Deleting it revokes every
|
|
695
|
+
prefix credential and forgets every revocation, so never reuse a revoked ID
|
|
696
|
+
afterwards. Generated IDs are random.
|
|
697
|
+
|
|
698
|
+
**Cluster mode.** Prefix credentials are node-local, like runtime credentials:
|
|
699
|
+
nothing replicates them, and the caller fans out.
|
|
700
|
+
1. Create on one node with `ttlSeconds`.
|
|
701
|
+
2. Create on every other node with the first node's `accessKeyId`,
|
|
702
|
+
`secretAccessKey`, `expiresAt` and `region`.
|
|
703
|
+
- Each node records its own `createdAt`; every other field matches the first
|
|
704
|
+
node's record.
|
|
705
|
+
- An identical replay returns the stored record unchanged. Any difference
|
|
706
|
+
gives `ECREDENTIAL_CONFLICT`.
|
|
707
|
+
- A late create never extends the lifetime.
|
|
708
|
+
3. To revoke, call `revokePrefixCredential` on every node and retry until each
|
|
709
|
+
node acknowledges.
|
|
710
|
+
- A node that never saw the ID records the revocation for 12 h, so a create
|
|
711
|
+
that reaches it later is refused with `ECREDENTIAL_REVOKED`.
|
|
712
|
+
- `revoked` is `true` only where a live credential was removed.
|
|
713
|
+
|
|
714
|
+
**Revocation guarantee.**
|
|
715
|
+
- Once a node has acknowledged a revocation, it refuses every request admitted
|
|
716
|
+
after the acknowledgement. It also refuses any later create for that ID until
|
|
717
|
+
the ID could no longer be created anyway: an `expiresAt` create must lie
|
|
718
|
+
within 12 h of the first node's create, and the revocation is kept at least
|
|
719
|
+
that long.
|
|
720
|
+
- A node that never acknowledged serves the key until the caller's retry
|
|
721
|
+
reaches it, or until `expiresAt`.
|
|
722
|
+
- The hard bound is `expiresAt`, at most 12 h after the first create. Each node
|
|
723
|
+
judges expiry by its own clock, so clock offsets shift these instants.
|
|
724
|
+
- `listPrefixCredentials()` lets the caller reconcile nodes.
|
|
725
|
+
|
|
502
726
|
## Bucket Backup/Restore
|
|
503
727
|
|
|
504
728
|
```typescript
|
|
@@ -716,7 +940,9 @@ const metrics = await storage.getMetrics();
|
|
|
716
940
|
```
|
|
717
941
|
|
|
718
942
|
- `getHealth()` reports running state, storage directory and pool observation, auth enabled state, credential counts, bucket count, object count, total bytes, and cluster health.
|
|
719
|
-
- `
|
|
943
|
+
- `getHealth().auth` also reports `prefixCredentialCount` (live prefix credentials), `prefixCredentialTombstoneCount` (unexpired revocations) and `prefixCredentialStoreAvailable` (`false` while the prefix credential file is unreadable and every prefix credential is refused).
|
|
944
|
+
- `getMetrics()` returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, prefix credential (`objectstorage_prefix_credentials_total`, `objectstorage_prefix_credential_tombstones_total`, `objectstorage_prefix_credential_store_available`), and cluster-enabled metrics.
|
|
945
|
+
- `getHealth()` reads the prefix credential figures through the `getPrefixCredentialStatus` IPC command (no TypeScript method of its own), and `getMetrics()` derives its prefix metrics from `getHealth()`. The command takes no parameters and returns `{ available, credentialCount, tombstoneCount }`: counts and the store state only, never an access key ID, secret, bucket or prefix.
|
|
720
946
|
|
|
721
947
|
## Runtime Stats
|
|
722
948
|
|
|
@@ -794,6 +1020,123 @@ interface IClusterHealth {
|
|
|
794
1020
|
- Drive health is based on live native probe checks on the configured local drive paths. Capacity values are cheap filesystem snapshots.
|
|
795
1021
|
- `quorumHealthy` means the local node currently sees majority quorum and enough available placements in every erasure set to satisfy the configured write quorum.
|
|
796
1022
|
- Repair fields expose the background healer's currently available runtime state. They are best-effort and limited to what the engine tracks today, such as whether a scan is active, the last completed run, and the last error.
|
|
1023
|
+
- `repairs.shardsChecked`, `shardsHealed` and `failed` count the last completed scan's placement checks, shards written and failures. `repairs.shardsPending` counts the pending shards (see [Self-Healing](#self-healing)) that scan left pending because their drive was out of reach, refused the write, or their object was busy. While a scan runs, `lastRunStartedAt` is the running scan's start and the counts are still the previous scan's; `runHealingScan()` returns one scan's start and counts together.
|
|
1024
|
+
|
|
1025
|
+
## Supported S3 Operations
|
|
1026
|
+
|
|
1027
|
+
The router fails closed, in standalone and cluster mode alike. A request
|
|
1028
|
+
selects one operation from its method, its path (`/`, `/{bucket}`,
|
|
1029
|
+
`/{bucket}/{key}`) and its query names, and each query name must be one that
|
|
1030
|
+
operation implements. Anything else gets `NotImplemented` (501, with the S3 XML
|
|
1031
|
+
error body) before the request takes a lock, reads its body or touches a bucket
|
|
1032
|
+
or object. A credential the request is not allowed for still gets
|
|
1033
|
+
`AccessDenied` (403) first. Until 11.0.0 such requests were served as another
|
|
1034
|
+
operation: `PUT ?tagging` overwrote the object with the XML body,
|
|
1035
|
+
`DELETE ?lifecycle` deleted an empty bucket, `DELETE ?versionId=โฆ` deleted the
|
|
1036
|
+
current object, and `GET ?versioning` answered with an object listing.
|
|
1037
|
+
|
|
1038
|
+
| Operation | Request | Query names besides the selector |
|
|
1039
|
+
|---|---|---|
|
|
1040
|
+
| ListBuckets | `GET /` | none |
|
|
1041
|
+
| CreateBucket, DeleteBucket, HeadBucket | `PUT`, `DELETE`, `HEAD /{bucket}` | none |
|
|
1042
|
+
| ListObjects | `GET /{bucket}` | `prefix`, `delimiter`, `max-keys`, `marker`, `encoding-type` |
|
|
1043
|
+
| ListObjectsV2 | `GET /{bucket}?list-type=2` | `prefix`, `delimiter`, `max-keys`, `continuation-token`, `start-after`, `fetch-owner`, `encoding-type` |
|
|
1044
|
+
| ListObjectVersions | `GET /{bucket}?versions` | `prefix`, `delimiter`, `max-keys`, `key-marker`, `version-id-marker`, `encoding-type` |
|
|
1045
|
+
| ListMultipartUploads | `GET /{bucket}?uploads` | `prefix`, `encoding-type` |
|
|
1046
|
+
| GetBucketLocation | `GET /{bucket}?location` | none |
|
|
1047
|
+
| GetBucketVersioning | `GET /{bucket}?versioning` | none |
|
|
1048
|
+
| GetBucketLifecycleConfiguration | `GET /{bucket}?lifecycle` | none |
|
|
1049
|
+
| GetBucketPolicy, PutBucketPolicy, DeleteBucketPolicy | `GET`, `PUT`, `DELETE /{bucket}?policy` | none |
|
|
1050
|
+
| DeleteObjects | `POST /{bucket}?delete` | none |
|
|
1051
|
+
| GetObject, HeadObject | `GET`, `HEAD /{bucket}/{key}` | `versionId`, `response-cache-control`, `response-content-disposition`, `response-content-encoding`, `response-content-language`, `response-content-type`, `response-expires` |
|
|
1052
|
+
| DeleteObject | `DELETE /{bucket}/{key}` | `versionId` |
|
|
1053
|
+
| PutObject, CopyObject | `PUT /{bucket}/{key}`, CopyObject with `x-amz-copy-source` | none |
|
|
1054
|
+
| CreateMultipartUpload | `POST /{bucket}/{key}?uploads` | none |
|
|
1055
|
+
| UploadPart, UploadPartCopy | `PUT /{bucket}/{key}?partNumber=โฆ&uploadId=โฆ` | none |
|
|
1056
|
+
| CompleteMultipartUpload | `POST /{bucket}/{key}?uploadId=โฆ` | none |
|
|
1057
|
+
| AbortMultipartUpload | `DELETE /{bucket}/{key}?uploadId=โฆ` | none |
|
|
1058
|
+
| ListParts | `GET /{bucket}/{key}?uploadId=โฆ` | `max-parts`, `part-number-marker` |
|
|
1059
|
+
|
|
1060
|
+
Every operation also accepts the `x-id` label that AWS SDKs append and the SigV4
|
|
1061
|
+
query-signing parameters of a presigned URL (`X-Amz-Algorithm`,
|
|
1062
|
+
`X-Amz-Credential`, `X-Amz-Date`, `X-Amz-Expires`, `X-Amz-SignedHeaders`,
|
|
1063
|
+
`X-Amz-Signature`, `X-Amz-Security-Token`, `X-Amz-Content-Sha256`).
|
|
1064
|
+
Query-string signatures are not verified: a presigned request is served as an
|
|
1065
|
+
unsigned one, which is anonymous when auth is enabled.
|
|
1066
|
+
|
|
1067
|
+
An object GET or HEAD (GetObject, HeadObject, ListParts) ignores query names S3
|
|
1068
|
+
does not define, as S3 does, so links with cache-busters (`?v=1`) or tracking
|
|
1069
|
+
parameters (`?fbclid=โฆ`) keep working. A name S3 defines that the operation
|
|
1070
|
+
does not implement (`?acl`, `?partNumber=1`, โฆ) is still refused there, and
|
|
1071
|
+
every other request (bucket level, every write and delete) refuses unknown
|
|
1072
|
+
names too. The set of names S3 defines is taken from the AWS SDK v3 model
|
|
1073
|
+
(3.1039.0). With CORS enabled, the 501 and 405 responses carry the CORS headers.
|
|
1074
|
+
|
|
1075
|
+
The `response-*` parameters of GetObject and HeadObject replace the
|
|
1076
|
+
`Cache-Control`, `Content-Disposition`, `Content-Encoding`, `Content-Language`,
|
|
1077
|
+
`Content-Type` and `Expires` headers of a 200 or 206 response with the given
|
|
1078
|
+
value, verbatim. A value that is not a valid HTTP header value (a control
|
|
1079
|
+
character such as CR or LF) returns `InvalidArgument` (400). With auth enabled, an anonymous
|
|
1080
|
+
request that carries one returns `InvalidRequest` (400), as S3 does, because an
|
|
1081
|
+
override such as `response-content-type=text/html` would let a public object
|
|
1082
|
+
serve script from the storage origin; a presigned URL is such a request. With
|
|
1083
|
+
auth disabled the overrides apply.
|
|
1084
|
+
|
|
1085
|
+
Versioning is never enabled on a bucket:
|
|
1086
|
+
- `GET ?versioning` returns the empty `<VersioningConfiguration/>` of a bucket
|
|
1087
|
+
that was never versioned. `PUT ?versioning` is `NotImplemented`.
|
|
1088
|
+
- `GET ?lifecycle` returns `NoSuchLifecycleConfiguration` (404). `PUT` and
|
|
1089
|
+
`DELETE ?lifecycle` are `NotImplemented`.
|
|
1090
|
+
- `GET ?versions` lists every current object as its S3 null version
|
|
1091
|
+
(`VersionId` `null`, `IsLatest` true), with `CommonPrefixes` and paging on
|
|
1092
|
+
`key-marker` and `version-id-marker`. A truncated page returns the last key
|
|
1093
|
+
or common prefix as `NextKeyMarker`, plus `NextVersionIdMarker` `null` after a
|
|
1094
|
+
key. A `version-id-marker` needs a `key-marker`.
|
|
1095
|
+
- `versionId` on GET, HEAD and DELETE, and a `?versionId=` after the copy source
|
|
1096
|
+
of CopyObject and UploadPartCopy: `null` is the current object. A
|
|
1097
|
+
well-formed id (a lowercase UUID) names no version and returns
|
|
1098
|
+
`NoSuchVersion` (404; `NoSuchBucket` when the bucket is missing). Anything
|
|
1099
|
+
else, the empty string included, returns `InvalidArgument` (400). A `?`
|
|
1100
|
+
inside a copy-source key must be encoded as `%3F`.
|
|
1101
|
+
- These reads are authorized as `s3:GetBucketVersioning`,
|
|
1102
|
+
`s3:GetLifecycleConfiguration` and `s3:ListBucketVersions`. A `versionId`,
|
|
1103
|
+
`null` included, is authorized as `s3:GetObjectVersion` (GET, HEAD, and a
|
|
1104
|
+
copy source's `?versionId=`) or `s3:DeleteObjectVersion` (DELETE), not as
|
|
1105
|
+
`s3:GetObject` or `s3:DeleteObject`, so a policy that allows DeleteObject and
|
|
1106
|
+
denies DeleteObjectVersion keeps protecting history once buckets are
|
|
1107
|
+
versioned. Bucket tenants may use both actions in their bucket.
|
|
1108
|
+
|
|
1109
|
+
`encoding-type=url` percent-encodes, on ListObjects, ListObjectsV2,
|
|
1110
|
+
ListObjectVersions and ListMultipartUploads, every UTF-8 byte outside
|
|
1111
|
+
`A-Z a-z 0-9 - . _ ~` and `/` (a space is `%20`, never `+`) in the elements S3
|
|
1112
|
+
encodes for that operation, and the response carries
|
|
1113
|
+
`<EncodingType>url</EncodingType>`:
|
|
1114
|
+
- ListObjects: `Prefix`, `Marker`, `NextMarker`, `Delimiter`, `Key` and
|
|
1115
|
+
`CommonPrefixes/Prefix`.
|
|
1116
|
+
- ListObjectsV2: `Prefix`, `Delimiter`, `StartAfter`, `Key` and
|
|
1117
|
+
`CommonPrefixes/Prefix`; `ContinuationToken` and `NextContinuationToken`
|
|
1118
|
+
are opaque tokens, the same with or without `encoding-type`.
|
|
1119
|
+
- ListObjectVersions: `Prefix`, `KeyMarker`, `NextKeyMarker`, `Delimiter`,
|
|
1120
|
+
`Key` and `CommonPrefixes/Prefix`.
|
|
1121
|
+
- ListMultipartUploads: `Prefix` and `Key`.
|
|
1122
|
+
|
|
1123
|
+
Without the parameter keys are written as they are and no `EncodingType` is
|
|
1124
|
+
returned; any other value returns `InvalidArgument` (400). An object key may
|
|
1125
|
+
hold any UTF-8 character, also those XML 1.0 cannot carry (U+0001 and the other
|
|
1126
|
+
control characters except tab, line feed and carriage return): a listing that
|
|
1127
|
+
contains such a key is valid XML only with `encoding-type=url`. ListObjectsV2
|
|
1128
|
+
continuation tokens use only `A-Z a-z 0-9 - _`, so they never break the XML.
|
|
1129
|
+
|
|
1130
|
+
Accepted but not applied: owners are never returned (`fetch-owner`); responses
|
|
1131
|
+
say so by omitting `Owner`. Refused with `NotImplemented` although S3
|
|
1132
|
+
defines them: every other subresource (`?acl`, `?tagging`, `?retention`,
|
|
1133
|
+
`?legal-hold`, `?cors`, `?website`, `?object-lock`, `?attributes`, `?restore`,
|
|
1134
|
+
`?select`, โฆ), `partNumber` on GET and HEAD, the paging parameters of
|
|
1135
|
+
ListMultipartUploads (`key-marker`, `upload-id-marker`, `max-uploads`,
|
|
1136
|
+
`delimiter`), ListObjects V1 with V2 cursors, the ListBuckets parameters
|
|
1137
|
+
(`prefix`, `max-buckets`, `continuation-token`, `bucket-region`), headers
|
|
1138
|
+
hoisted into a presigned URL's query (`x-amz-checksum-crc32=โฆ`) on anything but
|
|
1139
|
+
an object read, and any `POST` that is not one of the operations above.
|
|
797
1140
|
|
|
798
1141
|
## Usage with AWS SDK v3
|
|
799
1142
|
|
|
@@ -999,7 +1342,23 @@ Client โโHTTP PUTโโโถ Node A (coordinator)
|
|
|
999
1342
|
2. **Objects are chunked** โ large objects split into fixed-size pieces (default 4 MB)
|
|
1000
1343
|
3. **Each chunk is erasure-coded** โ Reed-Solomon produces k data + m parity shards
|
|
1001
1344
|
4. **Shards are distributed** โ placed across different nodes and drives for fault isolation
|
|
1002
|
-
5. **Quorum guarantees consistency** โ writes need k+1 acks, reads need k shards
|
|
1345
|
+
5. **Quorum guarantees consistency** โ writes need k+1 acks, reads need k shards.
|
|
1346
|
+
A chunk with fewer than k+m written shards is published with the missing
|
|
1347
|
+
ones as pending shards that healing completes on their assigned drives.
|
|
1348
|
+
A node that membership marks offline is never contacted: its shards of a
|
|
1349
|
+
write are pending at once, reads take their shards from the other nodes,
|
|
1350
|
+
and below either quorum a request fails at once. Until a stopped node is
|
|
1351
|
+
marked offline (five missed heartbeat rounds), each chunk that needs it
|
|
1352
|
+
waits up to the QUIC idle timeout (10 s). Cluster QUIC connections send
|
|
1353
|
+
keep-alives every 2 s from both ends, so a slow but live peer stays
|
|
1354
|
+
connected however long a request takes; there is no whole-request
|
|
1355
|
+
deadline. A heartbeat to a silent peer therefore fails after 10 s. Each
|
|
1356
|
+
heartbeat round goes to every peer at once, so stopped nodes are marked
|
|
1357
|
+
offline after about 50 s by default, however many stop together (a whole
|
|
1358
|
+
location, for example). A node whose restart takes longer than that comes
|
|
1359
|
+
back to pending shards and healing work. `heartbeatTimeoutMs` (default 30 s) still bounds how long a live
|
|
1360
|
+
peer may take to answer a heartbeat, but cannot extend the wait for a
|
|
1361
|
+
silent one beyond 10 s.
|
|
1003
1362
|
|
|
1004
1363
|
### Cluster Root and Upgrade Requirements
|
|
1005
1364
|
|
|
@@ -1007,7 +1366,8 @@ Cluster mode never creates configured control or drive roots. Before startup,
|
|
|
1007
1366
|
create `storage.directory` and every `cluster.drives.paths` entry, make them
|
|
1008
1367
|
owned by the ObjectStorage process user, and remove group/world write permission.
|
|
1009
1368
|
Every configured root must be a real directory with no symlink path component,
|
|
1010
|
-
and every drive root must resolve to a distinct device/inode identity
|
|
1369
|
+
and every drive root must resolve to a distinct device/inode identity and,
|
|
1370
|
+
unless `cluster.drives.allowSharedDevice` is set, to a distinct device. The
|
|
1011
1371
|
control root may be distinct or may be the same identity as one drive root; it
|
|
1012
1372
|
cannot make two drive entries aliases of one another. Every root must support
|
|
1013
1373
|
the filesystem durability and locking semantics validated at startup.
|
|
@@ -1027,6 +1387,17 @@ has written or migrated the new state, do not restart an older binary and do not
|
|
|
1027
1387
|
run a mixed-version cluster. Back up the cluster roots before upgrading; rollback
|
|
1028
1388
|
requires restoring those roots from the pre-upgrade backup.
|
|
1029
1389
|
|
|
1390
|
+
Manifests with pending shards (written while a drive refused writes, see
|
|
1391
|
+
[Self-Healing](#self-healing)) are refused by engines that predate pending
|
|
1392
|
+
shards, because their chunks are short of placements. Before rolling a cluster
|
|
1393
|
+
back to such an engine, stop writing, complete or abort every open multipart
|
|
1394
|
+
upload (part descriptors record pending shards too, the scan does not count
|
|
1395
|
+
them, and an older engine drops them, so such an upload could not complete
|
|
1396
|
+
after the rollback), and then, on every node, run
|
|
1397
|
+
`runHealingScan()` until it reports `shardsPending` 0. Its scan starts after
|
|
1398
|
+
the request arrives, so it covers every write that finished before the call; a
|
|
1399
|
+
scan that could not list every manifest fails instead of reporting counts.
|
|
1400
|
+
|
|
1030
1401
|
### Erasure Coding
|
|
1031
1402
|
|
|
1032
1403
|
With the default `4+2` configuration:
|
|
@@ -1040,14 +1411,61 @@ With the default `4+2` configuration:
|
|
|
1040
1411
|
| 6+3 | 9 | 50% | 3 failures | 5 |
|
|
1041
1412
|
| 2+1 | 3 | 50% | 1 failure | 2 |
|
|
1042
1413
|
|
|
1414
|
+
### Streaming Reads
|
|
1415
|
+
|
|
1416
|
+
A cluster GET streams the object chunk by chunk; nothing is written to the
|
|
1417
|
+
control root. It reads from the generation of the manifest it loaded at its
|
|
1418
|
+
start, without holding the object's operation gate, so overwrites, deletes and
|
|
1419
|
+
other GETs of the same key run while it streams. Each chunk is rebuilt from the
|
|
1420
|
+
first k valid shards, read in parallel, and at most one rebuilt chunk waits
|
|
1421
|
+
ahead of the client.
|
|
1422
|
+
|
|
1423
|
+
- **Errors after the headers abort the response.** The first chunk is rebuilt
|
|
1424
|
+
before the status line, so an unreadable object still gets an error status.
|
|
1425
|
+
A failure in a later chunk ends the body with an error (the connection is
|
|
1426
|
+
closed), never with a clean end of body. A whole-object GET sends its last
|
|
1427
|
+
bytes only after the total length and the object MD5 match the manifest;
|
|
1428
|
+
range GETs are checked for length only.
|
|
1429
|
+
- **One deadline for the whole GET.** `cluster.getDeadlineMs` is the longest a
|
|
1430
|
+
single GET may run, counted from its start and including the client's
|
|
1431
|
+
transfer time; a GET that takes longer is aborted, and if its client stopped
|
|
1432
|
+
reading, its connection is closed. The default is 6 hours,
|
|
1433
|
+
the allowed range 1 minute to 7 days. The cleanup grace is derived from it:
|
|
1434
|
+
superseded and deleted generations keep their shards for the deadline plus
|
|
1435
|
+
the smaller of the deadline and one hour (7 hours by default, 2 minutes at
|
|
1436
|
+
the 1-minute floor), so every GET that started on a generation has ended
|
|
1437
|
+
before cleanup frees it. Writes in progress hold their object's gate and are
|
|
1438
|
+
never cleaned up, however short the grace. The setting must be the same on
|
|
1439
|
+
every node; a mismatch can abort streams, but never serves wrong bytes.
|
|
1440
|
+
CopyObject and CompleteMultipartUpload read their sources under the object
|
|
1441
|
+
gate with their own fixed 6-hour limit, counted from the start of the
|
|
1442
|
+
operation.
|
|
1443
|
+
- **GET memory budget.** Rebuilding chunks is charged to an engine-wide budget
|
|
1444
|
+
of `max(1 GiB, 16 ร chunk footprint)` at the configured chunk size. A chunk's
|
|
1445
|
+
footprint is โchunk/kโ ร (k + min(k, m)) + chunk: the k fetched shards, the
|
|
1446
|
+
data shards rebuilt for missing ones (a read rebuilds no parity shards) and
|
|
1447
|
+
the decoded chunk; 4+2 at 4 MiB is 10 MiB, 1+15 at 64 MiB is 192 MiB. A GET
|
|
1448
|
+
whose client went away keeps its charge until its local shard reads have
|
|
1449
|
+
ended, and shutdown waits for them before it releases the drive roots. The
|
|
1450
|
+
budget is separate from the upload budget. The first chunk waits at most 5 seconds for
|
|
1451
|
+
budget and is otherwise refused with `503 SlowDown` and `Retry-After: 1`;
|
|
1452
|
+
later chunks wait. Slow downloaders hold up to two rebuilt chunks each, so
|
|
1453
|
+
many of them can exhaust the budget and turn new GETs into `503`s under load.
|
|
1454
|
+
|
|
1043
1455
|
### QUIC Transport
|
|
1044
1456
|
|
|
1045
1457
|
Inter-node communication uses [QUIC](https://en.wikipedia.org/wiki/QUIC) via the `quinn` library:
|
|
1046
|
-
-
|
|
1458
|
+
- **Encrypted, not authenticated** โ each node generates a self-signed certificate at startup and accepts any peer; run it only on a trusted network (see [Cluster transport trust](#cluster-transport-trust))
|
|
1047
1459
|
- ๐ **Multiplexed streams** โ concurrent shard transfers without head-of-line blocking
|
|
1048
1460
|
- โก **Connection pooling** โ persistent connections to peer nodes
|
|
1049
1461
|
- ๐ **Natural backpressure** โ QUIC flow control prevents overloading slow peers
|
|
1050
1462
|
|
|
1463
|
+
### Cluster transport trust
|
|
1464
|
+
|
|
1465
|
+
The cluster transport does not authenticate nodes. Each node generates a self-signed certificate at startup and accepts any peer's certificate without verifying it. QUIC encrypts the traffic against passive eavesdropping, but not against an attacker on the path. Any host that can reach a node's `quicPort` can join the cluster, sync topology, and read, write, tombstone and purge shards. S3 credentials and bucket policies do not apply to this port.
|
|
1466
|
+
|
|
1467
|
+
Run cluster mode only on a network where every host that can reach `quicPort` (UDP, default 4000) is a cluster node: a private network or VPN between the nodes, with `quicPort` firewalled from everything else. This has been the case since cluster mode was introduced, and 11.0.0 does not change it. Mutual TLS with a cluster CA and certificate-checked joins are planned.
|
|
1468
|
+
|
|
1051
1469
|
### Cluster Membership
|
|
1052
1470
|
|
|
1053
1471
|
- **Static seed nodes** โ initial cluster defined in config
|
|
@@ -1057,13 +1475,40 @@ Inter-node communication uses [QUIC](https://en.wikipedia.org/wiki/QUIC) via the
|
|
|
1057
1475
|
|
|
1058
1476
|
### Self-Healing
|
|
1059
1477
|
|
|
1060
|
-
|
|
1061
|
-
|
|
1062
|
-
|
|
1063
|
-
|
|
1064
|
-
|
|
1065
|
-
|
|
1066
|
-
|
|
1478
|
+
Every 24 hours, and whenever `runHealingScan()` asks for it, each node scans
|
|
1479
|
+
the manifests it coordinated (manifests live on the node that wrote them):
|
|
1480
|
+
1. Every shard placement is checked at the drive it names, locally or through
|
|
1481
|
+
its node, against the checksum and size the manifest recorded. The check
|
|
1482
|
+
reads the shard record's header and the record's length only, never its
|
|
1483
|
+
payload, so a scan costs a few small reads per shard. Verifying payloads
|
|
1484
|
+
is not the scan's job: a read detects a rotted payload through its CRC32C
|
|
1485
|
+
and rebuilds the chunk from the other shards.
|
|
1486
|
+
2. A missing shard is rebuilt from k valid shards with Reed-Solomon, checked
|
|
1487
|
+
against the recorded checksum, and written to the drive its placement
|
|
1488
|
+
names. The manifest does not change.
|
|
1489
|
+
3. A pending shard is written to the drive its manifest recorded and then
|
|
1490
|
+
moved into the chunk's placements, in the same version and generation; the
|
|
1491
|
+
completed manifest is identical to one whose write never failed.
|
|
1492
|
+
|
|
1493
|
+
Healing never places a shard on a drive of its own choosing. Placements and
|
|
1494
|
+
pending shards on an offline node wait for that node; a pending shard whose
|
|
1495
|
+
drive refuses writes stays pending and is reported in `repairs.shardsPending`
|
|
1496
|
+
(replacing a permanently lost drive is not supported yet). A shard record that
|
|
1497
|
+
cannot be read, such as a truncated one, is reported in `repairs.failed` and
|
|
1498
|
+
left in place, because shard records are immutable; a node that cannot read a
|
|
1499
|
+
record answers with an error, never as if the shard were missing. A minority
|
|
1500
|
+
partition does not heal: its scan fails. Placements are checked without the
|
|
1501
|
+
object's operation gate, as a GET reads them. The chunks of an object that need
|
|
1502
|
+
a write are healed in batches of up to 1 GiB of chunk data: each batch takes
|
|
1503
|
+
the object's gate, reloads the manifest, heals only chunks that are unchanged
|
|
1504
|
+
in the same generation, publishes its pending shards with one manifest store
|
|
1505
|
+
and releases the gate before the next batch. An object whose gate is busy is
|
|
1506
|
+
skipped until the next scan.
|
|
1507
|
+
|
|
1508
|
+
Scans never overlap. A scan that outlasts the 24-hour interval delays the next
|
|
1509
|
+
one. `runHealingScan()` joins the next scan that starts after its request and
|
|
1510
|
+
never a scan that is already running; requests that arrive together share one
|
|
1511
|
+
scan.
|
|
1067
1512
|
|
|
1068
1513
|
### Erasure Set Formation
|
|
1069
1514
|
|
|
@@ -1083,7 +1528,7 @@ Drives are interleaved across nodes for maximum fault isolation. New nodes form
|
|
|
1083
1528
|
|
|
1084
1529
|
```typescript
|
|
1085
1530
|
import { ObjectStorage } from '@lossless.org/objectstorage';
|
|
1086
|
-
import { tap, expect } from '@git.zone/
|
|
1531
|
+
import { tap, expect } from '@git.zone/tsrepo/tapbundle';
|
|
1087
1532
|
|
|
1088
1533
|
let storage: ObjectStorage;
|
|
1089
1534
|
|
|
@@ -1266,6 +1711,24 @@ List scoped tenant credential metadata without returning secrets.
|
|
|
1266
1711
|
|
|
1267
1712
|
Return endpoint, port, region, bucket, access key, secret key, SSL flag, legacy descriptor fields, and S3/AWS env values for the bucket tenant.
|
|
1268
1713
|
|
|
1714
|
+
#### `createPrefixCredential(input: IPrefixCredentialInput): Promise<IPrefixCredential>`
|
|
1715
|
+
|
|
1716
|
+
Create an expiring credential limited to one prefix of one bucket, or install
|
|
1717
|
+
one created on another node (`accessKeyId`, `secretAccessKey`, `expiresAt`). An
|
|
1718
|
+
identical replay returns the stored record. The secret is returned only here.
|
|
1719
|
+
See [Prefix Credentials](#prefix-credentials) for the input rules and the
|
|
1720
|
+
`ObjectStorageCredentialError` codes.
|
|
1721
|
+
|
|
1722
|
+
#### `revokePrefixCredential(options): Promise<{ revoked: boolean }>`
|
|
1723
|
+
|
|
1724
|
+
Revoke a prefix credential and keep its ID refused until the latest moment it
|
|
1725
|
+
could still be created. Options: `{ accessKeyId }`. `revoked` is `true` only
|
|
1726
|
+
when a live credential was removed; unknown IDs are recorded as revoked too.
|
|
1727
|
+
|
|
1728
|
+
#### `listPrefixCredentials(options?): Promise<IPrefixCredentialMetadata[]>`
|
|
1729
|
+
|
|
1730
|
+
List live prefix credentials without secrets. Options: `{ bucketName? }`.
|
|
1731
|
+
|
|
1269
1732
|
#### `getBucketMigrationCapability(): Promise<IObjectStorageBucketMigrationCapability>`
|
|
1270
1733
|
|
|
1271
1734
|
Return standalone provider identity and receipt-key evidence plus source seal,
|
|
@@ -1324,7 +1787,9 @@ the bucket's exclusive credential. Options:
|
|
|
1324
1787
|
|
|
1325
1788
|
Delete a fenced bucket. Supplying `accessKeyId` also durably revokes that
|
|
1326
1789
|
exact tenant credential and deletes the policy; omitting it preserves the
|
|
1327
|
-
legacy credential/policy behavior.
|
|
1790
|
+
legacy credential/policy behavior. The bucket's prefix credentials are revoked
|
|
1791
|
+
first in both cases, and the deletion is refused if that revocation cannot be
|
|
1792
|
+
stored. Options:
|
|
1328
1793
|
`{ bucketName, accessKeyId?, fence, holdPublication? }`. Setting
|
|
1329
1794
|
`holdPublication: true` returns a durable `resourcePublicationBarrier`; omitted
|
|
1330
1795
|
or `false` preserves immediate behavior.
|
|
@@ -1390,6 +1855,21 @@ address. With `cluster.quicPort: 0`, the operating system assigns its port; use
|
|
|
1390
1855
|
Membership advertisements and peer `quicAddress` values contain that same bound
|
|
1391
1856
|
address, including after a restart.
|
|
1392
1857
|
|
|
1858
|
+
#### `runHealingScan(options?: IRunHealingScanOptions): Promise<IClusterHealingScanResult>`
|
|
1859
|
+
|
|
1860
|
+
Run a healing scan on this cluster node (see [Self-Healing](#self-healing)) and
|
|
1861
|
+
return its `startedAt`, `completedAt`, `durationMs`, `shardsChecked`,
|
|
1862
|
+
`shardsHealed`, `shardsPending` and `failed`. The request joins the next scan
|
|
1863
|
+
that starts after it arrives; a scan already running finishes first. It
|
|
1864
|
+
rejects in standalone mode, when the scan fails (without a majority of the
|
|
1865
|
+
cluster, or when a bucket's manifests cannot be listed) and when the engine
|
|
1866
|
+
shuts down before the scan completes, and it is refused while 16 requests
|
|
1867
|
+
already wait for a scan to start. The engine answers other management calls
|
|
1868
|
+
while the scan runs. `options.timeoutMs` (default 300,000) bounds the wait;
|
|
1869
|
+
the engine cannot cancel a scan, so one that outlasts it keeps running, its
|
|
1870
|
+
result is discarded, and a later call joins the next scan that starts after
|
|
1871
|
+
it.
|
|
1872
|
+
|
|
1393
1873
|
#### `getHealth(): Promise<IObjectStorageHealth>`
|
|
1394
1874
|
|
|
1395
1875
|
Return running state, storage directory, per-location pool observations, auth
|
|
@@ -1416,7 +1896,7 @@ objectstorage uses a **hybrid Rust + TypeScript** architecture:
|
|
|
1416
1896
|
โ โ โโ FileStore (single-node mode) โ
|
|
1417
1897
|
โ โ โโ DistributedStore (cluster mode) โ
|
|
1418
1898
|
โ โ โโ ErasureCoder (Reed-Solomon) โ
|
|
1419
|
-
โ โ โโ
|
|
1899
|
+
โ โ โโ ShardBackend (per-drive shards) โ
|
|
1420
1900
|
โ โ โโ QuicTransport (quinn) โ
|
|
1421
1901
|
โ โ โโ ClusterState & Membership โ
|
|
1422
1902
|
โ โ โโ HealingService โ
|
|
@@ -1433,7 +1913,7 @@ objectstorage uses a **hybrid Rust + TypeScript** architecture:
|
|
|
1433
1913
|
|
|
1434
1914
|
**Why Rust?** The original TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves these with bounded-memory streaming I/O, backpressure, and direct `seek()` for range requests.
|
|
1435
1915
|
|
|
1436
|
-
**IPC Protocol:** TypeScript communicates with the `ruststorage` binary over newline-delimited JSON via stdin/stdout. The current management commands are `probeStoragePool`, `findOrphanObjectLeaves`, `getStoragePoolDiagnostic`, `start`, `stop`, `createBucket`, `createBucketTenant`, `ensureBucketTenant`, `deleteBucketTenant`, `rotateBucketTenantCredentials`, `listBucketTenants`, `getBucketTenantCredential`, `getBucketRetentionCapability`, `getBucketRetentionReceipt`, `getBucketMigrationCapability`, `createSourceMigration`, `createDestinationMigration`, `transitionSourceMigration`, `transitionDestinationMigration`, `cleanupSourceMigration`, `inspectBucketMigration`, `exportBucket`, `importBucket`, `deleteBucket`, `commitResourcePublication`, `getResourceFencingCapability`, `getBucketResourceFenceState`, `getStorageStats`, `listBucketSummaries`, `listCredentials`, `replaceCredentials`, and `
|
|
1916
|
+
**IPC Protocol:** TypeScript communicates with the `ruststorage` binary over newline-delimited JSON via stdin/stdout. The current management commands are `probeStoragePool`, `findOrphanObjectLeaves`, `getStoragePoolDiagnostic`, `start`, `stop`, `createBucket`, `createBucketTenant`, `ensureBucketTenant`, `deleteBucketTenant`, `rotateBucketTenantCredentials`, `listBucketTenants`, `getBucketTenantCredential`, `getBucketRetentionCapability`, `getBucketRetentionReceipt`, `getBucketMigrationCapability`, `createSourceMigration`, `createDestinationMigration`, `transitionSourceMigration`, `transitionDestinationMigration`, `cleanupSourceMigration`, `inspectBucketMigration`, `exportBucket`, `importBucket`, `deleteBucket`, `commitResourcePublication`, `getResourceFencingCapability`, `getBucketResourceFenceState`, `getStorageStats`, `listBucketSummaries`, `listCredentials`, `replaceCredentials`, `createPrefixCredential`, `revokePrefixCredential`, `listPrefixCredentials`, `getPrefixCredentialStatus`, `getClusterHealth`, and `runHealingScan`. `getStoragePoolDiagnostic` reads a weak, cached diagnostic source and remains responsive while another management command owns the server; it does not retain the mounted root after server ownership completes. `runHealingScan` answers when its scan completes, with the request's `id`; the engine serves other commands meanwhile, and a shutdown answers a pending request with an error before the `stop` response.
|
|
1437
1917
|
|
|
1438
1918
|
### S3-Compatible Operations
|
|
1439
1919
|
|
|
@@ -1483,7 +1963,8 @@ those versions cannot read the canonical v2 objects.
|
|
|
1483
1963
|
| ListParts | `GET /{bucket}/{key}?uploadId` | Optional `max-parts` and `part-number-marker` |
|
|
1484
1964
|
| CompleteMultipartUpload | `POST /{bucket}/{key}?uploadId` | |
|
|
1485
1965
|
| AbortMultipartUpload | `DELETE /{bucket}/{key}?uploadId` | |
|
|
1486
|
-
| ListMultipartUploads | `GET /{bucket}?uploads` | |
|
|
1966
|
+
| ListMultipartUploads | `GET /{bucket}?uploads` | Optional `prefix` |
|
|
1967
|
+
| GetBucketLocation | `GET /{bucket}?location` | Returns `server.region`; an empty constraint for `us-east-1` |
|
|
1487
1968
|
| GetBucketPolicy | `GET /{bucket}?policy` | |
|
|
1488
1969
|
| PutBucketPolicy | `PUT /{bucket}?policy` | |
|
|
1489
1970
|
| DeleteBucketPolicy | `DELETE /{bucket}?policy` | |
|
|
@@ -1507,6 +1988,14 @@ response echoes `ContinuationToken` and `StartAfter` when the request carried
|
|
|
1507
1988
|
them, reports `KeyCount` as contents plus common prefixes, and returns
|
|
1508
1989
|
`NextContinuationToken` whenever the page is truncated.
|
|
1509
1990
|
|
|
1991
|
+
A continuation token is opaque: pass `NextContinuationToken` back unchanged.
|
|
1992
|
+
It is the unpadded base64url (RFC 4648 ยง5) encoding of the UTF-8 key the next
|
|
1993
|
+
page resumes after, at most 1,366 characters, and is the same with or without
|
|
1994
|
+
`encoding-type=url`, so a client may change `encoding-type` between pages. A
|
|
1995
|
+
token the server could not have issued (outside the base64url alphabet, padded,
|
|
1996
|
+
not canonical, or not decoding to a 1 to 1,024-byte UTF-8 key) returns
|
|
1997
|
+
`InvalidArgument` (400); so do tokens issued before 11.0.0, which were raw keys.
|
|
1998
|
+
|
|
1510
1999
|
**ListObjects v1** (no `list-type`) reads `prefix`, `delimiter`, `max-keys` and
|
|
1511
2000
|
`marker`, and ignores the V2 cursors. The response always carries `Marker`,
|
|
1512
2001
|
empty when the request carried none, and carries `NextMarker` whenever the page
|
|
@@ -1523,14 +2012,16 @@ Paging rules shared by both versions:
|
|
|
1523
2012
|
cursor.
|
|
1524
2013
|
- `max-keys` above 1000 is clamped to 1000 and the response echoes the clamped
|
|
1525
2014
|
value in `MaxKeys`, as S3 does. A `max-keys` value that is not a non-negative
|
|
1526
|
-
integer is rejected with `InvalidRequest`, and so is a
|
|
1527
|
-
1,024-byte S3 key bound.
|
|
2015
|
+
integer is rejected with `InvalidRequest`, and so is a `start-after` or
|
|
2016
|
+
`marker` longer than the 1,024-byte S3 key bound.
|
|
1528
2017
|
- An empty `start-after`, `continuation-token` or `marker` value means no cursor.
|
|
1529
2018
|
- Keys are returned in ascending UTF-8 byte order, and a key is never returned
|
|
1530
2019
|
twice across pages of an unchanged bucket.
|
|
1531
2020
|
|
|
1532
2021
|
ListMultipartUploads is not paginated: it returns every in-flight upload of the
|
|
1533
|
-
bucket with `IsTruncated` false.
|
|
2022
|
+
bucket with `IsTruncated` false. An optional `prefix` parameter limits the
|
|
2023
|
+
result to uploads whose keys start with it and is echoed in the response;
|
|
2024
|
+
prefix credentials must supply one inside their prefix.
|
|
1534
2025
|
|
|
1535
2026
|
### On-Disk Format
|
|
1536
2027
|
|
|
@@ -1552,6 +2043,8 @@ bucket with `IsTruncated` false.
|
|
|
1552
2043
|
{bucket}.policy.json # Bucket policy (IAM JSON format)
|
|
1553
2044
|
.smartstorage/staging/v1/ # Private staging: a write lands here first; its
|
|
1554
2045
|
# key leaf appears only with the complete payload
|
|
2046
|
+
.smartstorage/credentials.json # Runtime and bucket tenant credentials (0600)
|
|
2047
|
+
.smartstorage/prefix-credentials.json # Prefix credentials and revocations (0600)
|
|
1555
2048
|
```
|
|
1556
2049
|
|
|
1557
2050
|
Rolling a standalone root back from 10.7.0 or later to 8.0.3โ10.6.0: a crash
|
|
@@ -1575,7 +2068,8 @@ were never published.
|
|
|
1575
2068
|
{object-address}.state.json
|
|
1576
2069
|
# Active/tombstoned object manifest
|
|
1577
2070
|
.multipart/{upload-id}/ # Descriptor-anchored multipart state
|
|
1578
|
-
|
|
2071
|
+
.smartstorage/credentials.json # This node's runtime credentials (not replicated)
|
|
2072
|
+
.smartstorage/prefix-credentials.json # This node's prefix credentials (not replicated)
|
|
1579
2073
|
.smartstorage/cluster/
|
|
1580
2074
|
identity.json # Node and cluster identity
|
|
1581
2075
|
topology.json # Last durable topology
|
|
@@ -1588,8 +2082,12 @@ were never published.
|
|
|
1588
2082
|
```
|
|
1589
2083
|
|
|
1590
2084
|
Superseded physical generations are queued with their exact shard placements.
|
|
1591
|
-
|
|
1592
|
-
|
|
2085
|
+
They keep their shards for the cleanup grace derived from `cluster.getDeadlineMs`
|
|
2086
|
+
(see Streaming Reads), and the cleanup worker, which runs once per grace period
|
|
2087
|
+
and at least hourly, then purges both shard records and tombstones. Earlier
|
|
2088
|
+
engines spooled GET responses into `.manifests/.get-tmp/`; startup removes a
|
|
2089
|
+
leftover spool directory once it holds only spool files, and otherwise leaves
|
|
2090
|
+
it in place, unused. Multipart
|
|
1593
2091
|
uploads older than `multipart.expirationDays`, plus committed uploads whose
|
|
1594
2092
|
first cleanup attempt failed, are retried during bounded cleanup passes
|
|
1595
2093
|
scheduled every `multipart.cleanupIntervalMinutes`; large registries are
|
|
@@ -1612,7 +2110,7 @@ established.
|
|
|
1612
2110
|
|
|
1613
2111
|
- [`@push.rocks/smartbucket`](https://code.foss.global/push.rocks/smartbucket) โ High-level S3-compatible abstraction layer
|
|
1614
2112
|
- [`@push.rocks/smartrust`](https://code.foss.global/push.rocks/smartrust) โ TypeScript โ Rust IPC bridge
|
|
1615
|
-
- [`@git.zone/
|
|
2113
|
+
- [`@git.zone/rsrepo`](https://code.foss.global/git.zone/rsrepo) โ Rust builds and cross-compilation for npm packages
|
|
1616
2114
|
|
|
1617
2115
|
## License and Legal Information
|
|
1618
2116
|
|