@elaraai/e3-cloud-files 1.1.0 → 1.1.2

package/README.md

@@ -18,9 +18,11 @@ changed, the same way on a local repository and on a deployed workspace.
   first state, and in its history the seed names the file. A record's
   mutation may be fed by files: each new file is the mutation's one
   argument, applied once.
-- **Push from disk.** `e3-cloud files push` puts files from your machine
-  into a source and takes them in, with your e3 login alone where the source
-  is the deployment's own bucket.
+- **Push from disk, or from a workspace.** `e3-cloud files push` puts files
+  from your machine into a source and takes them in, with your e3 login
+  alone where the source is the deployment's own bucket. A value an e3
+  workspace computes (`dataset:dev.customer_book`) pushes as the file of its
+  value would, with no file kept by hand, and the history names the dataset.
 - **Nothing polls.** A sync runs when someone asks for it, and reads only the
   files that changed since the last one.
 
@@ -154,14 +156,18 @@ export const customerCount = e3.task('customer_count', [customers], …);
 - **In its history** the seed reads as any commit, through the `files`
   channel: "Initialised from customers.beast2 (12,004 rows)", "Seeded again
   from customers.beast2 (12,100 rows)", on the record's page and in
-  `e3-cloud records history`. A rollback to an earlier seed brings its state
-  back, as to any commit.
+  `e3-cloud records history`; a file pushed from a dataset names it too,
+  with its value's hash ("…, from dev.customer_book at 3f9a2b1c4d5e"). A
+  rollback to an earlier seed brings its state back, as to any commit.
 - **A deploy that cannot keep the record** (its type changed) under
   `--schema reset` unassigns it: pending again, seeded by the next version's
   file once the new declarations are deployed.
 - The file is taken in as a typed input is (by pieces, resumable), then
   committed as a system commit with the file's key, ETag and SHA-256 as its
-  arguments, under one hold of the repository's running work.
+  arguments (and, for a file pushed from a dataset, the dataset's path and
+  its value's hash after them), under one hold of the repository's running
+  work. The report numbers each seed among the record's: `$init #1`, then
+  `$seed #2`, and so on.
 
 ## A record fed through a mutation
 
@@ -226,8 +232,52 @@ Next: e3 dataflow run https://e3.example.com/repos/acme dev
   what is not there. `--dry-run` plans and puts nothing.
 - The output is a row per file: the file, its target and kind, its size,
   `put`, `unchanged` or `refused`, and what the sync did (`taken in` with
-  the rows, `$init`/`$seed` or the mutation with its commit, `kept`, or the
-  refusal). The command exits 1 when any file was refused or failed.
+  the rows, `$init #1` or `$seed #n` with the rows, the mutation with its
+  commit, `kept`, or the refusal). The command exits 1 when any file was
+  refused or failed.
+
+### A workspace's dataset
+
+A value an e3 workspace computes (a task's output, an input or a record)
+pushes as the file of its value would, with no file kept by hand: name it
+`dataset:<workspace>.<name>` among the paths, in the repository `--from`
+names. That is a local repository's path or a repository's URL, and
+defaults as e3's own `[repo]` does (`$E3_REPO`, then `.`). Files,
+directories and datasets mix in one push.
+
+```
+$ e3-cloud files push https://e3.example.com/repos/acme dev dataset:dev.customers --from ./archive
+upload  s3://acme-files/acme/dev/
+  dataset:dev.customers (task output)  → record:customers  seed  3 MiB  put  $init #1  12,004 rows
+```
+
+- **Matched by its name**: a dataset goes to the `file` or `latest`
+  mapping that fills the input or the record of its name
+  (`dataset:dev.customers` → `record:customers`), else `--as` names its
+  target, any `file` or `latest` mapping's (a mutation's too). It is put at
+  a `file` mapping's key, or under a `latest` mapping's prefix as
+  `<name>.beast2` (refused `not_latest` as a file is). A `folder` or a
+  `listing` takes no dataset: a dataset is one value.
+- **Its bytes are its value's**, as `e3 dataset get -f beast2` writes it,
+  its header its declared type: read from the repository a segment at a
+  time, never held whole, and checked against the mapping's type before
+  anything is put (`type_mismatch`, naming both). They are read once to
+  measure them and again as they upload, and a dataset a run writes again in
+  between fails in its row, never put as something else: push it again.
+- **The same stored value is `unchanged`**: nothing put, nothing downstream
+  runs. A dataset changed since (a run, a value set) is put and taken in by
+  the target's rules: a record seeded again while nothing has mutated it,
+  kept once something has, seeded over it on `--reset`.
+- **Refused in its row**, the rest going on: a dataset that is unassigned (a
+  task output not produced since its workspace's last deploy, an input never
+  set, a record with no state yet), or one the workspace doesn't hold.
+- **Named where it lands**: the object is put with metadata naming the
+  dataset and its value's hash (`e3-dataset`, `e3-dataset-hash`), and a
+  seed's or a feed's commit records them, so its history reads "Initialised
+  from customers.beast2 (12,004 rows), from dev.customers at 3f9a2b1c4d5e".
+- A local repository works at either end: the push reads a local
+  repository's datasets through e3 on your machine, and a deployed one's
+  through its API with your e3 login.
 
 ## Making a beast2 file
 
@@ -236,7 +286,8 @@ A beast2 file already is an East value. The East SDK writes one from a value
 (`Beast2ElementWriter`, which the platform's own tests use for files of a
 gibibyte); east-py writes one from Python; `e3 convert` converts a file of
 another format; `e3 dataset get -f beast2` writes any dataset of a workspace
-as one. A typed mapping takes it as it stands, its header saying its type.
+as one, though a push takes the dataset itself (A workspace's dataset,
+above). A typed mapping takes it as it stands, its header saying its type.
 
 ## Decoding in a task
 
@@ -312,8 +363,8 @@ e3 dataflow run ./repo dev
   what a sync would take and takes nothing. A sync doesn't run the dataflow:
   `e3 dataflow run` does, and the sync prints the command when anything
   changed.
-- **`files push`** puts files from disk into a source and takes them in
-  (Pushing files from disk, above).
+- **`files push`** puts files from disk, and workspaces' datasets, into a
+  source and takes them in (Pushing files from disk, above).
 - **`files status`** shows each mapping with its shape, its type, the file
   it holds and the commit it made (or `pending` for a record not yet
   seeded), where each source is bound, and the last sync.
@@ -451,14 +502,20 @@ In `@elaraai/e3-cloud-types`, re-exported here:
   typed value taken (`FilesTakenType`: its rows), the commit a seed or feed
   made, and why a seed was kept or a feed waits.
 - `FilesStatusType` (each mapping with what it holds and whether its record
-  is pending), `FilesProblemType`, the push's `FilesPushRequestType` and
-  `FilesPushPlanType`.
-- Values stored by the first version (mappings by input, a refusal as one
-  text) are read by their header's type and lifted
-  (`@elaraai/e3-cloud-files/compat`): declarations, ledgers and reports a
-  workspace stored before this version read as they were, and are written
-  again in the current form on their next write. A local repository's state
-  file says its version (`version: 2`).
+  is pending), `FilesProblemType`, the push's `FilesPushRequestType` (each
+  file's `FilesPushFileType`, with the dataset it is the value of, if one)
+  and `FilesPushPlanType`.
+- `FilesSeedCommitType`: a seed's commit, `$init` or `$seed`, and its number
+  among the record's seeds, in a mapping's report.
+- `FilesDatasetOriginType`: where a pushed file came from, a dataset's path
+  and its value's hash, as the object's metadata carries it
+  (`filesDatasetMetadata`, read back by `filesDatasetOriginOf`).
+- Values stored by earlier versions (the first: mappings by input, a refusal
+  as one text; the second: a record's seeds not counted) are read by their
+  header's type and lifted (`@elaraai/e3-cloud-files/compat`):
+  declarations, ledgers and reports a workspace stored before this version
+  read as they were, and are written again in the current form on their next
+  write. A local repository's state file says its version (`version: 3`).
 - The inputs' types: `FilesFileType` (`Blob`), `FilesFolderType`
   (`Dict<String, Blob>`), `FilesListingType`.
 
@@ -471,7 +528,7 @@ In `@elaraai/e3-cloud-types`, re-exported here:
 | `@elaraai/e3-cloud-files/sync` | The sync (`syncFiles`) over what reads, writes and remembers, the push's plan (`planPush`), and the ports in memory for tests |
 | `@elaraai/e3-cloud-files/storage` | The sync's writer over an e3 store, and the cloud's over a staging (needs `@elaraai/e3-core`) |
 | `@elaraai/e3-cloud-files/s3` | The sync's reader of a bucket (needs `@aws-sdk/client-s3`) |
-| `@elaraai/e3-cloud-files/compat` | The first version's stored types, and their lift into the current ones |
+| `@elaraai/e3-cloud-files/compat` | The earlier versions' stored types, and their lift into the current ones |
 | `@elaraai/e3-cloud-files/examples/*` | The worked examples (`orders`, `typed`) and the decoding examples (`decode`) |
 
 ## License