cloesce 0.0.3-fix.6 → 0.0.4-unstable.0

package/dist/README.md

@@ -1,25 +1,23 @@
-# cloesce (experimental, `v0.0.3`)
+# cloesce (unstable, `v0.0.4`)
 
-Cloesce is a full stack compiler for the Cloudflare developer platform, allowing class definitions in high level languages to serve as the metadata basis for databases, a backend REST API, a frontend API client, and Cloudflare infrastructure.
+Cloesce is a full stack compiler for the Cloudflare developer platform, allowing class definitions in high level languages to serve as a metadata basis to create a database schema, backend REST API, frontend API client, and Cloudflare infrastructure (as of v0.0.4, D1 + Workers).
 
 Cloesce is working towards an alpha MVP (v0.1.0), with the general milestones being [here](https://cloesce.pages.dev/schreiber/v0.1.0_milestones/).
 
 Internal documentation going over design decisions and general thoughts for each milestone can be found [here](https://cloesce.pages.dev/).
 
-# Documentation `v0.0.3`
-
-Note that this version is very unstable (ie, it passes our set of happy-path tests).
+# Documentation `v0.0.4`
 
 ## Getting Started
 
-`v0.0.3` supports only Typescript-to-Typescript projects. An example project is shown [here](https://github.com/bens-schreiber/cloesce/tree/main/examples).
+`v0.0.4` supports only Typescript-to-Typescript projects. An example project is shown [here](https://github.com/bens-schreiber/cloesce/tree/main/examples).
 
 1. NPM
 
 - Create an NPM project and install cloesce
 
 ```sh
-npm i cloesce@0.0.3-fix.3
+npm i cloesce@0.0.4-unstable.0
 ```
 
 2. TypeScript
@@ -42,19 +40,19 @@ npm i cloesce@0.0.3-fix.3
 
 3. Cloesce Config
 
-- Create a `cloesce.config.json` with the following values:
+- Create a `cloesce.config.json` with the following keys:
 
 ```json
 {
-  "source": "./src", // or whatever src you want
-  "workersUrl": "http://localhost:5002/api", // or whatever url you want
-  "clientUrl": "http://localhost:5002/api"
+  "source": "./src",
+  "workersUrl": "http://localhost:5000/api",
+  "clientUrl": "http://localhost:5173/api"
 }
 ```
 
 4. Vite
 
-- `v0.0.3` does not yet support middleware, so you'll run into CORs problems. A vite proxy in some `vite.config.ts` can fix this:
+To prevent CORS issues, a Vite proxy can be used for the frontend:
 
 ```ts
 import { defineConfig } from "vite";
@@ -74,7 +72,7 @@ export default defineConfig({
 
 5. Wrangler Config
 
-- `v0.0.3` will generate the required areas of your wrangler config. A full config looks like this:
+- `v0.0.4` will generate the required areas of your wrangler config. A full config looks like this:
 
 ```toml
 compatibility_date = "2025-10-02"
@@ -89,12 +87,19 @@ database_name = "example"
 
 ## A Simple Model
 
-A model is a type which represents a database table, database view, backend api, frontend client and Cloudflare infrastructure (D1 + Workers).
+A model is a type which represents:
+
+- a database table,
+- database views
+- REST API
+- Client API
+- Cloudflare infrastructure (D1 + Workers)
 
-Suprisingly, it's pretty compact. A basic scalar model (being, without foreign keys) looks like this:
+Suprisingly, it's pretty compact. A basic model looks like this:
 
 ```ts
 // horse.cloesce.ts
+import { D1, GET, POST, PrimaryKey } from "cloesce/backend";
 
 @D1
 export class Horse {
@@ -104,7 +109,7 @@ export class Horse {
   name: string | null;
 
   @POST
-  async neigh(): Promise<string> {
+  neigh(): string {
     return `i am ${this.name}, this is my horse noise`;
   }
 }
@@ -113,25 +118,34 @@ export class Horse {
 - `@D1` denotes that this is a SQL Table
 - `@PrimaryKey` sets the SQL primary key. All models require a primary key.
 - `@POST` reveals the method as an API endpoint with the `POST` HTTP Verb.
+- All Cloesce models need to be under a `.cloesce.ts` file.
 
-After running `cloesce run`, you will get a fully generated project that can be ran with:
+To compile this model into a working full stack application, Cloesce must undergo both **compilation** and **migrations**. Compilation is the process of extracting the metadata language that powers Cloesce, ensuring it is a valid program, and then producing code to orchestrate the program across different domains (database, backend, frontend, cloud). Migrations utilize the history of validated metadata to create SQL code, translating the evolution of your models.
 
-```sh
-# migrate wrangler
-npx wrangler d1 migrations apply proj-name
+To compile, run `npx cloesce compile`.
+
+To create a migration, run `npx cloesce migrate <name>`.
 
-# build
-npx wrangler build
+After running the above commands, you will have a full project capable of being ran with:
 
-# run wrangler
-npx wrangler dev --port 5000
+```sh
+# Apply the generated migrations
+npx wrangler d1 migrations apply <db-name>
+
+# Run the backend
+npx wrangler dev
 ```
 
-Note the output in the `.generated/` dir:
+Note the output in the `.generated/` dir. These values should not be committed to git, as they depend on the file system of the machine running it.
 
 - `client.ts` is an importable API with all of your backend types and endpoints
-- `migrations/*.sql` is the generated SQL code (note, migrations beyond the initial aren't really supported yet)
 - `workers.ts` is the workers entrypoint.
+- `cidl.json` is the working metadata for the project
+
+Note the output in `migrations`, ex after running `npx cloesce migrate Initial`
+
+- `<date>_Initial.json` contains all model information necessary from SQL
+- `<date>_Initial.sql` contains the acual SQL migrations. In this early version of Cloesce, it's important to check migrations every time.
 
 ## Features
 
@@ -142,6 +156,8 @@ In order to interact with your database, you will need to define a WranglerEnv
 ```ts
 // horse.cloesce.ts
 
+import { WranglerEnv } from "cloesce/backend";
+
 @WranglerEnv
 export class Env {
   db: D1Database; // only one DB is supported for now-- make sure it matches the name in `wrangler.toml`
@@ -161,118 +177,117 @@ export class Horse {
 
   @POST
   async neigh(@Inject env: WranglerEnv): Promise<string> {
-    env.db.prepare(...);
+    await env.db.prepare(...);
 
     return `i am ${this.name}, this is my horse noise`;
   }
 }
 ```
 
-## Foreign Keys, One to One, Data Sources
+### Foreign Keys, One to One, Data Sources
 
-More complex model relationships are permitted via the `@ForeignKey`, `@OneToOne / @OneToMany @ManyToMany` and `@DataSource` decorators.
+Complex model relationships are permitted via the `@ForeignKey`, `@OneToOne / @OneToMany @ManyToMany` and `@DataSource` decorators.
 Foreign keys are scalar attributes which must reference some other model's primary key:
 
 ```ts
 @D1
-export class B {
+export class Dog {
   @PrimaryKey
   id: number;
 }
 
 @D1
-export class A {
+export class Person {
   @PrimaryKey
   id: number;
 
-  @ForeignKey(B)
-  bId: number;
+  @ForeignKey(Dog)
+  dogId: number;
 }
 ```
 
-This representation is true to the underlying SQL table: `A` has a column `bId` which is a foreign key to `B`. Cloesce allows you to actually join these tables together in your model representation:
+This representation is true to the underlying SQL table: `Person` has a column `dogId` which is a foreign key to `Dog`. Cloesce allows you to actually join these tables together in your model representation:
 
 ```ts
 @D1
-export class B {
+export class Dog {
   @PrimaryKey
   id: number;
 }
 
 @D1
-export class A {
+export class Person {
   @PrimaryKey
   id: number;
 
-  @ForeignKey(B)
-  bId: number;
+  @ForeignKey(Dog)
+  dogId: number;
 
-  @OneToOne("bId") // references A.bId
-  b: B | undefined; // This value is a "navigation property", which may or may not exist at runtime
+  @OneToOne("dogId") // references Person.dogId
+  dog: Dog | undefined; // This value is a "navigation property", which may or may not exist at runtime
 }
 ```
 
-In `v0.0.3`, there are no defaults, only very explicit decisons. Because of that, navigation properties won't exist at runtime unless you tell them to. Cloesce does this via a `DataSource`, which describes the foreign key dependencies you wish to include. All scalar properties are included by default and cannot be excluded.
+In `v0.0.4`, there are no defaults, only very explicit decisons. Because of that, navigation properties won't exist at runtime unless you tell them to. Cloesce does this via a `DataSource`, which describes the foreign key dependencies you wish to include. All scalar properties are included by default and cannot be excluded.
 
 ```ts
 @D1
-export class B {
+export class Dog {
   @PrimaryKey
   id: number;
 }
 
 @D1
-export class A {
+export class Person {
   @PrimaryKey
   id: number;
 
-  @ForeignKey(B)
-  bId: number;
+  @ForeignKey(Dog)
+  dogId: number;
 
-  @OneToOne("bId")
-  b: B | undefined;
+  @OneToOne("dogId")
+  dog: Dog | undefined;
 
   @DataSource
-  static readonly default: IncludeTree<A> = {
-    b: {}, // says: on model population, join A's B
+  static readonly default: IncludeTree<Person> = {
+    dog: {}, // says: on model population, join Persons's Dog
   };
 }
 ```
 
-Datasources are just SQL views and can be invoked in your queries. They are aliased in such a way that its identical to object properties. The frontend chooses which datasource to use in it's API client. `null` is a valid option, meaning no joins will occur.
+Data sources are just SQL views and can be invoked in your queries. They are aliased in such a way that its similiar to object properties. The frontend chooses which datasource to use in it's API client. `null` is a valid option, meaning no joins will occur.
 
 ```ts
 @D1
-export class A {
+export class Person {
   @PrimaryKey
   id: number;
 
-  @ForeignKey(B)
-  bId: number;
+  @ForeignKey(Dog)
+  dogId: number;
 
-  @OneToOne("bId")
-  b: B | undefined;
+  @OneToOne("dogId")
+  dog: Dog | undefined;
 
   @DataSource
-  static readonly default: IncludeTree<A> = {
-    b: {},
+  static readonly default: IncludeTree<Person> = {
+    dog: {},
   };
 
   @GET
-  static async get(id: number, @Inject env: WranglerEnv): Promise<A> {
+  static async get(id: number, @Inject env: WranglerEnv): Promise<Person> {
     let records = await env.db
-      .prepare("SELECT * FROM [A.default] WHERE [A.id] = ?") // A.default is the SQL view generated from the IncludeTree
+      .prepare("SELECT * FROM [Person.default] WHERE [Person.id] = ?") // Person.default is the SQL view generated from the IncludeTree
       .bind(id)
       .run();
 
-    // modelsFromSql is a provided function to turn sql rows into an object.
-    // More ORM functions will be expanded on in v0.0.4.
-    return modelsFromSql(A, records.results, A.default)[0] as A;
+    let persons = Orm.fromSql(Person, records.results, Person.default);
+    return persons.value[0];
   }
 }
 ```
 
-Note: In later versions, nearly all of the code from the example above won't be needed (we can usually infer primary keys, foreign keys, relationships, have default made include trees, and generate CRUD methods like `get`).
+Note that the `get` code can be simplified using CRUD methods or ORM primitives.
 
 ### One to Many
 
@@ -316,8 +331,6 @@ export class Dog {
 
 ### Many to Many
 
-NOTE: `M:M` relationships have a [bug](https://github.com/bens-schreiber/cloesce/issues/88) in `v0.0.3`, but the syntax is as follows:
-
 ```ts
 @D1
 export class Student {
@@ -333,11 +346,74 @@ export class Course {
   @PrimaryKey
   id: number;
 
-  @ManyToMany("StudentsCourses")
+  @ManyToMany("StudentsCourses") // same unique id => same jct table.
   students: Student[];
 }
 ```
 
+### ORM Methods
+
+Cloesce provides a suite of ORM methods for getting, listing, updating and inserting models.
+
+#### Upsert
+
+```ts
+@D1
+class Horse {
+  // ...
+
+  @POST
+  static async post(@Inject { db }: Env, horse: Horse): Promise<Horse> {
+    const orm = Orm.fromD1(db);
+    await orm.upsert(Horse, horse, null);
+    return (await orm.get(Horse, horse.id, null)).value;
+  }
+}
+```
+
+#### List, Get
+
+```ts
+@D1
+class Horse {
+  // ...
+  @GET
+  static async get(@Inject { db }: Env, id: number): Promise<Horse> {
+    const orm = Orm.fromD1(db);
+    return (await orm.get(Horse, id, "default")).value;
+  }
+
+  @GET
+  static async list(@Inject { db }: Env): Promise<Horse[]> {
+    const orm = Orm.fromD1(db);
+    return (await orm.list(Horse, "default")).value;
+  }
+}
+```
+
+### CRUD Methods
+
+Generic GET, POST, PATCH (and in a future version, DEL) boilerplate methods do not need to be copied around. Cloesce supports CRUD generation, a syntactic sugar that adds the methods to the compiler output.
+
+```ts
+@CRUD(["POST", "GET", "LIST"])
+@D1
+export class CrudHaver {
+  @PrimaryKey
+  id: number;
+  name: string;
+}
+```
+
+which will generate client API methods like:
+
+```ts
+static async get(
+      id: number,
+      dataSource: "none" = "none",
+): Promise<HttpResult<CrudHaver>>
+```
+
 ### Plain Old Objects
 
 Simple non-model objects can be returned and serialized from a model method:
@@ -355,7 +431,7 @@ export class Cat {
     id: number;
 
     @GET
-    async query(): Promise<CatStuff> {
+    query(): CatStuff {
         return {
             catFacts: ["cats r cool"],
             catNames: ["reginald"]
@@ -384,16 +460,24 @@ class Foo {
 
 ## Unit Tests
 
-- `src/extractor/ts` run `npm test`
+- `src/frontend/ts` run `npm test`
 - `src/generator` run `cargo test`
 
 ## Integration Tests
 
-- To run the regression tests: `cargo run --bin test regression`
-- To run the pass fail extractor tests: `cargo run --bin test run-fail`
+- Regression tests: `cargo run --bin test regression`
+- Pass fail extractor tests: `cargo run --bin test run-fail`
 
 Optionally, pass `--check` if new snapshots should not be created.
 
+To update integration snapshots, run:
+
+- `cargo run --bin update`
+
+To delete any generated snapshots run:
+
+- `cargo run --bin update -- -d`
+
 ## E2E
 
 - `tests/e2e` run `npm test`

package/dist/cli.js

@@ -3,7 +3,7 @@ import { WASI } from "node:wasi";
 import fs from "node:fs";
 import { readFile } from "fs/promises";
 import path from "node:path";
-import { command, run, subcommands, flag, option, optional, string, } from "cmd-ts";
+import { command, run, subcommands, flag, string, positional, option, optional, } from "cmd-ts";
 import { Project } from "ts-morph";
 import { CidlExtractor } from "./extractor/extract.js";
 import { ExtractorError, ExtractorErrorCode, getErrorInfo } from "./common.js";
@@ -35,9 +35,9 @@ const cmds = subcommands({
                     args: [
                         "generate",
                         "all",
+                        path.join(outputDir, "cidl.pre.json"),
                         path.join(outputDir, "cidl.json"),
                         "wrangler.toml",
-                        "migrations/migrations.sql",
                         path.join(outputDir, "workers.ts"),
                         path.join(outputDir, "client.ts"),
                         config.clientUrl,
@@ -48,89 +48,9 @@ const cmds = subcommands({
                 await generate(allConfig);
             },
         }),
-        wrangler: command({
-            name: "wrangler",
-            description: "Generate wrangler.toml configuration",
-            args: {},
-            handler: async () => {
-                await generate({
-                    name: "wrangler",
-                    wasmFile: "generator.wasm",
-                    args: ["generate", "wrangler", "wrangler.toml"],
-                });
-            },
-        }),
-        d1: command({
-            name: "d1",
-            description: "Generate database schema",
-            args: {},
-            handler: async () => {
-                const config = loadCloesceConfig(process.cwd());
-                const outputDir = config.outputDir ?? ".generated";
-                await generate({
-                    name: "d1",
-                    wasmFile: "generator.wasm",
-                    args: [
-                        "generate",
-                        "d1",
-                        path.join(outputDir, "cidl.json"),
-                        "migrations/migrations.sql",
-                    ],
-                });
-            },
-        }),
-        workers: command({
-            name: "workers",
-            description: "Generate workers TypeScript",
-            args: {},
-            handler: async () => {
-                const config = loadCloesceConfig(process.cwd());
-                const outputDir = config.outputDir ?? ".generated";
-                if (!config.workersUrl) {
-                    console.error("Error: workersUrl must be defined in cloesce.config.json");
-                    process.exit(1);
-                }
-                await generate({
-                    name: "workers",
-                    wasmFile: "generator.wasm",
-                    args: [
-                        "generate",
-                        "workers",
-                        path.join(outputDir, "cidl.json"),
-                        path.join(outputDir, "workers.ts"),
-                        "wrangler.toml",
-                        config.workersUrl,
-                    ],
-                });
-            },
-        }),
-        client: command({
-            name: "client",
-            description: "Generate client TypeScript",
-            args: {},
-            handler: async () => {
-                const config = loadCloesceConfig(process.cwd());
-                const outputDir = config.outputDir ?? ".generated";
-                if (!config.clientUrl) {
-                    console.error("Error: clientUrl must be defined in cloesce-config.json");
-                    process.exit(1);
-                }
-                await generate({
-                    name: "client",
-                    wasmFile: "generator.wasm",
-                    args: [
-                        "generate",
-                        "client",
-                        path.join(outputDir, "cidl.json"),
-                        path.join(outputDir, "client.ts"),
-                        config.clientUrl,
-                    ],
-                });
-            },
-        }),
         extract: command({
             name: "extract",
-            description: "Extract models and write cidl.json only",
+            description: "Extract models and write cidl.pre.json",
             args: {
                 projectName: option({
                     long: "project-name",
@@ -141,7 +61,6 @@ const cmds = subcommands({
                     long: "out",
                     short: "o",
                     type: optional(string),
-                    description: "Output path (default: .generated/cidl.json)",
                 }),
                 inp: option({
                     long: "in",
@@ -169,6 +88,54 @@ const cmds = subcommands({
                 await extract({ ...args });
             },
         }),
+        migrate: command({
+            name: "migrate",
+            description: "Creates a database migration.",
+            args: {
+                name: positional({ type: string, displayName: "name" }),
+                debug: flag({
+                    long: "debug",
+                    short: "d",
+                    description: "Show debug output",
+                }),
+            },
+            handler: async (args) => {
+                const config = loadCloesceConfig(process.cwd(), args.debug);
+                const cidlPath = path.join(config.outputDir ?? ".generated", "cidl.json");
+                if (!fs.existsSync(cidlPath)) {
+                    console.error("Err: No cloesce file found, have you ran `cloesce compile`?");
+                    process.exit(1);
+                }
+                const migrationsPath = "./migrations";
+                if (!fs.existsSync(migrationsPath)) {
+                    fs.mkdirSync(migrationsPath);
+                }
+                const migrationPrefix = path.join(migrationsPath, `${timestamp()}_${args.name}`);
+                let wasmArgs = [
+                    "migrations",
+                    cidlPath,
+                    `${migrationPrefix}.json`,
+                    `${migrationPrefix}.sql`,
+                ];
+                // Add last migration if exists
+                {
+                    const files = fs.readdirSync(migrationsPath);
+                    const jsonFiles = files.filter((f) => f.endsWith(".json"));
+                    // Sort descending by filename
+                    jsonFiles.sort((a, b) => b.localeCompare(a, undefined, { numeric: true }));
+                    if (jsonFiles.length > 0) {
+                        wasmArgs.push(path.join(migrationsPath, jsonFiles[0]));
+                    }
+                }
+                const migrateConfig = {
+                    name: "migrations",
+                    wasmFile: "generator.wasm",
+                    args: wasmArgs,
+                };
+                // Runs a generator command. Exits the process on failure.
+                await generate(migrateConfig);
+            },
+        }),
     },
 });
 async function extract(opts) {
@@ -177,7 +144,7 @@ async function extract(opts) {
     const config = loadCloesceConfig(projectRoot, opts.debug);
     const searchPaths = opts.inp ? [opts.inp] : (config.paths ?? [root]);
     const outputDir = config.outputDir ?? ".generated";
-    const outPath = opts.out ?? path.join(outputDir, "cidl.json");
+    const outPath = opts.out ?? path.join(outputDir, "cidl.pre.json");
     const truncate = opts.truncateSourcePaths ?? config.truncateSourcePaths ?? false;
     const cloesceProjectName = opts.projectName ??
         config.projectName ??
@@ -205,6 +172,9 @@ async function extract(opts) {
         if (truncate) {
             ast.wrangler_env.source_path =
                 "./" + path.basename(ast.wrangler_env.source_path);
+            if (ast.app_source) {
+                ast.app_source = "./" + path.basename(ast.app_source);
+            }
             for (const model of Object.values(ast.models)) {
                 model.source_path =
                     "./" + path.basename(model.source_path);
@@ -267,6 +237,16 @@ function loadCloesceConfig(root, debug = false) {
     }
     return {};
 }
+function timestamp() {
+    const d = new Date();
+    return (d.getFullYear().toString() +
+        String(d.getMonth() + 1).padStart(2, "0") +
+        String(d.getDate()).padStart(2, "0") +
+        "T" +
+        String(d.getHours()).padStart(2, "0") +
+        String(d.getMinutes()).padStart(2, "0") +
+        String(d.getSeconds()).padStart(2, "0"));
+}
 function readPackageJsonProjectName(cwd) {
     const pkgPath = path.join(cwd, "package.json");
     let projectName = path.basename(cwd);