@travetto/manifest 3.0.0-rc.17 → 3.0.0-rc.18

package/README.md

@@ -1,14 +1,269 @@
 <!-- This file was generated by @travetto/doc and should not be modified directly -->
 <!-- Please modify https://github.com/travetto/travetto/tree/main/module/manifest/DOC.ts and execute "npx trv doc" to rebuild -->
 # Manifest
-## Manifest support
+## Support for project indexing, manifesting, along with file watching
 
 **Install: @travetto/manifest**
 ```bash
 npm install @travetto/manifest
+
+# or
+
+yarn add @travetto/manifest
+```
+
+This module aims to be the boundary between the file system and the code.  The module provides:
+
+   
+   *  Project Manifesting
+   *  Manifest Delta
+   *  Class and Function Metadata
+   *  Runtime Indexing
+   *  Path Normalization
+   *  File Watching
+
+## Project Manifesting
+The project manifest fulfills two main goals: Compile-time Support, and Runtime Knowledge of the project.
+
+### Compile-time Support
+During the compilation process, the compiler needs to know every file that is eligible for compilation, when the file was last created/modified, and any specific patterns for interacting with a given file (e.g. transformers vs. testing code vs. support files that happen to share a common extension with code). 
+
+### Runtime Knowledge
+Additionally, once the code has been compiled (or even bundled after that), the executing process needs to know what files are available for loading, and any patterns necessary for knowing which files to load versus which ones to ignore. This allows for dynamic loading of modules/files without knowledge/access to the file system, and in a more performant manner.
+
+## Manifest Delta
+During the compilation process, it is helpful to know how the output content differs from the manifest, which is produced from the source input. The [ManifestDeltaUtil](https://github.com/travetto/travetto/tree/main/module/manifest/src/delta.ts#L21) provides the functionality for a given manifest, and will produce a stream of changes grouped by module.  This is the primary input into the [Compiler](https://github.com/travetto/travetto/tree/main/module/compiler#readme "The compiler infrastructure for the Travetto framework")'s incremental behavior to know when a file has changed and needs to be recompiled.
+
+## Class and Function Metadata
+
+For the framework to work properly, metadata needs to be collected about files, classes and functions to uniquely identify them, with support for detecting changes during live reloads.  To achieve this, every `class` is decorated with an additional field of `Ⲑid`.  `Ⲑid` represents a computed id that is tied to the file/class combination.
+
+`Ⲑid` is used heavily throughout the framework for determining which classes are owned by the framework, and being able to lookup the needed data from the [RootIndex](https://github.com/travetto/travetto/tree/main/module/manifest/src/root-index.ts) using the `getFunctionMetadata` method.
+
+**Code: Test Class**
+```typescript
+export class TestClass {
+  async doStuff(): Promise<void> { }
+}
+```
+
+**Code: Test Class Compiled**
+```javascript
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.TestClass = void 0;
+const tslib_1 = require("tslib");
+const Ⲑ_root_index_1 = tslib_1.__importStar(require("@travetto/manifest/src/root-index.js"));
+var ᚕf = "@travetto/manifest/doc/test-class.js";
+class TestClass {
+    static Ⲑinit = Ⲑ_root_index_1.RootIndex.registerFunction(TestClass, ᚕf, 197152026, { doStuff: { hash: 51337554 } }, false, false);
+    async doStuff() { }
+}
+exports.TestClass = TestClass;
+```
+
+**Terminal: Index Lookup at Runtime**
+```bash
+$ trv main ./doc/lookup.ts
+
+{
+  id: '@travetto/manifest:doc/test-class￮TestClass',
+  source: './doc/test-class.ts',
+  hash: 197152026,
+  methods: { doStuff: { hash: 51337554 } },
+  abstract: false,
+  synthetic: false
+}
+```
+
+## Module Indexing
+Once the manifest is created, the application runtime can now read this manifest, which allows for influencing runtime behavior. The most common patterns include:
+   
+   *  Loading all source files
+   *  Iterating over every test file
+   *  Finding all source folders for watching
+   *  Find all output folders for watching
+   *  Determining if a module is available or not
+   *  Resource scanning
+   *  Providing contextual information when provided a filename, import name, etc (e.g. logging, testing output)
+
+## Path Normalization
+By default, all paths within the framework are assumed to be in a POSIX style, and all input paths are converted to the POSIX style.  This works appropriately within a Unix and a Windows environment.  This module offers up [path](https://github.com/travetto/travetto/tree/main/module/manifest/src/path.ts#L7) as an equivalent to [Node](https://nodejs.org)'s [http](https://nodejs.org/api/path.html) library.  This allows for consistent behavior across all file-interactions, and also allows for easy analysis if [Node](https://nodejs.org)'s [http](https://nodejs.org/api/path.html) library is ever imported.
+
+## File Watching
+The module also leverages [fetch](https://www.npmjs.com/package/@parcel/watcher), to expose a single function of `watchFolders`. Only the [Compiler](https://github.com/travetto/travetto/tree/main/module/compiler#readme "The compiler infrastructure for the Travetto framework") module packages [fetch](https://www.npmjs.com/package/@parcel/watcher) as a direct dependency.  This means, that in production, by default all watch operations will fail with a missing dependency.
+
+**Code: Watch Folder Signature**
+```typescript
+export type WatchEvent = { action: 'create' | 'update' | 'delete', file: string };
+type EventFilter = (ev: WatchEvent) => boolean;
+
+export type WatchEventListener = (ev: WatchEvent, folder: string) => void;
+export type WatchConfig = {
+  /**
+   * Predicate for filtering events
+   */
+  filter?: EventFilter;
+  /**
+   * List of top level folders to ignore
+   */
+  ignore?: string[];
+  /**
+   * If watching a folder that doesn't exist, should it be created?
+   */
+  createMissing?: boolean;
+  /**
+   * Include files that start with '.'
+   */
+  includeHidden?: boolean;
+};
+
+/**
+ * Leverages @parcel/watcher to watch a series of folders
+ * @param folders
+ * @param onEvent
+ * @private
+ */
+export async function watchFolders(
+  folders: string[] | [folder: string, targetFolder: string][],
+  onEvent: WatchEventListener,
+  config: WatchConfig = {}
+): Promise<() => Promise<void>> {
+```
+
+This method allows for watching one or more folders, and registering a callback that will fire every time a file changes, and which of the registered folders it was triggered within. The return of the `watchFolders` is a cleanup method, that when invoked will remove and stop all watching behavior.
+
+## Anatomy of a Manifest
+
+**Code: Manifest for @travetto/manifest**
+```typescript
+{
+  "generated": 1868155200000,
+  "moduleType": "commonjs",
+  "mainModule": "@travetto/manifest",
+  "mainFolder": "module/manifest",
+  "workspacePath": "<generated>",
+  "monoRepo": true,
+  "outputFolder": ".trv_output",
+  "toolFolder": ".trv_build",
+  "compilerFolder": ".trv_compiler",
+  "packageManager": "npm",
+  "modules": {
+    "@travetto/manifest": {
+      "main": true,
+      "name": "@travetto/manifest",
+      "version": "3.0.0-rc.17",
+      "local": true,
+      "internal": false,
+      "sourceFolder": "module/manifest",
+      "outputFolder": "node_modules/@travetto/manifest",
+      "files": {
+        "$root": [
+          [ "DOC.html", "unknown", 1868155200000 ],
+          [ "LICENSE", "unknown", 1868155200000 ],
+          [ "README.md", "md", 1868155200000 ]
+        ],
+        "doc": [
+          [ "DOC.ts", "ts", 1868155200000, "doc" ],
+          [ "doc/lookup.ts", "ts", 1868155200000, "doc" ],
+          [ "doc/test-class.ts", "ts", 1868155200000, "doc" ]
+        ],
+        "$index": [
+          [ "__index__.ts", "ts", 1868155200000 ]
+        ],
+        "$package": [
+          [ "package.json", "package-json", 1868155200000 ]
+        ],
+        "test": [
+          [ "test/path.ts", "ts", 1868155200000, "test" ],
+          [ "test/root-index.ts", "ts", 1868155200000, "test" ]
+        ],
+        "test/fixtures": [
+          [ "test/fixtures/simple.ts", "fixture", 1868155200000, "test" ]
+        ],
+        "$transformer": [
+          [ "support/transformer.function-metadata.ts", "ts", 1868155200000, "compile" ]
+        ],
+        "src": [
+          [ "src/delta.ts", "ts", 1868155200000 ],
+          [ "src/dependencies.ts", "ts", 1868155200000 ],
+          [ "src/manifest-index.ts", "ts", 1868155200000 ],
+          [ "src/module.ts", "ts", 1868155200000 ],
+          [ "src/package.ts", "ts", 1868155200000 ],
+          [ "src/path.ts", "ts", 1868155200000 ],
+          [ "src/root-index.ts", "ts", 1868155200000 ],
+          [ "src/types.ts", "ts", 1868155200000 ],
+          [ "src/typings.d.ts", "typings", 1868155200000 ],
+          [ "src/util.ts", "ts", 1868155200000 ],
+          [ "src/watch.ts", "ts", 1868155200000 ]
+        ],
+        "bin": [
+          [ "bin/context.d.ts", "typings", 1868155200000 ],
+          [ "bin/context.js", "js", 1868155200000 ]
+        ]
+      },
+      "profiles": [ "std" ],
+      "parents": []
+    }
+  }
+}
 ```
 
-This module provides functionality for basic path functionality and common typings for manifests
+### General Context
+The general context describes the project-space and any important information for how to build/execute the code.
+
+The context contains:
+   
+   *  A generated timestamp
+   *  Module Type: [commonjs](https://nodejs.org/api/modules.html) or [module](https://nodejs.org/api/esm.html)
+   *  The main module to execute. **Note**: This primarily pertains to mono-repo support when there are multiple modules in the project
+   *  The root path of the project/workspace
+   *  Whether or not the project is a mono-repo. **Note**: This is determined by using the 'workspaces' field in your [object Object]
+   *  The location where all compiled code will be stored.  Defaults to: `.trv_output`.  **Note**: Can be overridden in your [object Object] in 'travetto.outputFolder'
+   *  The location where the intermediate compiler will be created. Defaults to: `.trv_compiler`
+   *  The location where tooling will be able to write to. Defaults to: `.trv_output`
+   *  Which package manager is in use [Npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) or [Yarn](https://yarnpg.com)
+
+### Modules
+The modules represent all of the [Travetto](https://travetto.dev)-aware dependencies (including dev dependencies) used for compiling, testing and executing.  A prod-only version is produced when packaging the final output.
 
-### Module Indexing
-The bootstrap process will also produce an index of all source files, which allows for fast in-memory scanning.  This allows for all the automatic discovery that is used within the framework.
+Each module contains:
+   
+   *  The dependency npm name
+   *  The dependency version
+   *  A flag to determine if its a local module
+   *  A flag to determine if the module is public (could be published to npm)
+   *  The path to the source folder, relative to the workspace root
+   *  The path to the output folder, relative to the workspace output root
+   *  The list of all files
+   *  The profiles a module applies to.  Values are std, test, compile, doc.  Any empty value implies std
+   *  Parent modules that imported this module
+
+### Module Files
+The module files are a simple categorization of files into a predetermined set of folders:
+   
+   *  $root - All uncategorized files at the module root
+   *  $index - __index__.ts, index.ts files at the root of the project
+   *  $package - The [Package JSON](https://docs.npmjs.com/cli/v9/configuring-npm/package-json) for the project
+   *  src - Code that should be automatically loaded at runtime. All .ts files under the src/ folder
+   *  test - Code that contains test files. All .ts files under the test/ folder
+   *  test/fixtures - Test resource files, pertains to the main module only. Located under test/fixtures/
+   *  resources - Packaged resource, meant to pertain to the main module only. Files, under resources/
+   *  support - All .ts files under the support/ folder
+   *  support/resources - Packaged resource files, meant to be included by other modules, under support/resources/
+   *  support/fixtures - Test resources meant to shared across modules.  Under support/fixtures/
+   *  doc - Documentation files. All .ts files under the doc/ folder
+   *  $transformer - All .ts files under the pattern support/transform*.  These are used during compilation and never at runtime
+   *  bin - Entry point .js files.  All .js files under the bin/ folder
+
+Within each file there is a pattern of either a 3 or 4 element array:
+
+**Code: Sample file**
+```typescript
+[
+  "test/path.ts", // The module relative source path
+  "ts", // The file type ts, js, package-json, typings, md, json, unknown
+  1676751649201.1897, // Stat timestamp
+  "test" // Optional profile
+]
+```

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@travetto/manifest",
-  "version": "3.0.0-rc.17",
-  "description": "Manifest support",
+  "version": "3.0.0-rc.18",
+  "description": "Support for project indexing, manifesting, along with file watching",
   "keywords": [
     "path",
     "package",

package/src/watch.ts

@@ -1,12 +1,6 @@
 import fs from 'fs/promises';
 import { path } from './path';
 
-export type WatchEvent = { action: 'create' | 'update' | 'delete', file: string };
-
-export type WatchEventListener = (ev: WatchEvent, folder: string) => void;
-type EventFilter = (ev: WatchEvent) => boolean;
-type WatchConfig = { filter?: EventFilter, ignore?: string[], createMissing?: boolean, includeHidden?: boolean };
-
 async function getWatcher(): Promise<typeof import('@parcel/watcher')> {
   try {
     return await import('@parcel/watcher');
@@ -16,6 +10,29 @@ async function getWatcher(): Promise<typeof import('@parcel/watcher')> {
   }
 }
 
+export type WatchEvent = { action: 'create' | 'update' | 'delete', file: string };
+type EventFilter = (ev: WatchEvent) => boolean;
+
+export type WatchEventListener = (ev: WatchEvent, folder: string) => void;
+export type WatchConfig = {
+  /**
+   * Predicate for filtering events
+   */
+  filter?: EventFilter;
+  /**
+   * List of top level folders to ignore
+   */
+  ignore?: string[];
+  /**
+   * If watching a folder that doesn't exist, should it be created?
+   */
+  createMissing?: boolean;
+  /**
+   * Include files that start with '.'
+   */
+  includeHidden?: boolean;
+};
+
 /**
  * Leverages @parcel/watcher to watch a series of folders
  * @param folders