@c6fc/spellcraft 0.0.6 → 0.0.8

package/README.md

@@ -1,17 +1,189 @@
-# SpellCraft for JSonnet workflows
+# ✨ SpellCraft ✨
 
-Spellcraft is an opinionated approach to manifesting and applying configurations for various tools. SpellCraft is the core engine which can accept plugins like @c6fc/spellcraft-packer for Hashicorp Packer configurations, or @c6fc/spellcraft-aws for interacting with Amazon Web Services
+**The Sorcerer's Toolkit for Unified Configuration Management.**
 
-## Installation
+SpellCraft is a powerful framework for generating and managing configurations across a diverse toolchain. It was born from the challenge of orchestrating tools like Terraform, Packer, Kubernetes, and Ansible, each with its own fragmented and isolated configuration format (YAML, JSON, HCL, etc.).
 
-Install spellcraft with `npm install` and add plugins with `npx spellcraft importModule`
+SpellCraft provides a single, unified source of truth, allowing you to generate tightly-integrated, context-aware configurations for any tool, all from one place.
+
+[![NPM Version](https://img.shields.io/npm/v/@c6fc/spellcraft.svg)](https://www.npmjs.com/package/@c6fc/spellcraft)
+[![License](https://img.shields.io/npm/l/@c6fc/spellcraft.svg)](https://github.com/your-repo/spellcraft/blob/main/LICENSE)
+
+---
+
+## The SpellCraft Philosophy
+
+SpellCraft is built on three core principles to provide a superior Infrastructure-as-Code experience:
+
+1.  **Declarative Power (Jsonnet):** Configurations are written in [Jsonnet](https://jsonnet.org/), a superset of JSON. This gives you the power of variables, functions, conditionals, loops, and inheritance, eliminating the endless copy-pasting and structural limitations of plain YAML or JSON. Define a component once, and reuse it everywhere.
+
+2.  **Seamless Extensibility (Node.js):** Sometimes, declarative logic isn't enough. SpellCraft allows you to "escape" the confines of Jsonnet by writing custom logic in Node.js. Need to fetch a secret from a vault, call a third-party API, or perform complex data manipulation? Simply write a JavaScript function and expose it directly to your Jsonnet code as a `std.native()` function.
+
+3.  **Robust Modularity (NPM):** SpellCraft's module system is built on the battle-tested foundation of NPM. This means you can version, share, and manage your infrastructure modules just like any other software dependency. Leverage public or private NPM registries to build a reusable, maintainable, and collaborative infrastructure codebase.
+
+## Quick Start
+
+Get up and running with SpellCraft in minutes.
+
+### 1. Installation
+
+Install the SpellCraft CLI and core library into your project.
+
+```sh
+npm install --save @c6fc/spellcraft
+```
+
+### 2. Import a Module
+
+Modules are the building blocks of SpellCraft. Let's import a module for interacting with AWS. The `importModule` command will install the package from NPM and link it into your project.
+
+```sh
+npx spellcraft importModule @c6fc/spellcraft-aws-auth
+
+# Expected Output:
+# [*] Attempting to install @c6fc/spellcraft-aws-auth...
+# [+] Successfully installed @c6fc/spellcraft-aws-auth.
+# [+] Linked @c6fc/spellcraft-aws-auth as SpellCraft module 'awsauth'
+```
+This makes the `@c6fc/spellcraft-aws-auth` package available in your Jsonnet files under the name `awsauth`.
+
+### 3. Create Your First Spell
+
+A "Spell" is a `.jsonnet` file that defines the files you want to create. The top-level keys of the output object become filenames.
+
+Create a file named `manifest.jsonnet`:
+```jsonnet
+// manifest.jsonnet
+local modules = import 'modules';
+
+{
+  // The 'awsauth' module provides a native function `getCallerIdentity()`.
+  // We call it here and direct its output to a file named 'aws-identity.json'.
+  'aws-identity.json': modules.awsauth.getCallerIdentity(),
+
+  // We can also create YAML files. SpellCraft has built-in handlers for common types.
+  'config.yaml': {
+    apiVersion: 'v1',
+    kind: 'ConfigMap',
+    metadata: {
+      name: 'my-app-config',
+    },
+    data: {
+      region: std.native('envvar')('AWS_REGION') || 'us-east-1',
+      // The result of the native function is just data. We can reuse it!
+      callerArn: modules.awsauth.getCallerIdentity().Arn,
+    },
+  },
+}
+```
+
+### 4. Generate the Artifacts
+
+Use the `generate` command to render your `.jsonnet` file into the `render/` directory.
 
 ```sh
-$ npm i -p @c6fc/spellcraft
-$ npx spellcraft importModule @c6fc/spellcraft-packer packer
+npx spellcraft generate manifest.jsonnet
+
+# Expected Output:
+# [+] Linked @c6fc/spellcraft-aws-auth as awsauth.libsonnet
+# ...
+# [+] Registered native functions [getCallerIdentity, ...] to modules.awsauth
+# [+] Evaluating Jsonnet file manifest.jsonnet
+# [+] Writing files to: render
+#   -> aws-identity.json
+#   -> config.yaml
+# [+] Generation complete.
+```
+
+Check your `render/` directory. You will find two files created from your single source of truth!
+
+```
+.
+├── manifest.jsonnet
+├── node_modules/
+├── package.json
+└── render/
+    ├── aws-identity.json
+    └── config.yaml
+```
+
+## The SpellCraft CLI
+
+The `spellcraft` CLI is your primary interface for managing modules and generating files.
+
+### Core Commands
+
+-   `generate <filename>`: Renders a `.jsonnet` file and writes the output to the `render/` directory.
+-   `importModule <npmPackage> [name]`: Installs an NPM package and links it as a SpellCraft module. If `[name]` is omitted, it uses the default name defined by the module.
+
+### Extensible CLI
+
+Modules can extend the SpellCraft CLI with their own custom commands. For example, after importing `@c6fc/spellcraft-aws-auth`, you gain new AWS-related commands.
+
+Run `npx spellcraft --help` to see all available commands, including those added by modules.
+
+```sh
+$ npx spellcraft --help
+
+# ... output showing core commands and module-added commands ...
+Commands:
+  spellcraft generate <filename>                Generates files from a configuration
+  spellcraft importModule <npmPackage> [name]   Configures the current project to use a SpellCraft module
+  spellcraft aws-identity                       Display the AWS IAM identity of the SpellCraft context
+  spellcraft aws-exportcredentials              Export the current credentials as environment variables
 ```
 
-The importModule command above will expose the methods from `@c6fc/spellcraft-packer` with `local packer = require "packer"`
+## Programmatic Usage (API)
+
+For more advanced workflows, such as integration into larger automation scripts, you can use the `SpellFrame` class directly in your Node.js code.
+
+The typical flow is:
+1.  Instantiate `SpellFrame`.
+2.  Load necessary modules.
+3.  (Optional) Run module initializers with `init()`.
+4.  Render the Jsonnet file with `render()`.
+5.  Write the resulting object to disk with `write()`.
+
+```javascript
+// my-automation-script.js
+const { SpellFrame } = require('@c6fc/spellcraft');
+const path = require('path');
+
+// 1. Instantiate the SpellFrame
+// Options allow you to customize output paths, cleaning behavior, etc.
+const frame = new SpellFrame({
+  renderPath: "dist", // Output to 'dist/' instead of 'render/'
+  cleanBeforeRender: true,
+});
+
+(async () => {
+  try {
+    // 2. Load modules programmatically (this assumes they are in package.json)
+    // This loads modules listed in 'spellcraft_modules/packages.json'
+    // and from the local 'spellcraft_modules/' directory.
+    // frame.loadModuleByName('my-module-key', 'my-npm-package');
+
+    // 3. Initialize modules (if any modules registered an init function)
+    await frame.init();
+
+    // 4. Render the master Jsonnet file
+    const manifest = await frame.render(path.resolve('./manifest.jsonnet'));
+
+    // The result is available in memory
+    console.log('Rendered Manifest:', JSON.stringify(manifest, null, 2));
+
+    // 5. Write the manifest object to the filesystem
+    frame.write(manifest);
+
+    console.log('Successfully wrote files to the dist/ directory!');
+
+  } catch (error) {
+    console.error('An error occurred during the SpellCraft process:', error);
+    process.exit(1);
+  }
+})();
+```
 
+## Creating Your Own Spells (Modules)
 
-## More details to be added soon
+When you're ready to start writing your own modules and unleashing the true power of SpellCraft, check out **[create-spellcraft-module](https://www.npmjs.com/package/@c6fc/spellcraft)**

package/bin/spellcraft.js

@@ -78,18 +78,26 @@ const spellframe = new SpellFrame();
                     describe: 'Jsonnet configuration file to consume',
                     type: 'string',
                     demandOption: true,
+                }).option('skip-module-cleanup', {
+                    alias: 's',
+                    type: 'boolean',
+                    description: 'Leave temporary modules intact after rendering'
                 });
             },
             async (argv) => { // No JSDoc for internal handler
-                try {
+                // try {
+                    if (argv['s']) {
+                        sfInstance.cleanModulesAfterRender = false;
+                    }
+
                     await sfInstance.init();
                     await sfInstance.render(argv.filename);
                     await sfInstance.write();
                     console.log("[+] Generation complete.");
-                } catch (error) {
+                /*} catch (error) {
                     console.error(`[!] Error during generation: ${error.message.red}`);
                     process.exit(1);
-                }
+                }*/
             })
 
             .command("importModule <npmPackage> [name]", "Configures the current project to use a SpellCraft plugin as an import", (yargsInstance) => {
@@ -107,7 +115,7 @@ const spellframe = new SpellFrame();
             },
             async (argv) => {
                 await sfInstance.importSpellCraftModuleFromNpm(argv.npmPackage, argv.name);
-                console.log(`[+] Module '${argv.npmPackage.green}' ${argv.name ? `(aliased as ${argv.name.green})` : ''} linked successfully.`);
+                console.log(`[+] Module '${argv.npmPackage.green}' ${argv.name ? `(aliased as ${argv.name.green}) ` : ''}linked successfully.`);
             });
 
         // No JSDoc for CLI extensions loop if considered internal detail

package/package.json

@@ -8,7 +8,7 @@
   },
   "name": "@c6fc/spellcraft",
   "description": "Extensible JSonnet CLI platform",
-  "version": "0.0.6",
+  "version": "0.0.8",
   "main": "src/index.js",
   "directories": {
     "lib": "lib"

package/src/index.js

@@ -1,4 +1,3 @@
-// ./src/index.js
 'use strict';
 
 const fs = require("fs");
@@ -23,9 +22,10 @@ exports.SpellFrame = class SpellFrame {
 
     constructor(options = {}) {
         const defaults = {
-            renderPath: "./render",
-            modulePath: "./modules",
+            renderPath: "render",
+            spellcraftModuleRelativePath: ".spellcraft_linked_modules",
             cleanBeforeRender: true,
+            cleanModulesAfterRender: true,
             useDefaultFileHandlers: true
         };
 
@@ -42,26 +42,31 @@ exports.SpellFrame = class SpellFrame {
         this.lastRender = null;
         this.activePath = null;
         this.loadedModules = [];
-        this.modulePath = path.resolve(path.join(process.cwd(), this.modulePath));
         this.magicContent = {}; // { modulefile: [...snippets] }
         this.registeredFunctions = {}; // { modulefile: [...functionNames] }
 
         this.renderPath = path.resolve(this.currentPackagePath, this.renderPath);
-        this.modulePath = path.resolve(this.currentPackagePath, this.modulePath);
+        this.modulePath = path.resolve(this.currentPackagePath, this.spellcraftModuleRelativePath);
 
-        this.jsonnet = new Jsonnet()
-            .addJpath(path.join(__dirname, '../lib')) // For core SpellCraft libsonnet files
-            .addJpath(this.modulePath); // For dynamically generated module imports
+        this.jsonnet = new Jsonnet();
 
-        // Add built-in native functions
-        this.addNativeFunction("envvar", (name) => process.env[name] || false, "name");
-        this.addNativeFunction("path", () => this.activePath || process.cwd()); // Use activePath if available
+        this.addJpath(path.join(__dirname, '../lib')) // For core SpellCraft libsonnet files
+            .addJpath(path.join(this.modulePath)) // For dynamically generated module imports
+            .addNativeFunction("envvar", (name) => process.env[name] || false, "name")
+            .addNativeFunction("path", () => this.activePath || process.cwd()) // Use activePath if available
+            .cleanModulePath();
 
+        this.loadedModules = this.loadModulesFromPackageList();
+        this.loadModulesFromModuleDirectory();
+
+        return this;
+    }
+
+    cleanModulePath() {
         if (!fs.existsSync(this.modulePath)) {
             fs.mkdirSync(this.modulePath, { recursive: true });
         }
 
-        // Clean up the modules on init
         try {
             fs.readdirSync(this.modulePath)
                 .map(e => path.join(this.modulePath, e))
@@ -71,9 +76,6 @@ exports.SpellFrame = class SpellFrame {
             throw new Error(`[!] Could not create/clean up temporary module folder ${path.dirname(this.modulePath).green}: ${e.message.red}`);
         }
 
-        this.loadedModules = this.loadModulesFromPackageList();
-        this.loadModulesFromModuleDirectory();
-
         return this;
     }
 
@@ -126,7 +128,8 @@ exports.SpellFrame = class SpellFrame {
     }
 
     addJpath(jpath) {
-        this.jsonnet = this.jsonnet.addJpath(jpath);
+        // console.log(`[*] Adding Jpath ${jpath}`);
+        this.jsonnet.addJpath(jpath);
         return this;
     }
 
@@ -266,7 +269,7 @@ exports.SpellFrame = class SpellFrame {
 
         jsModuleFiles.forEach(file => {
             this.loadFunctionsFromFile(file, as);
-            console.log(`[+] Loaded [${this.registeredFunctions.join(', ').cyan}] from ${path.basename(file).green} into module.${as.green}`);
+            console.log(`[+] Loaded [${this.registeredFunctions[as].join(', ').cyan}] from ${path.basename(file).green} into modules.${as.green}`);
         });
 
         return this;
@@ -275,12 +278,10 @@ exports.SpellFrame = class SpellFrame {
     loadModulesFromModuleDirectory() {
         const spellcraftModulesPath = path.join(this.currentPackagePath, 'spellcraft_modules');
         if (!fs.existsSync(spellcraftModulesPath)) {
-            return { registeredFunctions: [], magicContent: [] };
+            return this;
         }
 
-        const spellcraftConfig = thisPackage.config || thisPackage.spellcraft; // Allow 'spellcraft' key too
-
-        if (!!spellcraftConfig?.spellcraft_module_default_name) {
+        if (!!this.currentPackage?.config?.spellcraft_module_default_name) {
             console.log("[-] This package is a SpellCraft module. Skipping directory-based module import.");
             return { registeredFunctions: [], magicContent: [] };
         }
@@ -308,11 +309,10 @@ exports.SpellFrame = class SpellFrame {
             console.log(`[+] Successfully installed ${npmPackage.blue}.`);
         }
 
-        const importModuleConfig = this.getModulePackage(npmPackage).config;
+        const importModuleConfig = this.getModulePackage(`${npmPackage}/package.json`).config;
         const currentPackageConfig = this.currentPackage.config;
 
         if (!name && !!!importModuleConfig?.spellcraft_module_default_name) {
-            // console.log("Package config:", moduleJson);
             throw new Error(`[!] No import name specified for ${npmPackage.blue}, and it has no 'spellcraft_module_default_name' in its package.json config.`.red);
         }
 
@@ -360,13 +360,32 @@ exports.SpellFrame = class SpellFrame {
 
         this.activePath = path.dirname(absoluteFilePath); // Set active path for relative 'path()' calls
 
+        this.magicContent.modules.push(this.loadedModules.flatMap(e => {
+            return `\t${e}:: import '${e}.libsonnet'`;
+        }));
+
+        if (this.registeredFunctions.modules.length > 0) {
+            fs.writeFileSync(path.join(this.modulePath, `modules`), `{\n${this.magicContent.modules.join(",\n")}\n}`, 'utf-8');
+            console.log(`[+] Registered native functions [${this.registeredFunctions.modules.join(', ').cyan}] to modules.${'modules'.green}`);
+        }
+
+        delete this.magicContent.modules;
+
         Object.keys(this.magicContent).forEach(e => {
-            fs.writeFileSync(path.join(this.modulePath, e), `{\n${this.magicContent[e].join(",\n")}\n}`, 'utf-8');
+            fs.appendFileSync(path.join(this.modulePath, `${e}.libsonnet`), ` + {\n${this.magicContent[e].join(",\n")}\n}`, 'utf-8');
             console.log(`[+] Registered native functions [${this.registeredFunctions[e].join(', ').cyan}] to modules.${e.green} `);
         });
-        
+
         console.log(`[+] Evaluating Jsonnet file ${path.basename(absoluteFilePath).green}`);
         this.lastRender = JSON.parse(await this.jsonnet.evaluateFile(absoluteFilePath));
+
+        if (this.cleanModulesAfterRender) {
+            this.cleanModulePath();
+
+            fs.rmdirSync(this.modulePath);
+        } else {
+            console.log(`[*] Leaving ${this.spellcraftModuleRelativePath} in place.`.magenta);
+        }
         
         return this.lastRender;
     }