@fnndsc/chili 3.2.5

package/docs/13_run_a_plugin.adoc

@@ -0,0 +1,271 @@
+= run a plugin
+
+== intro
+
+Assuming some data has been uploaded into ChRIS, perhaps one of the most common intents is to run some analysis on this data. The simplest case would be running a single specific analysis, or in the parlance of ChRIS, a single _plugin_. A ChRIS plugin can be thought of as a command line program with its associated _flags_ -- these _flags_ control how the plugin will run. Each plugin will have its own unique set of flags.
+
+Additionally, each ChRIS plugin has two _implicit_ parameters that point to where the input data is in ChRIS and where to save the output data. We don't need to worry about this since ChRIS takes care of that for us.
+
+Mostly the only thing we need to know is _where_ in ChRIS is the data we wish to analyze, what analysis program to run, and what _flags_ (if any) to use in the analysis.
+
+== prerequisites
+
+The following prerequisites are relevant:
+
+* you are in some Linux (or macOS) terminal with access to the `chili` program. Again, consult the appropriate manual page if this hasn't been setup yet.
+
+* you have already logged into a ChRIS instance. Please consult the appropriate manual page if you haven't logged in yet.
+
+== first gather your data
+
+Let's assume you have data on your computer that you want to process. Let's assume this data is a set of image files, e.g. DICOM images. Of course, it can be anything. The first step is to get this data into ChRIS. Let's say your login in your computer is user `jane` and you  have data in a directory here:
+
+```
+/home/jane/data/brainproject/scan-1
+```
+
+you can upload this to ChRIS with
+
+```
+chili host upload ~/data/brainproject/scan-1 /home/jane.doe/data/brainproject/scan-1
+```
+
+The first part `~/data/brainproject/scan-1` is the location of the data on your local computer, and the second part `/home/jane.doe/data/brainproject/scan-1` is the location in ChRIS to which you want to upload. Here, we assume your ChRIS login name is `jane.doe`. Note how the internal ChRIS file system mimics a typical Linux system with a `/home/<username>` layout.
+
+You should see something like this:
+
+```
+   ____ _     ___ _     ___ 
+  / ___| |__ |_ _| |   |_ _|
+ | |   | '_ \ | || |    | | 
+ | |___| | | || || |___ | | 
+  \____|_| |_|___|_____|___|
+                            
+The ChRIS Interactive Line Interface
+Scanning files to upload...
+Uploading [████████████████████████████████████████] 100% | ETA: 0s | 8/8 files | 4.01 KB/4.01 KB
+
+Upload Summary:
+┌──────────────────────────────┬────────────────────┐
+│ Metric                       │ Value              │
+├──────────────────────────────┼────────────────────┤
+│ Total files                  │ 8                  │
+├──────────────────────────────┼────────────────────┤
+│ Successfully uploaded        │ 8                  │
+├──────────────────────────────┼────────────────────┤
+│ Failed to upload             │ 0                  │
+├──────────────────────────────┼────────────────────┤
+│ Total data uploaded          │ 4.01 KB            │
+├──────────────────────────────┼────────────────────┤
+│ Average upload speed         │ 5.18 KB/s          │
+├──────────────────────────────┼────────────────────┤
+│ Duration                     │ 0.77 seconds       │
+└──────────────────────────────┴────────────────────┘
+
+```
+
+== now create a feed
+
+Having done this, the next step is to create a _feed_. A _feed_ is simply the name for an analysis that you want to run. The ChRIS UI can interpret this _feed_ as a graphical representation showing all the analysis performed on the data and how different analyses relate to each other.
+
+A _feed_ **always** starts with a single root node. We need to create a new feed with a root node linked to the data we just uploaded.
+
+```
+chili feed create --params "title: A demo analysis" --dirs "/home/jane.doe/brainproject/scan-1"
+```
+
+where the text "A demo analysis" is the title of the feed, and the `--dirs` flag denotes the "directories" inside ChRIS to add to the root. Here, we only add one, the directory containing the data we just uploaded, but you can specify multiple ChRIS directories separating them with a comma character. If successful, this should return something like:
+
+```
+   ____ _     ___ _     ___ 
+  / ___| |__ |_ _| |   |_ _|
+ | |   | '_ \ | || |    | | 
+ | |___| | | || || |___ | | 
+  \____|_| |_|___|_____|___|
+                            
+The ChRIS Interactive Line Interface
+
+Feed Creation Result:
+┌────────────────────┬──────────────────────────────────────────────────┐
+│ Property           │ Value                                            │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Status             │ Success                                          │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Plugin ID          │ 37                                               │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Feed ID            │ 37                                               │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Feed Name          │ A demo analysis                                  │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Owner              │ jane.doe                                         │
+└────────────────────┴──────────────────────────────────────────────────┘
+```
+
+Note the ID of the plugin. This is needed in the next step.
+
+== now let's run an analysis on this root node
+
+Having created the feed, the next step is to run your actual analysis plugin. From the feed creation output above, note the **Plugin ID** value (in this example, it's 37). This is the plugin instance ID of the `pl-dircopy` that forms the root of your feed. We need this ID to tell ChRIS that our next plugin should use this data as input.
+
+Let's say you want to run `pl-dcm2niix` to convert your DICOM images to NIfTI format. You would invoke it like this:
+
+```
+chili pluginInstanceID=37 plugin run pl-dcm2niix --params "outputdir: converted"
+```
+
+Let's break this down:
+
+* `pluginInstanceID=37` - This sets the **context** to plugin instance 37 (our pl-dircopy root node). This tells ChRIS: "use the output of instance 37 as input for the next plugin"
+
+* `plugin run` - The chili command to execute a plugin
+
+* `pl-dcm2niix` - The **searchable** identifier for the plugin. This can be:
+  - Simple name: `pl-dcm2niix` (matches by name)
+  - Exact name: `name_exact: pl-dcm2niix` (precise match)
+  - With version: `name: pl-dcm2niix, version: 2.1.1` (specific version)
+  - By ID: `id: 42` (if you know the plugin's database ID)
+
+* `--params "outputdir: converted"` - Plugin-specific parameters as comma-separated key:value pairs
+
+If successful, you should see output similar to:
+
+```
+   ____ _     ___ _     ___
+  / ___| |__ |_ _| |   |_ _|
+ | |   | '_ \ | || |    | |
+ | |___| | | || || |___ | |
+  \____|_| |_|___|_____|___|
+
+The ChRIS Interactive Line Interface
+
+Plugin Instance Created:
+┌────────────────────┬──────────────────────────────────────────────────┐
+│ Property           │ Value                                            │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Status             │ Success                                          │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Instance ID        │ 38                                               │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Plugin Name        │ pl-dcm2niix                                      │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Feed ID            │ 37                                               │
+├────────────────────┼──────────────────────────────────────────────────┤
+│ Status             │ scheduled                                        │
+└────────────────────┴──────────────────────────────────────────────────┘
+```
+
+The new plugin instance (ID 38 in this example) has been scheduled to run. ChRIS will execute it on a compute resource, reading data from the output of instance 37 (the dircopy) and writing results to instance 38's output directory.
+
+== understanding the feed structure
+
+After creating the feed and running the plugin, your ChRIS filesystem will look like this:
+
+```
+/home/jane.doe/
+  └── feeds/
+      └── feed_37/                    # The feed we created
+          ├── pl-dircopy_37/          # Root node (instance ID 37)
+          │   └── data/               # Contains: brainproject/scan-1/
+          └── pl-dcm2niix_38/         # Child node (instance ID 38)
+              └── data/               # Will contain: converted NIfTI files
+```
+
+Notice the directory naming pattern: `<plugin-name>_<instance-id>`. This encodes both:
+1. Which plugin created the data
+2. The instance ID (for referencing in further operations)
+
+The `data/` subdirectory in each plugin instance directory contains that plugin's output.
+
+== viewing results
+
+Once the plugin completes (this may take time depending on the plugin), you can browse the results:
+
+```
+chili chefs ls /home/jane.doe/feeds/feed_37/pl-dcm2niix_38/data/
+```
+
+This will show the NIfTI files produced by pl-dcm2niix.
+
+== continuing the analysis
+
+To run another plugin using the output of pl-dcm2niix as input, simply use its instance ID:
+
+```
+chili pluginInstanceID=38 plugin run pl-freesurfer --params "subject: patient001"
+```
+
+This creates a new plugin instance (let's say ID 39) that reads from instance 38's output, creating a graph:
+
+```
+pl-dircopy_37 (root)
+  └── pl-dcm2niix_38
+        └── pl-freesurfer_39
+```
+
+== parallel branches
+
+You can also create parallel branches by running multiple plugins from the same parent:
+
+```
+# Branch 1: Brain extraction
+chili pluginInstanceID=38 plugin run pl-fsl-bet --params "threshold: 0.5"
+
+# Branch 2: Registration
+chili pluginInstanceID=38 plugin run pl-fsl-flirt --params "reference: MNI152"
+```
+
+This creates:
+
+```
+pl-dircopy_37 (root)
+  └── pl-dcm2niix_38
+        ├── pl-fsl-bet_40      # Parallel branch 1
+        └── pl-fsl-flirt_41    # Parallel branch 2
+```
+
+== asynchronous execution
+
+**Important:** Plugin execution in ChRIS is **asynchronous**. When you run a plugin:
+
+1. ChRIS validates the request
+2. Schedules the job on a compute resource
+3. Returns immediately with the instance ID
+4. The plugin runs in the background
+
+The plugin status starts as "scheduled", transitions to "started" when running, and eventually becomes "finishedSuccessfully" or "finishedWithError".
+
+You can check status by:
+
+```
+chili feeds list
+# Look for your feed and check instance statuses
+```
+
+Or use the ChRIS web UI to visualize the feed graph and see real-time status updates.
+
+== the chell alternative
+
+The `chell` interactive shell provides a more intuitive way to run plugins by treating them like local executables. Instead of managing plugin instance IDs explicitly, you simply navigate to your data and type the plugin name:
+
+```
+chell> cd ~/uploads/brainproject/scan-1
+chell> pl-dcm2niix-v2.1.1 --outputdir converted
+Feed created: 37
+Job scheduled: pl-dcm2niix-v2.1.1 (ID: 38)
+Output will be in: ~/feeds/feed_37/pl-dcm2niix_38/data/
+```
+
+See `chell/docs/pluginrun.adoc` for comprehensive documentation on this workflow.
+
+== summary
+
+Running a plugin analysis in ChRIS involves:
+
+1. **Upload data** to ChRIS using `chili chefs upload`
+2. **Create a feed** using `chili feed create --dirs <path>`
+3. **Note the plugin instance ID** from the feed creation output
+4. **Run your plugin** using `chili pluginInstanceID=<id> plugin run <searchable> --params <params>`
+5. **View results** by browsing the feed filesystem structure
+6. **Continue the analysis** by running additional plugins using the new instance IDs
+
+The key concept: Each plugin instance has an ID, and you specify the "previous" instance ID to create parent-child relationships in the analysis graph.

package/docs/20_plugins.adoc

@@ -0,0 +1,16 @@
+= plugins
+
+== intro
+
+`plugins` are a central concept within ChRIS and can be accessed via several subcommands in `chili`
+
+* `plugin`
+* `plugins`
+* `pluginMetas`
+* `computesofplugin`
+* `instancesofplugin`
+* `parametersofplugin`
+
+== how to use
+
+See various document chapters for information and examples on each.

package/docs/21_pluginReadme.adoc

@@ -0,0 +1,123 @@
+= plugin readme 
+
+== intro
+
+Each plugin should have a corresponding `README` file that the developer created documenting important information. This `README` can exist in various formats, but is typically in github markdown syntax and unsurprisingly is assumed to be accessible on github.
+
+The `plugin` subcommand can be used to return the `readme` of a given plugin. It operates by examining the `url` of the plugin json representation and searching in that location for a `README` file in various formats.
+
+== how to use
+
+Various use cases exist in which to collect and print this `readme` information
+
+=== getting a list of available plugins
+
+A tabulated list of plugins is useful, and can be achieved with:
+
+```
+chili plugins list --fields id,name,version,title --page 500
+```
+
+the `--page 500` is a "hack" to collect all the plugins in one list. Adjust as necessary.
+
+Typically, such a call will return
+
+```
+   ____ _     ___ _     ___ 
+  / ___| |__ |_ _| |   |_ _|
+ | |   | '_ \ | || |    | | 
+ | |___| | | || || |___ | | 
+  \____|_| |_|___|_____|___|
+                            
+The ChRIS Interactive Line Interface
+┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│                                                         plugins                                                          │
+├────┬────────────────────┬─────────┬──────────────────────────────────────────────────────────────────────────────────────┤
+│ id │ name               │ version │ title                                                                                │
+├────┼────────────────────┼─────────┼──────────────────────────────────────────────────────────────────────────────────────┤
+│ 16 │ pl-pfdorun         │ 2.2.14  │ A ChRIS plugin app                                                                   │
+│ 14 │ pl-shexec          │ 1.2.16  │ Execute shell-type commands across input spaces                                      │
+│ 12 │ pl-dcm2mha_cnvtr   │ 1.2.24  │ A ChRIS plugin app to convert dcm files to mha and vice-versa                        │
+│ 11 │ pl-simpledsapp     │ 2.1.3   │ Simple chris ds app                                                                  │
+│ 10 │ pl-bulk-rename     │ 0.1.2   │ Bulk Rename                                                                          │
+│  9 │ pl-markimg         │ 1.5.8   │ An app to mark landmark points and lines on an input image                           │
+│ 15 │ pl-lld_inference   │ 2.2.10  │ A ChRIS plugin that runs an inference model to predict landmark points on leg images │
+│  8 │ pl-lld_inference   │ 2.2.11  │ A ChRIS plugin that runs an inference model to predict landmark points on leg images │
+│  7 │ pl-neurofiles-push │ 0.2.0   │ Push files into /neuro                                                               │
+│  5 │ pl-csv2json        │ 1.2.4   │ An app to convert CSV generated from pl-lld_inference to a JSON representation       │
+│ 17 │ pl-dylld           │ 4.4.40  │ Leg-Length Discrepency - Dynamic Compute Flow                                        │
+│  4 │ pl-dylld           │ 4.4.42  │ Leg-Length Discrepency - Dynamic Compute Flow                                        │
+│  3 │ pl-dicommake       │ 2.3.6   │ DICOM image make                                                                     │
+│  2 │ pl-lld_chxr        │ 1.1.2   │ A ChRIS plugin to analyze the result produced by an LLD analysis                     │
+│  1 │ pl-dicom_dirsend   │ 1.1.6   │ A ChRIS plugin to send DICOMs to a remote PACS store                                 │
+│ 13 │ pl-dircopy         │ 2.1.2   │ A ChRIS fs app to copy obj storage directories                                       │
+│  6 │ pl-topologicalcopy │ 1.0.2   │ A ChRIS ts app to copy filtered output dirs from a list of plugin instances          │
+└────┴────────────────────┴─────────┴──────────────────────────────────────────────────────────────────────────────────────┘
+
+
+```
+
+
+=== simplest case
+
+Every element within CUBE has a corresponding unique ID, and plugins are of course no exception. The simplest mechanism to print the readme is to use the `plugin` subcommand with a given plugin id in the pattern
+
+```
+plugin readme <id>
+```
+
+So, given the table above, to get the readme of the `pl-shexec` we note its id is `14`, and thus:
+
+```
+chili plugin readme 14
+```
+
+=== using a searchable
+
+A more intuitive manner is to use a `--search <searchable>` (see elsewhere for full description of `<searchables>`). This is typically useful if the name of the plugin is known and it is simply more convenient to use this explicitly:
+
+```
+chili plugin readme --search "name: pl-shexec"
+```
+
+In fact, the intial command using the plugin <id> is equivalent to:
+
+```
+chili plugin readme --search "id: 14"
+```
+
+Note the quotes in the above are for readability and to protect the embedded space. One could just as easily have
+
+
+```
+chili plugin readme --search name:pl-shexec
+chili plugin readme --search id:14
+```
+
+=== using a context specification
+
+A final mechanism for specifying the plugin to process, is to use a _context_ construction. Here, we specify the plugin as a _context_ and call the `readme` subcommand without any arguments:
+
+```
+chili plugin=14 plugin readme
+```
+
+Since the plugin context is also a _searchable_ we can equally do
+
+```
+chili plugin=name:pl-shexec plugin readme
+```
+
+and even
+
+```
+chili plugin=name:pl-dylld,version=4.4.40 plugin readme
+```
+
+Take note that using a _context_ specification *will update the plugin context that persists across subsequent operations*.
+
+== final notes
+
+A successful readme call is dependent on a correct/complete set of meta data for a plugin. Careless developers might not specifiy the "url" in their plugin code meta parameters, in which case no readme lookup can occur. Should this be the case, a friendly PR nudge to the plugin developer might be needed.
+
+_-30-_

package/docs/30_filesystem.adoc

@@ -0,0 +1,93 @@
+= filesystem interaction
+
+== intro
+
+While `chili` is great at managing abstract things like "feeds" and "plugins," sometimes you just want to look at your files. You want to know what data you have, where it is, and how to organize it.
+
+We are building a suite of commands called `chefs` (Chris Experimental File System) to give you a familiar, shell-like way to browse your ChRIS storage.
+
+== the chefs ls command
+
+If you have ever used a Linux or Mac terminal, you know the `ls` command. It lists files. We have brought this same convenience to `chili`.
+
+```bash
+chili chefs ls
+```
+
+This command looks at your current ChRIS folder context (remember _context_?) and lists everything in it.
+
+=== what you will see
+
+The output is designed to be clean and readable, using colors to help you distinguish item types:
+
+|===
+| Type | Color | Example
+
+| **Directory**
+| **Bold Blue**
+| `data/`
+
+| **Link**
+| **Cyan**
+| `link -> target`
+
+| **File**
+| Standard
+| `image.dcm`
+|===
+
+It organizes the names into neat columns, so even if you have hundreds of files, you can scan them easily.
+
+=== looking elsewhere
+
+You aren't limited to just looking at your current directory. You can peek into any path you have access to:
+
+```bash
+chili chefs ls /home/jane.doe/uploads
+```
+
+This will list the contents of that specific folder without changing your current location.
+
+== moving around
+
+You can also change your current working directory using the `cd` command, just like in a real shell:
+
+```bash
+chili chefs cd /home/jane.doe/analysis
+```
+
+And if you ever get lost, you can ask "where am I?" with:
+
+```bash
+chili chefs pwd
+```
+
+== making things
+
+Just like a real filesystem, you can create new directories and files.
+
+=== creating folders
+
+Need a new place to store your data? Use `mkdir`:
+
+```bash
+chili chefs mkdir new_project
+```
+
+This creates a folder named `new_project` in your current directory. You can also create folders deeper in the hierarchy:
+
+```bash
+chili chefs mkdir /home/jane.doe/analysis/run_01
+```
+
+=== touching files
+
+Sometimes you just need an empty file, maybe as a placeholder or a signal. Use `touch`:
+
+```bash
+chili chefs touch empty_file.txt
+```
+
+This creates a zero-byte file at the specified path. If you need to create a file with content, check out the specialized file operations.
+
+These tools are designed to make the remote ChRIS filesystem feel as comfortable and accessible as the one on your own laptop.

package/docs/31_fileops.adoc

@@ -0,0 +1,51 @@
+= advanced file operations
+
+== intro
+
+While `chefs` gives you the basics (`ls`, `touch`), sometimes you need to do more with files. You might want to create a file with specific text, upload a local file to ChRIS, or peek at what's inside a file stored on the server.
+
+For these tasks, `chili` provides the `file` command suite.
+
+== creating files with content
+
+The `chefs touch` command is great for empty files, but what if you want to write a note or a script directly?
+
+```bash
+chili file create notes.txt --content "meeting at 2pm"
+```
+
+This creates `notes.txt` in your current folder and fills it with your text.
+
+=== specifying paths
+
+You aren't stuck in your current directory. You can tell `chili` exactly where to put the file:
+
+```bash
+chili file create --name README.md --path /home/jane.doe/project --content "# Project X"
+```
+
+Or, just include the path in the filename:
+
+```bash
+chili file create /home/jane.doe/project/README.md --content "# Project X"
+```
+
+== uploading from local
+
+This is where it gets powerful. You can "create" a file in ChRIS by reading from a file on your own computer.
+
+```bash
+chili file create data.csv --from-file ./local_data.csv
+```
+
+This takes `./local_data.csv` from your laptop and creates `data.csv` on the ChRIS server. It's a quick and easy way to get single files into the system.
+
+== viewing files
+
+Want to see what's in a file without downloading it?
+
+```bash
+chili file view results.json
+```
+
+This prints the content of the file directly to your terminal. It's perfect for checking configuration files, logs, or small datasets.