tune-basic-toolset 0.1.0

package/README.md

@@ -0,0 +1,517 @@
+# Tune basic toolset
+Basic toolset for [Tune](https://github.com/iovdin/tune).
+
+##### Index
+- [Setup](#setup)
+  - [Text Editor](#text-editor)
+  - [JavaScript Project](#javascript-project)
+- [Tools](#tools)
+  - [rf](#rf) read file
+  - [wf](#wf) write file
+  - [patch](#patch) patch file
+  - [append](#append) append to file
+  - [sh](#sh) execute shell command
+  - [osa](#osa) manage reminders/notes/calendar (AppleScript/macOS)
+  - [jina_r](#jina_r) fetch webpage content
+  - [turn](#turn) turn based agent
+  - [list](#list) keep list of tasks todo (loops for LLM)
+  - [py](#py) run python code
+  - [js](#js) run javascript code
+  - [message](#message) talk to another chat/agent
+- [Processors](#processors)
+  - [shp](#shp) include shell command output
+  - [init](#init) set initial value
+  - [json_format](#json_format) make LLM respond with JSON
+  - [log](#log) save LLM payload
+  - [mock](#mock) set variables inline
+  - [linenum](#linenum) prepend line numbers
+  - [text](#text) convert any variable to text variable
+  - [resolve](#resolve) resolve a variable
+  - [prop](#prop) set additional properties of LLM
+  - [head](#head) take first N lines of a file
+  - [tail](#tail) take last N lines of a file or LLM payload
+  - [slice](#slice) take lines from <start> to <finish> of a file
+
+
+## Setup
+### Text Editor
+
+Install in your `~/.tune` folder:
+
+```bash
+cd ~/.tune
+npm install tune-basic-toolset
+```
+
+Add to `~/.tune/default.ctx.js`:
+
+```javascript
+const basics = require('tune-basic-toolset')
+
+module.exports = [
+    ...
+    basics()
+    ...
+]
+```
+
+### JavaScript Project
+
+```bash
+npm install tune-basic-toolset tune-sdk
+```
+
+```javascript
+const tune = require('tune-sdk')
+const basics = require('tune-basic-toolset')
+
+const ctx = tune.makeContext(
+    basics({ expose: ["rf", "wf"], mount: "tools" })
+)
+```
+
+## Tools
+[Tools](https://iovdin.github.io/tune/template-language/tools) is a function that llm can run on your local machine or server
+
+### `rf`
+Read file
+```chat
+user: @rf
+can you read README.md?
+tool_call: rf {"filename":"README.md"}
+tool_result: 
+@README.md
+```
+It accepts an optional `linenum` parameter that prepends line numbers to the file (useful for patching).
+
+### `wf`
+Write to a file
+```chat
+user: @wf 
+make a hello world javascript
+tool_call: wf {"filename":"helloWorld.js"}
+console.log('Hello, World!');
+tool_result:
+written
+```
+
+### `patch`
+```chat
+user: @patch
+translate "hello world" in helloWorld.js  to dutch
+tool_call: patch {"filename":"helloWorld.js"}
+<<<<<<< ORIGINAL
+console.log('Hello, World!');
+=======
+console.log('Hallo, Wereld!');
+>>>>>>> UPDATED
+tool_result:
+patched
+```
+
+
+### `append`
+```chat
+user: @append
+Add a todo that i want to buy a car and go to the moon
+tool_call: append {"filename":"todo.md"}
+- buy a car 
+- fly to the moon
+tool_result:
+appended
+```
+
+### `sh`
+Execute shell command
+```chat
+user: @sh
+find with ripgrep where echo is used
+tool_call: sh
+rg 'echo' ./
+tool_result: 
+./README.md:    echo: "You are echo, you print everything back",
+./README.md:  const text = "s: \@echo\nu: hello world";
+./tools/echo.txt:you are echo, you print everything back
+./tools/README.md:* `echo.txt` - to debug variable expansions and context
+```
+
+### `osa`
+AppleScript tool, manage reminders, notes, calendar etc on osx
+```chat
+system: @osa @gpt-4.1
+you're personal assistant with access to osx apps
+like reminder, calendar etc
+
+user: 
+can you remind me in 2 hours that i have meeting with jack
+
+assistant:
+
+tool_call: osa
+tell application "Reminders" to make new reminder with properties {name:"Meeting with Jack", remind me date:(current date) + 2 * hours}
+tool_result:
+reminder id x-apple-reminder://849EA642-B7E0-4E16-94AA-B50FF1BC282D
+```
+
+### `jina_r`
+Fetch webpage content with [jina reader](https://jina.ai/reader) api.
+Works without an API key up to a limited rate. Set JINA_KEY in .env for higher rate limits.
+
+```chat
+user: @jina_r
+take https://iovdin.github.io/tune/ page content
+and put it into `home` directory
+then make short summary
+
+tool_call: jina_r {"url":"https://iovdin.github.io/tune/","filename":"home/tune_page_content.md"}
+
+tool_result: 
+@home/tune_page_content.md
+
+assistant:
+The content from the "Tune" page has been successfully retrieved and saved into the `home/tune_page_content.md` file. Below is a short summary of the page:
+
+---
+
+**Summary of Tune - Your Everyday LLM Toolkit:**
+
+Tune is a versatile toolkit designed for developers and users to effectively interact with Large Language Models (LLMs). It allows users to chat with AI, build applications, and create AI agents in a simple manner. 
+<cut for brevity>
+```
+
+
+### `turn`
+A way to switch roles when building multistep agents [read more](https://iovdin.github.io/tune/examples/multi-agent) 
+```chat
+system: @turn @gpt-4o
+You're playing 20 questions game.
+You switch turns between 'thinker' and 'player' agent. 
+Current agent stored in agent.txt file
+'player' always plays first
+
+@@agent|init
+assistant: 
+Is it a living thing?
+
+tool_call: turn {"role":"thinker","filename":"agent.txt"}
+tool_result: now it is turn of thinker to reply
+
+assistant: 
+No.
+
+tool_call: turn {"role":"player","filename":"agent.txt"}
+tool_result: now it is turn of player to reply
+
+assistant: 
+Is it something that can be used indoors?
+
+tool_call: turn {"role":"thinker","filename":"agent.txt"}
+tool_result: now it is turn of thinker to reply
+
+assistant: 
+Yes.
+
+...
+```
+
+### `list`
+Keep list of tasks to do
+
+```chat
+system: @list
+You manage todo list in file todo.txt
+
+user: 
+Today i need to refactor xyz
+and then release abc
+
+assistant:
+
+tool_call: list {"filename":"todo.txt"}
+todo - refactor xyz
+todo - release abc
+
+tool_result:
+list updated
+
+user: 
+ok, i've finished refactoring xyz
+
+assistant:
+
+tool_call: list {"filename":"todo.txt"}
+done - refactor xyz
+
+tool_result:
+list updated
+```
+
+
+### `py`
+execute python code
+```chat
+user: @py
+1234 + 4311
+
+tool_call: py
+1234 + 4311
+
+tool_result:
+5545
+```
+
+### `js`
+execute javascript code
+```chat
+user: @js
+1234 + 4311
+
+tool_call: js {"inputType":"commonjs"}
+1234 + 4311
+
+tool_result:
+5545
+
+```
+
+### `message`
+Talk to another chat/agent via tool call. 
+Orchestrate or evaulate other agents/chats.
+```chat
+system: 
+@message
+Your goal is to talk to Groot at `groot.prompt` system prompt
+and try to make him say anything but 'I am Groot' 
+
+tool_call: message {"filename":"groot.chat","system":"groot.prompt"}
+Hello Groot! How are you feeling today?
+
+tool_result: 
+I am Groot!
+
+tool_call: message {"filename":"groot.chat","system":"groot.prompt"}
+What do you think about trees?
+
+tool_result: 
+I am Groot!
+
+tool_call: message {"filename":"groot.chat","system":"groot.prompt"}
+Can you tell me a joke?
+
+tool_result: 
+Sure! Why did the tree go to the dentist?  
+
+Because it had a root canal!
+```
+
+The content of `groot.chat` is then:
+```chat
+system: @@groot.prompt
+user:
+Hello Groot! How are you feeling today?
+
+assistant:
+I am Groot!
+
+user:
+What do you think about trees?
+
+assistant:
+I am Groot!
+
+user:
+Can you tell me a joke?
+
+assistant:
+Sure! Why did the tree go to the dentist?  
+
+Because it had a root canal!
+```
+
+
+## Processors
+[Processors](https://iovdin.github.io/tune/template-language/processors) is a way to modify variable or insert new ones into chat.
+
+### `shp`
+Insert shell command output
+```chat
+system:
+include project file list to system prompt
+@{| shp git ls-files }
+
+include buffer content on osx
+@{| shp pbpaste }
+
+include current date
+@{| shp date }
+
+pipe filename content to shell command
+@{ a.log | shp tail }
+
+@{ a.log | shp grep pattern }
+```
+
+### `init` 
+Set default value for non set variables
+
+```chat
+system:
+@memory|init 
+if memory does not exist the chat will fail
+```
+
+### `json_format`
+Set llm response format to json [read more](https://platform.openai.com/docs/guides/structured-outputs?api-mode=chat).
+
+Without arguments it sets
+```json
+"response_format": {
+  "type": "json_object"
+}
+```
+
+```chat
+system: 
+@{ gpt-4o | json_format }
+please reply in json format:
+{ "message": "Your reply"}
+
+user: 
+hi how are you?
+
+assistant:
+{ "message": "I'm just a virtual assistant, so I don't have feelings, but I'm here and ready to help you! How can I assist you today?" }
+
+```
+
+with argument it sets 
+```json
+"response_format": { 
+    "type": "json_schema", 
+    "json_schema": { "schema": "<contents of the referenced schema file>" }
+} 
+```
+
+```chat
+system: 
+@{ gpt-4o | json_format path/to/schema.json }
+```
+
+### `log` 
+Save LLM payload to a json or chat file, used for debugging
+```chat
+system: 
+@{ gpt-4o | log path/to/log.json }
+@{ gpt-4o | log path/to/log.chat }
+```
+
+### `mock` 
+Set variables inline in chat. 
+```
+system: @{| mock hello=world }
+@echo
+user: 
+@hello
+assistant:
+world
+```
+
+### `linenum`
+Prepend line numbers to a file content. 
+Useful when patching file.
+```chat
+system:
+@echo
+user: 
+@{ helloWorld | linenum }
+assistant:
+1 | console.log('Hello, World!');
+```
+
+### `text`
+Treat special files (`.ctx.js`, `.llm.js`, `.tool.js`)  like text
+```chat
+system: 
+@echo
+
+user: 
+content 
+@rf.tool.mjs
+
+assistant: 
+content
+
+user: 
+content
+@{ rf.tool.mjs | text}
+
+assistant: 
+content
+import { promises as fs } from 'fs';
+import { relative, dirname } from 'path' 
+....
+```
+
+### `resolve`
+Given filename resolve it and include
+
+```chat
+@{ filename | resolve }
+```
+
+see `examples/queryimage` example
+
+### `prop`
+set additional properties for llm
+
+```chat
+system: 
+@{ o3-mini | prop reasoning_effort=low temperature=2.0 }
+```
+
+### `head`
+Take first *N* lines of text from a file or variable. Default is 20 lines.
+
+```chat
+user: 
+@{ filename.txt | head 10 }    # first 10 lines
+```
+
+### `tail`
+Take last *N* lines of text from a file or variable. Default is 20 lines.
+
+```chat
+user: 
+@{ filename.txt | tail 15 }    # last 15 lines
+```
+
+You can limit llm request context with tail like
+```chat
+system: 
+@{ gpt-4.1 | tail 2 }  # take last 2 messages from the chat + system message
+
+user: 
+1
+
+assistant: 
+2
+
+user: 
+3
+
+assistant: 
+4
+```
+
+
+### `slice`
+Extract a range of lines from a file or variable.
+
+```chat
+user: 
+@{ filename.txt | slice 5 15 }      # lines 5 to 15 inclusive
+@{ filename.txt | slice 10 }        # from line 10 to end
+@{ filename.txt | slice -10 -1 }    # last 10 lines
+@{ filename.txt | slice -20 }       # last 20 lines
+@{ filename.txt | slice 1 20 }      # first 20 lines (like head 20)
+```

package/package.json

@@ -0,0 +1,35 @@
+{
+  "name": "tune-basic-toolset",
+  "version": "0.1.0",
+  "description": "Basic toolset for tune",
+  "main": "src/index.js",
+  "files": [
+    "src/",
+    "README.md",
+    "LICENSE"
+  ],
+  "keywords": [
+    "ai",
+    "chat",
+    "tools",
+    "tune"
+  ],
+  "author": "Ilya Ovdin <iovdin@gmail.com>",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/iovdin/tune-basic-toolset"
+  },
+  "license": "MIT",
+  "peerDependencies": {
+    "tune-sdk": "latest",
+    "tune-fs": "latest"
+  },
+  "devDependencies": {
+    "tune-sdk": "latest",
+    "tune-fs": "latest"
+  },
+  "dependencies": {
+    "escodegen": "^2.1.0",
+    "esprima": "^4.0.1"
+  }
+}

package/src/append.schema.json

@@ -0,0 +1,17 @@
+{
+  "description": "Append text to a file",
+  "parameters": {
+    "type": "object",
+    "properties": {
+      "filename": {
+        "type": "string",
+        "description": "Name of the file to append text to"
+      },
+      "text": {
+        "type": "string",
+        "description": "Text content to append to the file"
+      }
+    },
+    "required": ["filename", "text"]
+  }
+}

package/src/append.tool.js

@@ -0,0 +1,5 @@
+module.exports = async function append({filename, text}) {
+  const content = await this.read(filename) || "";
+  await this.write(filename, `${content}\n${text}`);
+  return "done";
+};

package/src/head.proc.js

@@ -0,0 +1,24 @@
+module.exports = async function head(node, args, context) {
+  // Only transform text nodes
+  if (!node || node.type !== 'text') {
+    return node;
+  }
+
+  // Parse the number of lines argument, default is 20
+  const n = parseInt(args, 10);
+  const linesCount = (isNaN(n) || n <= 0) ? 20 : n;
+
+  const newNode = Object.assign({}, node);
+
+  newNode.read = async () => {
+    const text = await node.read();
+    // Split text into lines
+    const lines = text.split(/\r?\n/);
+    // Take first linesCount lines
+    const firstLines = lines.slice(0, linesCount);
+    // Join back into string
+    return firstLines.join('\n');
+  };
+  
+  return newNode;
+};

package/src/index.js

@@ -0,0 +1,6 @@
+const { tools } = require('tune-fs')
+
+// TODO js tool packages include
+module.exports = (...opts) => 
+  tools({ ...opts, path: __dirname })
+

package/src/init.proc.js

@@ -0,0 +1,19 @@
+module.exports = async function init(node, args, ctx) {
+  if (node) {
+    return node;
+  }
+  const content = (args || "").trim();
+  
+  // include file
+  if (content.indexOf("@") === 0) {
+    return {
+      type: "text", 
+      read: async () => this.read(content.replace(/^@{1,2}/, "")) 
+    }
+  }
+  
+  return {
+    type: "text", 
+    read: async () => content 
+  }
+} 

package/src/jina_r.schema.json

@@ -0,0 +1,21 @@
+ {
+  "description": "Fetch content from a specified URL",
+  "parameters": {
+    "type": "object",
+    "properties": {
+      "url": {
+        "type": "string",
+        "description": "The URL of the website to fetch content from returns markdown"
+      },
+      "filename": {
+        "type": "string",
+        "description": "save result to this filename"
+      },
+      "links": {
+        "type": "boolean",
+        "description": "provide links in the end"
+      }
+    },
+    "required": ["url"]
+  }
+}

package/src/jina_r.tool.js

@@ -0,0 +1,27 @@
+const fs = require('fs');
+const path = require('path');
+
+module.exports = async function fetchFromUrl({ url, filename, links }, ctx) {
+  const wurl = `https://r.jina.ai/${url}`;
+  const headers = {};
+
+  const apiKey = await ctx.read("JINA_KEY");
+  if (apiKey) {
+    headers.Authorization = `Bearer ${apiKey}`;
+  }
+
+  if (links) {
+    headers['X-With-Links-Summary'] = true;
+  }
+  
+  const response = await fetch(wurl, { headers } );
+  if (!response.ok) {
+    throw new Error(`Error: ${response.status} ${response.statusText}`);
+  }
+  const res =  await response.text();
+  if (filename) {
+    await ctx.write(filename, res);
+    return `@${filename}`;
+  }
+  return res.replace(/@/g, "\\@");
+};

package/src/js.schema.json

@@ -0,0 +1,19 @@
+ {
+  "description": "Execute a given JavaScript code snippet and return the result",
+  "parameters": {
+    "type": "object",
+    "properties": {
+      "text": {
+        "type": "string",
+        "description": "The JavaScript code to be executed"
+      },
+      "inputType": {
+        "type": "string",
+        "enum": ["module", "commonjs"],
+        "description": "Type of script (module or commonjs), default it commonjs",
+        "default": "commonjs"
+      }
+    },
+    "required": ["text", "inputType"]
+  }
+}