whistle.script 1.2.1

package/.editorconfig

@@ -0,0 +1,18 @@
+# http://editorconfig.org
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+indent_size = 2
+indent_style = space
+insert_final_newline = true
+max_line_length = 80
+trim_trailing_whitespace = true
+
+[*.md]
+max_line_length = 0
+trim_trailing_whitespace = false
+
+[COMMIT_EDITMSG]
+max_line_length = 0

package/.eslintrc

@@ -0,0 +1 @@
+{ "extends": "eslint-config-imweb" }

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 avenwu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,264 @@
+# whistle.script
+whistle.script为[whistle](https://github.com/avwo/whistle)的一个扩展脚本插件，可以直接在界面上引用全局安装的Node模块及Node的内容模块编写脚本操作请求及其响应，所有正常Node程序可以实现的功能，都可以通过该插件实现，包括：
+
+1. HTTP[s]:
+   - 动态设置[whistle规则](https://avwo.github.io/whistle/rules/)
+   - 拦截请求响应
+   - 控制请求响应速度
+   - 修改请求url、请求方法、请求头、请求内容
+   - 修改响应状态码、响应头、响应内容
+   - 在插件界面的Console上显示脚本程序 `console.xxx` 的内容，如果可以打印响应的内容或调试信息等
+2. WebSocket:
+   - 动态设置[whistle规则](https://avwo.github.io/whistle/rules/)
+   - 拦截请求响应
+   - 修改发送或收到的数据
+   - 直接向WebSocket客户端或服务端发送数据
+   - 在插件界面的Console上显示脚本程序 `console.xxx` 的内容，如果可以打印发送和接收到的数据或调试信息等，从而通过该插件可以直接查看WebSocket的数据
+3. Tunnel: 基本功能同WebSocket，可以用来直接操作Socket请求，如Protobuf协议的请求等
+
+# 安装
+
+1. 安装Node: [官网下载安装最新版本(LTS和Stable都可以)](https://nodejs.org/)
+2. 安装最新版的[whistle](https://github.com/avwo/whistle)。
+
+		npm install -g whistle
+		
+		# Mac、Linux用户可能需要加sudo
+		sudo npm install -g whistle
+
+3. 安装script插件:
+
+		w2 i whistle.script
+
+# 使用
+
+打开script插件的界面，创建一个名字为 `test` 的脚本:
+
+1. 可以通过 `Plugins->Home->script`打开或右键并选择 `在新标签页中打开` 
+2. 直接访问 [http://local.whistlejs.com/plugin.script](http://local.whistlejs.com/plugin.script/)
+
+ ![whistle.script界面](https://user-images.githubusercontent.com/11450939/126302159-0c533ea7-3bc0-484a-bd30-698d5a7881df.gif)
+
+#### 设置规则
+
+1. 设置HTTP或HTTPs请求的[whistle规则](https://avwo.github.io/whistle/rules/)(操作HTTPs需要[开启HTTPs拦截](https://avwo.github.io/whistle/webui/https.html))
+
+	在界面中的`test` 脚本输入(也可以在其它编辑器编辑后再copy进来):
+	
+		exports.handleRequestRules = (ctx) => {
+			// ctx.fullUrl 可以获取请求url
+			// ctx.headers 可以获取请求头
+			// ctx.options 里面包含一些特殊的请求头字段，分别可以获取一些额外信息，如请求方法、设置的规则等
+			ctx.rules = ['www.qq.com file://{test.html}'];
+		 	ctx.values = { 'test.html': 'Hello world.' };
+		};
+	
+	Note: 如果里面包含一些异步方法可以采用 async 函数，即：`exports.handleRequestRules = async () => {}`
+	
+	在whistle的Rules配置界面上输入规则:
+	
+		whistle.script://test www.ifeng.com www.qq.com www.baidu.com echo.websocket.org
+	
+	分别访问[http://www.ifeng.com](http://www.ifeng.com)和[http://www.qq.com](http://www.qq.com)，前者可以正常访问，后者输出 `Hello world.`。
+	
+	具体效果见图：[demo1](https://user-images.githubusercontent.com/11450939/126302225-2598772c-d6a3-45e3-97d6-685fbed1ba37.gif)
+	
+	如果需要通过配置给脚本传递一些额外参数，可以如下配置(注意中间不能有空格):
+	
+		whistle.script://test(a,b,c) www.ifeng.com www.qq.com www.baidu.com echo.websocket.org
+	
+	可以在脚本中通过 `process.args` 获取:
+	
+		exports.handleRequestRules = (ctx) => {
+		  console.log(process.args); // output: ["a", "b", "c"]
+		  ctx.rules = ['www.qq.com file://{test.html}'];
+		  ctx.values = { 'test.html': 'Hello world.' };
+		};
+
+2. 设置WebSocket请求的规则(需要[开启HTTPs拦截](https://avwo.github.io/whistle/webui/https.html)):
+
+		exports.handleWebSocketRules = (ctx) => {
+		  // ctx.fullUrl 可以获取请求url
+		  // ctx.headers 可以获取请求头
+		  // ctx.options 里面包含一些特殊的请求头字段，分别可以获取一些额外信息，如请求方法、设置的规则等
+		  this.rules = '127.0.0.1 echo.websocket.org';
+		};
+
+	接下来的操作同上。
+
+3. 设置Tunnel请求的规则(要测试可以暂时[关闭HTTPs拦截](https://avwo.github.io/whistle/webui/https.html)):
+
+		exports.handleTunnel = (ctx) => {
+		  // ctx.fullUrl 可以获取请求url
+		  // ctx.headers 可以获取请求头
+		  // ctx.options 里面包含一些特殊的请求头字段，分别可以获取一些额外信息，如请求方法、设置的规则等
+		  this.rules = '127.0.0.1 www.baidu.com';
+		};
+
+
+	接下来的操作同上。
+
+#### 操作请求
+
+1. 操作HTTP或HTTPs请求(操作HTTPs需要[开启HTTPs拦截](https://avwo.github.io/whistle/webui/https.html))
+	``` js
+	exports.handleRequest = async (ctx, request) => {
+		// ctx.fullUrl 可以获取请求url
+		// ctx.headers 可以获取请求头
+		// ctx.options 里面包含一些特殊的请求头字段，分别可以获取一些额外信息，如请设置的规则等
+		// ctx.method 获取和设置请求方法
+		// ctx.req
+		// ctx.res
+		const { req, res } = ctx;
+		const client = request((svrRes) => {
+			res.writeHead(svrRes.statusCode, svrRes.headers);
+			svrRes.pipe(res);
+			// try {
+				// const body = await ctx.getStreamBuffer(svrRes);
+				// delete svrRes.headers['content-encoding'];
+				// res.writeHead(svrRes.statusCode, svrRes.headers);
+				// res.end(body);
+			// } catch (err) {}
+		});
+		req.pipe(client);
+	};
+	```
+	在whistle的Rules配置界面上输入规则:
+	``` txt
+	# 这里不能用whistle.script，否则请求不会转发到handleRequest
+	# whistle.script只会执行handleXxxRules
+	# 你也可以通过在handleXxxRules里面设置 script://test(a,b,c)，实现转发
+	script://test www.ifeng.com www.qq.com www.baidu.com echo.websocket.org
+	```
+	分别访问[http://www.ifeng.com](http://www.ifeng.com)和[http://www.qq.com](http://www.qq.com)，可以在script的界面中的Consle看到打印出来的请求的url、响应状态吗和头部。
+
+	具体效果见图：[demo2](https://user-images.githubusercontent.com/11450939/126302210-e3aa0b56-9001-4e03-83c8-8986d8f544ff.gif)
+
+	需要在配置中带上参数，可以参考上面的规则设置
+2. 操作WebSocket请求(需要[开启HTTPs拦截](https://avwo.github.io/whistle/webui/https.html))
+	``` js
+	exports.handleWebSocket = async (socket, connect) => {
+		// 与服务器建立连接
+		const svrSocket = await connect();
+		// 客户端 pong 服务端
+		socket.on('pong', (data) => {
+			svrSocket.pong(data);
+		});
+		// 客户端 ping 服务pong 端
+		socket.on('ping', (data) => {
+			svrSocket.ping(data);
+		});
+		// 服务端 ping 客户端
+		svrSocket.on('ping', (data) => {
+			socket.ping(data);
+		});
+		// 服务端 pong 客户端
+		svrSocket.on('pong', (data) => {
+			socket.pong(data);
+		});
+		// 正常断开 WebSocket 连接
+		socket.on('disconnect', (code, message, opts) => {
+			console.log(code, 'client disconnect');
+			svrSocket.disconnect(code, opts);
+		});
+		// 正常断开 WebSocket 连接
+		svrSocket.on('disconnect', (code, message, opts) => {
+			console.log(code, 'server disconnect');
+			socket.disconnect(code, opts);
+		});
+		// 获取客户端解析后的帧数据
+		socket.on('message', (data, opts) => {
+			console.log(data, 'client data');
+			svrSocket.send(data, opts);
+		});
+		// 获取服务端解析后的帧数据
+		svrSocket.on('message', (data, opts) => {
+			console.log(data, 'server data');
+			socket.send(data, opts);
+		});
+	};
+
+	```
+
+		whistle规则配置同上，访问[https://www.websocket.org/echo.html](https://www.websocket.org/echo.html)，点击下面的connect按钮及send按钮，可以如下效果：[demo3](https://user-images.githubusercontent.com/11450939/126302243-26c8b4af-851c-4b00-87b9-3286e9e67251.gif)
+3. 操作Tunnel请求
+	``` js
+	exports.handleTunnel = async (socket, connect) => {
+		const svrSocket = await connect();
+		socket.pipe(svrSocket).pipe(socket);
+	};
+	```
+	whistle规则配置同上
+
+4. 鉴权
+插件 `v1.2.0` 版本开始支持自定义鉴权方法（要求 Whistle 版本 >= `v2.7.16`）：
+``` js
+exports.auth = async (req, options) => {
+	// 给请求添加自定义头，必须与 `x-whistle-` 开头
+	// 这样可以在插件的其他 hook 里面获取到该请求头（除了 http 请求的 reqRead 钩子）
+	req.setHeader('x-whistle-test', '1111111111');
+	// return false; // 直接返回 403
+};
+```
+
+5. pipe
+插件 `v1.2.1` 版本开始支持自定义 pipe 方法：
+``` js
+
+exports.handleReqRead = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleReqWrite = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleResRead = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleResWrite = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleWsReqRead = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleWsReqWrite = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleWsResRead = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleWsResWrite = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleTunnelReqRead = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleTunnelReqWrite = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleTunnelResRead = (req, res, options) => {
+  req.pipe(res);
+};
+
+exports.handleTunnelResWrite = (req, res, options) => {
+  req.pipe(res);
+};
+
+```
+
+# 如何引入第三方模块
+使用绝对路径引入，如假设你的模块安装路径为 `/Users/test/node_modules/xxx`，则可以在脚本里面通过 `require('/Users/test/node_modules/xxx')` 引入。
+
+# License
+
+[MIT](https://github.com/whistle-plugins/whistle.script/blob/master/LICENSE)
+

package/index.js

@@ -0,0 +1 @@
+module.exports = require('./lib');

package/initialize.js

@@ -0,0 +1,26 @@
+
+
+module.exports = ({ debugMode }) => {
+  const logger = require('./lib/logger');
+  const LEVELS = ['fatal', 'error', 'warn', 'info', 'debug', 'trace'];
+
+  /* eslint-disable no-console */
+  const originalLog = console.log;
+  console.log = (...args) => {
+    logger.log(args, 'log');
+    if (debugMode) {
+      originalLog.apply(console, args);
+    }
+  };
+
+  LEVELS.forEach((level) => {
+    const originalFn = console[level];
+    console[level] = (...args) => {
+      logger.log(args, level);
+      if (debugMode) {
+        originalFn.apply(console, args);
+      }
+    };
+  });
+
+};

package/lib/auth.js

@@ -0,0 +1,39 @@
+const scripts = require('./scripts');
+const { getRemoteUrl, getContext, getFn, noop, request, AUTH_URL, STATS_URL, DATA_URL, isRemote } = require('./util');
+
+module.exports = async (req, options) => {
+  if (!isRemote(req)) {
+    const ctx = getContext(req);
+    const { auth, verify } = scripts.getHandler(ctx);
+    const check = getFn(auth, verify);
+    return check && check(req, options);
+  }
+  const authUrl = getRemoteUrl(req, AUTH_URL);
+  if (authUrl) {
+    const { auth, headers } = await request(authUrl, req.headers);
+    if (auth === false) {
+      return false;
+    }
+    if (headers) {
+      Object.keys(headers).forEach((key) => {
+        req.set(key, headers[key]);
+      });
+    }
+  }
+  const statsUrl = getRemoteUrl(req, STATS_URL);
+  const dataUrl = getRemoteUrl(req, DATA_URL);
+  if (statsUrl) {
+    req.getReqSession((session) => {
+      if (session) {
+        request(statsUrl, req.headers, session).then(noop);
+      }
+    });
+  }
+  if (dataUrl) {
+    req.getSession((session) => {
+      if (session) {
+        request(dataUrl, req.headers, session).then(noop);
+      }
+    });
+  }
+};

package/lib/dataSource.js

@@ -0,0 +1,6 @@
+const { EventEmitter } = require('events');
+
+const dataSource = new EventEmitter();
+dataSource.setMaxListeners(1000);
+
+module.exports = dataSource;

package/lib/index.js

@@ -0,0 +1,22 @@
+const handlePipe = require('./pipe');
+
+exports.auth = require('./auth');
+exports.uiServer = require('./uiServer');
+exports.rulesServer = require('./rulesServer');
+exports.resRulesServer = require('./resRulesServer');
+exports.tunnelRulesServer = require('./tunnelRulesServer');
+exports.server = require('./server');
+exports.tunnelServer = require('./tunnelServer');
+
+exports.reqRead = handlePipe('reqRead');
+exports.reqWrite = handlePipe('reqWrite');
+exports.resRead = handlePipe('resRead');
+exports.resWrite = handlePipe('resWrite');
+exports.wsReqRead = handlePipe('wsReqRead');
+exports.wsReqWrite = handlePipe('wsReqWrite');
+exports.wsResRead = handlePipe('wsResRead');
+exports.wsResWrite = handlePipe('wsResWrite');
+exports.tunnelReqRead = handlePipe('tunnelReqRead');
+exports.tunnelReqWrite = handlePipe('tunnelReqWrite');
+exports.tunnelResRead = handlePipe('tunnelResRead');
+exports.tunnelResWrite = handlePipe('tunnelResWrite');

package/lib/logger.js

@@ -0,0 +1,56 @@
+const util = require('util');
+
+const MAX_COUNT = 666;
+const MIN_COUNT = 600;
+const COUNT = 60;
+let logs = [];
+let index = 0;
+
+const leftPad = (num) => {
+  if (num > 99) {
+    return num;
+  }
+  if (num > 9) {
+    return `0${num}`;
+  }
+  return `00${num}`;
+};
+
+const getId = () => {
+  if (index > 999) {
+    index = 0;
+  }
+  return `${Date.now()}-${leftPad(index++)}`;
+};
+
+const log = (args, level) => {
+  const msg = [];
+  for (let i = 0, len = args.length; i < len; i++) {
+    const arg = args[i];
+    if (arg instanceof Error) {
+      msg.push(arg.stack || arg);
+    } else {
+      msg.push(arg);
+    }
+  }
+  logs.push({
+    level,
+    id: getId(),
+    msg: util.format.apply(null, msg),
+  });
+  const len = logs.length;
+  if (len > MAX_COUNT) {
+    logs = logs.slice(0, len - MIN_COUNT);
+  }
+};
+
+exports.log = log;
+exports.getLogs = (id) => {
+  id = id || String(Date.now() - 5000);
+  for (let i = 0, len = logs.length; i < len; i++) {
+    if (logs[i].id > id) {
+      return logs.slice(i, i + COUNT);
+    }
+  }
+  return [];
+};

package/lib/pipe.js

@@ -0,0 +1,32 @@
+const iconv = require('iconv-lite');
+const scripts = require('./scripts');
+const util = require('./util');
+
+const getHandlerName = (name) => {
+  return `handle${name[0].toUpperCase()}${name.substring(1)}`;
+};
+
+module.exports = (name) => {
+  const eventName = name[0] === 'r' ? 'request' : 'connect';
+  return (server, options) => {
+    options = Object.assign({
+      getCharset: util.getCharset,
+      isText: util.isText,
+      iconv,
+    }, options);
+    server.on(eventName, async (req, res) => {
+      const oReq = req.originalReq;
+      oReq.ruleValue = oReq.ruleValue || oReq.pipeValue;
+      const handleRequest = scripts.getHandler({ req })[getHandlerName(name)];
+      if (!util.isFunction(handleRequest)) {
+        return req.pipe(res);
+      }
+      try {
+        await handleRequest(req, res, options);
+      } catch (err) {
+        req.emit('error', err);
+        console.error(err); // eslint-disable-line
+      }
+    });
+  };
+};

package/lib/resRulesServer.js

@@ -0,0 +1,26 @@
+const Koa = require('koa');
+const util = require('./util');
+const scripts = require('./scripts');
+
+module.exports = (server, options) => {
+  const app = new Koa();
+  app.use(async (ctx) => {
+    const { req } = ctx;
+    const rulesUrl = req.sessionStorage.get(util.RES_RULES_URL);
+    if (rulesUrl != null) {
+      req.sessionStorage.remove(util.RES_RULES_URL);
+      if (rulesUrl) {
+        const result = await util.request(rulesUrl, req.headers);
+        ctx.body = util.formateRules(result);
+      }
+      return;
+    }
+    util.setupContext(ctx, options);
+    const { handleResponseRules } = scripts.getHandler(ctx);
+    if (util.isFunction(handleResponseRules)) {
+      await handleResponseRules(ctx);
+      util.responseRules(ctx);
+    }
+  });
+  server.on('request', app.callback());
+};

package/lib/rulesServer.js

@@ -0,0 +1,41 @@
+const Koa = require('koa');
+const util = require('./util');
+const scripts = require('./scripts');
+
+module.exports = (server, options) => {
+  const app = new Koa();
+  app.use(async (ctx) => {
+    const { req } = ctx;
+    if (util.isRemote(req)) {
+      const rulesUrl = util.getRemoteUrl(req, util.REQ_RULES_URL);
+      const resRulesUrl = util.getRemoteUrl(req, util.RES_RULES_URL);
+      req.sessionStorage.set(util.RES_RULES_URL, resRulesUrl || '');
+      if (rulesUrl) {
+        const result = await util.request(rulesUrl, req.headers);
+        ctx.body = util.formateRules(result);
+      }
+      return;
+    }
+    util.setupContext(ctx, options);
+    const {
+      handleWebSocketRules,
+      handleWebsocketRules,
+      handleRequestRules,
+    } = scripts.getHandler(ctx);
+    let handleReqRules;
+    if (/^ws/.test(ctx.fullUrl)) {
+      if (util.isFunction(handleWebSocketRules)) {
+        handleReqRules = handleWebSocketRules;
+      } else if (util.isFunction(handleWebsocketRules)) {
+        handleReqRules = handleWebsocketRules;
+      }
+    } else if (util.isFunction(handleRequestRules)) {
+      handleReqRules = handleRequestRules;
+    }
+    if (handleReqRules) {
+      await handleReqRules(ctx);
+      util.responseRules(ctx);
+    }
+  });
+  server.on('request', app.callback());
+};

package/lib/scripts.js

@@ -0,0 +1,110 @@
+const vm = require('vm');
+const iconv = require('iconv-lite');
+
+const GLOBAL_VARS = [
+  'process',
+  'Buffer',
+  'clearImmediate',
+  'clearInterval',
+  'clearTimeout',
+  'setImmediate',
+  'setInterval',
+  'setTimeout',
+  'console',
+  'module',
+  'require',
+];
+/* eslint-disable no-console */
+const TIMEOUT = 600;
+const RULE_VALUE_RE = /^([\w\-.]+)(?:\((.+)\))?.*$/;
+const scripts = {};
+const VM_OPTIONS = {
+  displayErrors: false,
+  timeout: TIMEOUT,
+};
+let CONTEXT = vm.createContext();
+
+setInterval(() => {
+  CONTEXT = vm.createContext();
+}, 30000);
+
+const getScript = (code, name) => {
+  code = code && code.trim();
+  if (!code) {
+    return;
+  }
+  code = `const d = require('domain').create();
+          d.on('error', console.error);
+          d.run(() => {
+            ${code}
+          });`;
+  try {
+    return new vm.Script(`(function(){\n${code}\n})()`, { filename: name, timeout: TIMEOUT });
+  } catch (err) {
+    console.error(err);
+  }
+};
+
+const execScript = (script, value) => {
+  if (!script) {
+    return {};
+  }
+  GLOBAL_VARS.forEach((key) => {
+    CONTEXT[key] = global[key];
+  });
+  CONTEXT.require = require;
+  CONTEXT.iconv = iconv;
+  CONTEXT.process = {
+    args: value ? value.split(',') : [],
+  };
+  CONTEXT.exports = {};
+  CONTEXT.module = { exports: CONTEXT.exports };
+  try {
+    script.runInContext(CONTEXT, VM_OPTIONS);
+    return CONTEXT.module.exports || {};
+  } catch (err) {
+    console.error(err);
+  } finally {
+    Object.keys(CONTEXT).forEach((key) => {
+      if (GLOBAL_VARS.indexOf(key) === -1) {
+        delete CONTEXT[key];
+      }
+    });
+  }
+  return {};
+};
+
+const parseRuleValue = (ctx) => {
+  const { ruleValue } = ctx.req.originalReq;
+  if (!RULE_VALUE_RE.test(ruleValue)) {
+    return '';
+  }
+  return {
+    name: RegExp.$1,
+    value: RegExp.$2,
+  };
+};
+
+exports.get = (name) => {
+  return scripts[name];
+};
+
+exports.getHandler = (ctx) => {
+  const { name, value } = parseRuleValue(ctx);
+  return execScript(scripts[name], value);
+};
+
+const set = (name, text) => {
+  scripts[name] = typeof text === 'string' ? getScript(text) : text;
+};
+exports.set = set;
+
+exports.remove = (name) => {
+  delete scripts[name];
+};
+
+exports.load = (storage) => {
+  storage.getFileList().forEach(({ name, data }) => {
+    set(name, data);
+  });
+};