@alwaysai/device-agent 0.0.20 → 0.1.1

package/readme.md

@@ -36,7 +36,7 @@ and add the following line to the end of the file:
 
 On the target device, run:
 
-```
+```bash
 $ curl -fsSL https://artifacts.alwaysai.co/device-agent/install-device-agent.sh | sudo -E bash -
 ```
 
@@ -52,7 +52,7 @@ Provisioning the device performs the following:
 
 Run the following command on the target device to provision it:
 
-```
+```bash
 $ curl -fsSL https://artifacts.alwaysai.co/device-agent/provision.sh | bash -s -- --email <email> --password <password> [--device-name <device_name>]
 ```
 
@@ -63,6 +63,8 @@ Where:
   devices page of the alwaysAI Dashboard. If a device name is not provided, one
   will be generated for you and logged to the console for reference.
 
+**Important note**: If your password contains one or more special characters and you receive an error message that your password does not match your username but you are sure that it is correctly entered, please try enclosing your password with double quotes and re-running the provisioning command. You can also reset your password at https://console.alwaysai.co/dashboard by logging out, navigating to the sign in page, and clicking '`Forgot Password?`'.
+
 Confirm the Device Agent is running with the following command:
 
 ```bash
@@ -74,7 +76,7 @@ $ pm2 list
 └────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘
 ```
 
-To restart the Device Agent run the following command:
+To restart and update the Device Agent run the following command:
 
 ```bash
 $ pm2 restart aai-agent
@@ -91,7 +93,7 @@ Use --update-env to update environment variables
 If you'd like to only provision the device, but not start the Device Agent in
 the background (skip step 3), run with the `--provision-only` flag:
 
-```
+```bash
 $ curl -fsSL https://artifacts.alwaysai.co/device-agent/provision.sh | bash -s -- --email <email> --password <password> [--device-name <device_name>] --provision-only
 ```
 
@@ -116,11 +118,129 @@ project page of the alwaysAI Dashboard as well!
 
 Now you can deploy to your device from the alwaysAI Dashboard.
 
+## Enable Analytics through the alwaysAI Device Agent
+
+### Configure the Device Agent
+You can send information from your device to the alwaysAI cloud securely using
+the Device Agent. These instructions assume you have provisioned your device
+using the default script parameters and have the Device agent running (i.e. the
+provisioning script was not run with the `--provision-only` flag set).
+
+First, the Device Agent must be configured to enable Analytics Pass-through support. Confirm that the `ALWAYSAI_ANALYTICS_PASSTHROUGH` environment variable is set:
+
+```bash
+$ pm2 env 0 | grep ALWAYSAI
+ALWAYSAI_ANALYTICS_PASSTHROUGH: 1
+ALWAYSAI_LOG_TO_CONSOLE: 
+ALWAYSAI_LOG_LEVEL: debug
+ALWAYSAI_DEVICE_AGENT_MODE: cloud
+```
+
+Then confirm the RabbitMQ container is up and running:
+
+```bash
+$ docker ps
+CONTAINER ID   IMAGE           COMMAND                  CREATED          STATUS          PORTS                                                                    NAMES
+596157124a4b   rabbitmq:3.11   "docker-entrypoint.s…"   32 minutes ago   Up 21 minutes   4369/tcp, 5671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:5672->5672/tcp   alwaysAIRabbitMQContainer
+```
+
+### Configure the Application
+
+From the application side, the `ALWAYSAI_CONNECT_TO_DEVICE_AGENT` environment
+variable must be set. There are a few options:
+
+#### In the app source
+
+In your `app.py` file, make sure the top of your import statements looks like
+this:
+
+```python
+import os
+os.environ['ALWAYSAI_CONNECT_TO_DEVICE_AGENT']='1'
+import edgeiq
+```
+You can import any other modules after `edgeiq`, but the other orders must be
+maintained.
+
+#### In the Dockerfile
+
+```bash
+ENV ALWAYSAI_CONNECT_TO_DEVICE_AGENT=1
+```
+
+#### In a docker-compose.yaml
+
+Add the following section to the service for your app:
+
+```bash
+environment:
+  - ALWAYSAI_CONNECT_TO_DEVICE_AGENT=1
+```
+
+### Enable Publishing to Cloud
+
+To enable cloud publishing for your application you can either run
+`aai app enable-cloud-publish`, or update your `alwaysai.app.json` by adding the
+following component in addition to any `models` or `scripts` components:
+
+```json
+"analytics: {
+  "enable_cloud_publish": true
+}
+```
+
+So, a valid `alwaysai.app.json` for publishing analytics might look like this:
+
+```json
+{
+  "scripts": {
+    "start": "python app.py"
+  },
+  "models": {
+    "alwaysai/mobilenet_ssd": 4
+  },
+  "analytics": {
+    "enable_cloud_publish": true
+  }
+}
+```
+
+Finally, add a command to publish analytics to your `app.py`. Whenever this
+command is used, it will publish the contents of the analytics message to the
+cloud -- you can choose to call this every frame, or once every event
+occurrence, however your app is designed. Each core computer vision service has
+it's own `publish_analytics` method, which is called on the instance of the
+class. Or, you can published a JSON-serializable message with
+`edgeiq.publish_analytics()`. For instance, to publish object detection results
+you can use:
+
+```python
+results = obj_detect.detect_objects(frame, confidence_level=.5)
+try:
+    obj_detect.publish_analytics(results, tag=frame_count)
+except edgeiq.PublishError as e:
+    print(e)
+```
+
+Finally, make sure to save all of your changes, and publish your application with
+
+```bash
+$ aai app publish
+```
+
+You can test that analytics are being viewed with `wscat`, using your application's project ID and a secure API key:
+
+```bash
+$ wscat -c "wss://analytics.alwaysai.co?projectId=[PROJECT_ID]&apiKey=[API_KEY]"
+```
+
+Please contact the alwaysAI team if you need a secure API key.
+
 ## The alwaysAI Device Agent Command Line interface
 
 The Device Agent can also be used directly on the device with it's command line interface.
 
-```
+```bash
 $ aai-agent --help
 Usage: aai-agent <subcommand> ...
 
@@ -140,8 +260,6 @@ Subcommands:
    app show-models    : Show the application models
    app add-model      : Add a model to an alwaysAI app
    app remove-model   : Remove a model from an alwaysAI app
-   app replace-models : Replace all models of an alwaysAI app with new models
-   app update-models  : Update all models for an alwaysAI app
    app get-all-envs   : Get environment variables for an application
    app set-env        : Set environment variables for a service
    device init        : Initialize device
@@ -152,8 +270,8 @@ Subcommands:
 
 To see the output logs, run the following command:
 
-```
-export ALWAYSAI_LOG_TO_CONSOLE=1
+```bash
+$ export ALWAYSAI_LOG_TO_CONSOLE=1
 ```
 
 ### Install the application on the device
@@ -161,7 +279,7 @@ export ALWAYSAI_LOG_TO_CONSOLE=1
 Now you can install the application on the device using the device agent. Run
 the following on the device where the Device Agent is installed:
 
-```
+```bash
 $ aai-agent app install --project <project_id> --release <release_hash>
 ```
 
@@ -170,27 +288,27 @@ $ aai-agent app install --project <project_id> --release <release_hash>
 Run the following commands on the device where the Device Agent is installed:
 
 Get application status:
-```
+```bash
 $ aai-agent app status --project <project_id>
 ```
 
 Start the application:
-```
+```bash
 $ aai-agent app start --project <project_id>
 ```
 
 Show the application logs:
-```
+```bash
 $ aai-agent app logs --project <project_id>
 ```
 
 Stop the application:
-```
+```bash
 $ aai-agent app stop --project <project_id>
 ```
 
 Uninstall the application:
-```
+```bash
 $ aai-agent app uninstall --project <project_id>
 ```
 
@@ -204,7 +322,7 @@ If a new version of a model is published for an existing model ID, and an
 application is already configured to be using that model ID, updating to the
 new model can simply be done with:
 
-```
+```bash
 $ aai-agent app stop --project <project_id>
 $ aai-agent app update-models --project <project_id>
 $ aai-agent app start --project <project_id>
@@ -215,7 +333,7 @@ $ aai-agent app start --project <project_id>
 If you'd like to install an entirely new model to an application, replacing the
 model the app was originally configured with, run the following command:
 
-```
+```bash
 $ aai-agent app stop --project <project_id>
 $ aai-agent app replace-models --project <project_id> --models <model_id_1> [<model_id_2> ...]
 $ aai-agent app start --project <project_id>
@@ -225,7 +343,7 @@ If you plan on using this method, you can make your application source model ID
 agnostic by providing the model ID to edgeIQ in the following way:
 
 Select the first model in the config list:
-```
+```python
 obj_detect = edgeiq.ObjectDetection(edgeiq._globals.MODEL_ID_LIST[0])
 ```
 
@@ -233,13 +351,13 @@ obj_detect = edgeiq.ObjectDetection(edgeiq._globals.MODEL_ID_LIST[0])
 
 To download a model package from the alwaysAI cloud and unpack to a specific directory, run:
 
-```
+```bash
 $ aai-agent get-model-package <model ID> [--path <destination path>]
 ```
 
 For example, to download `alwaysai/yolo_v3` to `~/alwaysai` run:
 
-```
+```bash
 $ aai-agent get-model-package alwaysai/yolo_v3 --path ~/alwaysai
 ```
 
@@ -251,13 +369,13 @@ Once the command completes, you'll see the model package in the
 The Device Agent enables you to set and update environment variables for all
 services or a single service of you application.
 
-```
+```bash
 $ aai-agent app set-env <key=val> --project <project_id> [--service <service>]
 ```
 
 For example, to set the following environment variable for the `alwaysai`
 service run the following command on the device:
 
-```
+```bash
 $ aai-agent app set-env TEST_ENV=1 --project <project_id> --service alwaysai
 ```

package/src/application-control/config.ts

@@ -1,21 +1,24 @@
 import { AgentConfigFile } from '../infrastructure/agent-config';
 import { parse, stringify } from 'yaml';
-import { buildApp, getAppDir } from './utils';
+import {
+  buildApp,
+  getAppDir,
+  requireAppInstalled,
+  requireAppReady
+} from './utils';
 import { JsSpawner } from 'alwaysai/lib/util';
 import compose from 'docker-compose';
 import { assign, merge } from 'lodash';
 import { AppJsonFile } from 'alwaysai/lib/core/app';
 import { AppConfig } from '@alwaysai/app-configuration-schemas';
-import { restartApp } from './status';
+import { isAppStarted, restartApp } from './status';
 import { logger } from '../util/logger';
 
 export async function readAppCfgFile(props: {
   projectId: string;
 }): Promise<AppConfig> {
   const { projectId } = props;
-  if (!(await AgentConfigFile().isAppPresent({ projectId }))) {
-    throw new Error(`App ${projectId} is not present!`);
-  }
+  await requireAppInstalled({ projectId });
   const appDir = getAppDir(projectId);
   const appJson = AppJsonFile(appDir);
   if (!appJson.exists()) {
@@ -37,9 +40,6 @@ export async function writeAppCfgFile(props: {
   appCfg: AppConfig;
 }) {
   const { projectId, appCfg } = props;
-  if (!(await AgentConfigFile().isAppPresent({ projectId }))) {
-    throw new Error(`App ${projectId} is not present!`);
-  }
   const appDir = getAppDir(projectId);
   const appJson = AppJsonFile(appDir);
   try {
@@ -65,25 +65,22 @@ export async function updateAppCfgFile(props: {
 
 export async function updateAppCfg(props: {
   projectId: string;
-  appReleaseHash: string;
   newAppCfg: AppConfig;
 }) {
-  const { projectId, appReleaseHash, newAppCfg } = props;
-  logger.info(`Updating app config for ${projectId}:${appReleaseHash}.`);
+  const { projectId, newAppCfg } = props;
+  logger.info(`Updating app config for ${projectId}`);
   const appDir = getAppDir(projectId);
 
-  if (await AgentConfigFile().isAppPresent({ projectId })) {
-    if (!(await AgentConfigFile().isAppReady({ projectId }))) {
-      throw new Error('Application already has installation in progress!');
-    }
-    logger.info('Application is already installed, updating');
-    await AgentConfigFile().setAppInstalling({
-      projectId,
-      version: appReleaseHash
-    });
-  } else {
-    throw new Error('Application is not installed!');
-  }
+  await requireAppReady({ projectId });
+
+  const appReleaseHash = await AgentConfigFile().getAppVersion({
+    projectId
+  });
+
+  await AgentConfigFile().setAppInstalling({
+    projectId,
+    version: appReleaseHash
+  });
 
   await writeAppCfgFile({ projectId, appCfg: newAppCfg });
   await buildApp({ appDir });
@@ -93,16 +90,21 @@ export async function updateAppCfg(props: {
     version: appReleaseHash
   });
 
-  await restartApp({ projectId });
+  if (await isAppStarted({ projectId })) {
+    await restartApp({ projectId });
+  }
 
-  logger.info(`Updated app config and rebuilt ${projectId}:${appReleaseHash}.`);
+  logger.info(
+    `Updated app config and rebuilt ${projectId}: ${JSON.stringify(
+      newAppCfg,
+      null,
+      2
+    )}`
+  );
 }
 
 export async function readDockerCompose(props: { projectId: string }) {
   const { projectId } = props;
-  if (!(await AgentConfigFile().isAppReady({ projectId }))) {
-    throw new Error(`App ${projectId} is not ready!`);
-  }
   const appDir = getAppDir(projectId);
   const spawner = JsSpawner({ path: appDir });
   const composeContents = await spawner.readFile('docker-compose.yaml');
@@ -115,9 +117,6 @@ export async function writeDockerCompose(props: {
   dockerCompose: any;
 }) {
   const { projectId, dockerCompose } = props;
-  if (!(await AgentConfigFile().isAppReady({ projectId }))) {
-    throw new Error(`App ${projectId} is not ready!`);
-  }
   const appDir = getAppDir(projectId);
   const spawner = JsSpawner({ path: appDir });
   const composeOutput = stringify(dockerCompose);

package/src/application-control/environment-variables.test.ts

@@ -0,0 +1,171 @@
+import { AgentConfigFile } from '../infrastructure/agent-config';
+import { readDockerCompose, writeDockerCompose } from './config';
+import { getAllEnvs, setEnv } from './environment-variables';
+import { isAppStarted, restartApp } from './status';
+import { buildApp, requireAppReady } from './utils';
+
+jest.mock('./config');
+
+jest.mock('./utils');
+jest.mocked(requireAppReady).mockResolvedValue();
+jest.mocked(buildApp).mockResolvedValue();
+
+jest.mock('./status');
+jest.mocked(isAppStarted).mockResolvedValue(true);
+jest.mocked(restartApp).mockResolvedValue();
+
+jest.mock('../infrastructure/agent-config');
+const mockSetAppInstalling = jest.fn().mockResolvedValue({});
+const mockSetAppInstalled = jest.fn().mockResolvedValue({});
+const mockGetAppVersion = jest.fn().mockResolvedValue('test-version');
+jest.mocked(AgentConfigFile as jest.Mock).mockReturnValue({
+  setAppInstalling: mockSetAppInstalling,
+  setAppInstalled: mockSetAppInstalled,
+  getAppVersion: mockGetAppVersion
+});
+const projectId1 = 'test-project';
+
+describe('Test environment variable get and set', () => {
+  beforeEach(() => {
+    jest.clearAllMocks();
+  });
+  describe('Test getAllEnvs()', () => {
+    test('get all envs with one service and no envs', async () => {
+      const compose = {
+        services: {
+          alwaysai: {}
+        }
+      };
+      jest.mocked(readDockerCompose).mockResolvedValue(compose);
+
+      const envVars = await getAllEnvs({ projectId: projectId1 });
+      expect(jest.mocked(readDockerCompose)).toBeCalledWith({
+        projectId: projectId1
+      });
+      expect(envVars).toEqual({
+        alwaysai: {}
+      });
+    });
+    // TODO: Add test with env file
+    test('get all envs with one service and two envs', async () => {
+      const compose = {
+        services: {
+          alwaysai: {
+            environment: ['TEST=1', 'TEST2=2']
+          }
+        }
+      };
+      jest.mocked(readDockerCompose).mockResolvedValue(compose);
+
+      const envVars = await getAllEnvs({ projectId: projectId1 });
+      expect(jest.mocked(readDockerCompose)).toBeCalledWith({
+        projectId: projectId1
+      });
+      expect(envVars).toEqual({
+        alwaysai: { TEST: '1', TEST2: '2' }
+      });
+    });
+    test('get all envs with two services and envs', async () => {
+      const compose = {
+        services: {
+          alwaysai: {
+            environment: ['TEST=3', 'TEST2=4']
+          },
+          edgeiq: {
+            environment: ['TEST3=5', 'TEST4=6']
+          }
+        }
+      };
+      jest.mocked(readDockerCompose).mockResolvedValue(compose);
+
+      const envVars = await getAllEnvs({ projectId: projectId1 });
+      expect(jest.mocked(readDockerCompose)).toBeCalledWith({
+        projectId: projectId1
+      });
+      expect(envVars).toEqual({
+        alwaysai: { TEST: '3', TEST2: '4' },
+        edgeiq: { TEST3: '5', TEST4: '6' }
+      });
+    });
+  });
+  describe('Test setEnv()', () => {
+    test('Set one env', async () => {
+      const compose = {
+        services: {
+          alwaysai: {}
+        }
+      };
+      jest.mocked(readDockerCompose).mockResolvedValue(compose);
+
+      const envVars = { alwaysai: { TEST: '1' } };
+      await setEnv({ projectId: projectId1, envVars });
+      expect(jest.mocked(readDockerCompose)).toBeCalledWith({
+        projectId: projectId1
+      });
+      expect(jest.mocked(writeDockerCompose)).toBeCalledWith({
+        projectId: projectId1,
+        dockerCompose: {
+          services: {
+            alwaysai: {
+              environment: ['TEST=1']
+            }
+          }
+        }
+      });
+    });
+    test('Update one env', async () => {
+      const environment = ['TEST1=1', 'TEST2=2'];
+      const compose = {
+        services: {
+          alwaysai: { environment }
+        }
+      };
+      jest.mocked(readDockerCompose).mockResolvedValue(compose);
+
+      const envVars = { alwaysai: { TEST1: '2' } };
+      await setEnv({ projectId: projectId1, envVars });
+      expect(jest.mocked(readDockerCompose)).toBeCalledWith({
+        projectId: projectId1
+      });
+      expect(jest.mocked(writeDockerCompose)).toBeCalledWith({
+        projectId: projectId1,
+        dockerCompose: {
+          services: {
+            alwaysai: {
+              environment: ['TEST1=1', 'TEST2=2', 'TEST1=2']
+            }
+          }
+        }
+      });
+    });
+    test('Update one env with two services', async () => {
+      const environment = ['TEST1=3', 'TEST2=4'];
+      const compose = {
+        services: {
+          alwaysai: { environment },
+          other: { environment: ['OTHER_TEST=1'] }
+        }
+      };
+      jest.mocked(readDockerCompose).mockResolvedValue(compose);
+
+      const envVars = { alwaysai: { TEST1: '2' } };
+      await setEnv({ projectId: projectId1, envVars });
+      expect(jest.mocked(readDockerCompose)).toBeCalledWith({
+        projectId: projectId1
+      });
+      expect(jest.mocked(writeDockerCompose)).toBeCalledWith({
+        projectId: projectId1,
+        dockerCompose: {
+          services: {
+            alwaysai: {
+              environment: ['TEST1=3', 'TEST2=4', 'TEST1=2']
+            },
+            other: {
+              environment: ['OTHER_TEST=1']
+            }
+          }
+        }
+      });
+    });
+  });
+});