@powerhousedao/academy 0.1.0-dev.0

package/docs/academy/02-AdvancedTutorial/04-WorkWithData/03-WorkingWithSubgraphs/03-WorkingWithSubgraphs.md

@@ -0,0 +1,312 @@
+# Working with Subgraphs
+
+This tutorial will demonstrate how to create and customize a subgraph using our ToDoList project as an example.
+Let's start with the basics and gradually add more complex features and functionality.
+
+## What is a Subgraph?
+
+A subgraph in Powerhouse is a **GraphQL-based modular data component** that extends the functionality of your document models. While document models handle the core state and operations, subgraphs can:
+1. Connect to external APIs or databases
+2. Add custom queries and mutations
+3. Automate interactions between different document models
+4. Provide additional backend functionality
+
+### Subgraphs can retrieve data from:
+
+- **The Reactor** – The core Powerhouse data system or network node.   
+- **Operational Data Stores** – Structured data storage for operational processes, offering real-time updates, for querying structured data.  
+- **Analytics Stores** – Aggregated historical data, useful for insights, reporting and business intelligence.
+
+### Subgraphs consist of:
+
+- **A schema** – Which defines the GraphQL Queries and Mutations.
+- **Resolvers** – Which handle data fetching and logic.
+- **Context Fields** – Additional metadata that helps in resolving data efficiently.
+
+#### Additionaly, context fields allow resolvers to access extra information, such as:
+- **User authentication** (e.g., checking if a user is an admin).
+- **External data sources** (e.g., analytics).
+
+
+
+```typescript title="Example of a context field"
+context: {
+  admin: async (session) => {
+    const admins = await operationalStore.get("admins");
+    return admins.includes(session.user);
+  }
+}
+```
+
+## 1. How to generate a subgraph?
+
+Lets start by generating a new subgraph. For our tutorial we will create a new subgraph within our ToDoList project.   
+Open your project and start your terminal.
+The Powerhouse toolkit provides a command-line utility to create new subgraphs easily.   
+
+```bash title="Run the following command to generate a new subgraph"
+ph generate --subgraph <to-do-list-subgraph>
+```
+
+```bash title="Expected Output"
+Loaded templates: node_modules/@powerhousedao/codegen/dist/codegen/.hygen/templates
+       FORCED: ./subgraphs/to-do-list-subgraph/index.ts
+     skipped: ./subgraphs/index.ts
+      inject: ./subgraphs/index.ts
+```
+
+### What Happened?
+1. A new subgraph was created in `./subgraphs/to-do-list-subgraph/`
+2. The subgraph was automatically registered in your project's registry
+3. Basic boilerplate code was generated with an example query
+
+If we now run 'phreactor' we will see the new subgraph being registered during the startup of the Reactor.
+  > Registered /todolist subgraph.
+
+## 2. Customizing your subgraph with a schema
+
+Now that we've generated our subgraph, let's open it and define the schema inside the `index.ts` file.
+
+### 2.1 Define the Schema
+
+Here we define the schema (typeDefs) which defines the structure of your queries and mutations.
+For educational purposes we will define a simple query that mimics the functionality of the todoList interface (or editor): 
+- Returns the total number of todo's
+- The number of todo's that are checked
+- The number of todo's that are not checked
+
+
+
+```graphql
+type Query {
+  fileIds: [String]
+}
+```
+
+### What Happened?
+
+- Added two queries: todoList and todoItems
+- Created an operational table todo_items to store the todo items
+- Added resolvers to fetch and filter todo items
+- Removed the example code
+- The todoItems query accepts an optional checked parameter to filter items by their status
+- The todoList query returns the full list with its statistics
+
+
+### 2.2 Implement the Resolver for the subgraph's schema
+Resolvers define how data is retrieved or modified.
+If you query for a specific value you can retrieve the value from either the reactor itself or an operational datastore.    
+We'll look into this in more detail in the next section.
+
+```js
+resolvers: {
+  Query: {
+    fileIds: async () => {
+      return ["file1", "file2", "file3"];
+    }
+  }
+}
+```
+
+### 2.3 Add Operational Data Storage (Optional)
+If you need to persist data, initialize an operational datastore inside onSetup():
+
+```typescript title="Adding an operational datastore"
+async onSetup() {
+  await this.createOperationalTables();
+}
+
+async createOperationalTables() {
+  await this.operationalStore.schema.createTableIfNotExists(
+    "fileIds",
+    (table) => {
+      table.string("id").primary();
+    }
+  );
+}
+```
+
+### 2.4 Fetching from an Operational Store
+If your subgraph interacts with an Operational Data Store, modify the resolver:
+
+```typescript title="Example of a resolver that fetches data from an operational store"
+resolvers: {
+  Query: {
+    fileIds: async (_, __, { operationalStore }) => {
+      return await operationalStore.getAll("fileIds");
+    }
+  }
+}
+```
+
+### 2.5 Connecting to a Processor (Optional, but Recommended)
+
+#### Why Connect a Processor?
+Subgraphs alone are limited. A subgraph only queries data, but doesn't generate or store it.
+To make subgraphs useful, connect them with processors that update the data dynamically.
+**A processor listens to system events and updates the operational store in real-time.**
+
+Making use of a processor allows for:
+- **Live updates** instead of static data.
+- **Scalable architecture** through event-driven data changes.
+- **Seamless integration** with other Powerhouse components.
+
+Inside your processor, you can listen for new files, which in turn update the datastore:
+
+```typescript title="Example: Creating a Simple Processor"
+async process(event) {
+  if (event.type === "ADD_FILE") {
+    await this.operationalStore.insert("fileIds", { id: event.fileId });
+  }
+}
+```
+
+Then, modify your subgraph resolver to return real-time data:
+
+```js
+resolvers: {
+  Query: {
+    fileIds: async (_, __, { operationalStore }) => {
+      return await operationalStore.getAll("fileIds");
+    }
+  }
+}
+```
+
+## 3. Testing the Subgraph
+
+### 3.1. Start the Reactor
+To activate the subgraph, run:
+
+```bash
+ph reactor
+```
+Or, for full system startup:
+
+```bash title="Start the Reactor & Connect in Studio or Locally
+ph dev 
+```
+
+### 3.2. Access GraphQL Playground
+Open your browser and go to:
+
+```bash
+http://localhost:4001/<subgraph-name>
+```
+Example:
+
+```bash
+http://localhost:4001/test-subgraph
+```
+
+### 3.3. Run a Query
+
+```graphql
+query {
+  fileIds
+}
+```
+
+### 3.4. Expected Response
+If everything works, you should see:
+
+```json
+{
+  "data": {
+    "fileIds": ["file1", "file2", "file3"]
+  }
+}
+```
+
+## 4. Working with the Supergraph or Gateway
+
+A supergraph is a GraphQL schema that combines multiple underlying GraphQL APIs, known as subgraphs, into a single, unified graph. This architecture allows different teams to work independently on their respective services (subgraphs) while providing a single entry point for clients or users to query all available data
+
+### 4.1 Key Concepts
+
+*   **Subgraph:** An independent GraphQL service with its own schema. Each subgraph typically represents a specific domain or microservice within a larger system.
+*   **Gateway/Router:** A server that sits in front of the subgraphs. It receives client queries, consults the supergraph schema, and routes parts of the query to the relevant subgraphs. It then stitches the results back together before sending the final response to the client.
+
+### 4.2 Benefits of Using a Supergraph
+
+*   **Federated Architecture:** Enables a microservices-based approach where different teams can own and operate their services independently.
+*   **Scalability:** Individual subgraphs can be scaled independently based on their specific needs.
+*   **Improved Developer Experience:** Clients interact with a single, consistent GraphQL API, simplifying data fetching and reducing the need to manage multiple endpoints.
+*   **Schema Evolution:** Subgraphs can evolve their schemas independently, and the supergraph can be updated without breaking existing clients, as long as breaking changes are managed carefully.
+*   **Clear Separation of Concerns:** Each subgraph focuses on a specific domain, leading to more maintainable and understandable codebases.
+
+
+### 4.3 Use the Powerhouse Supergraph
+
+The Powerhouse supergraph for any given remote drive or reactor can be found under `http://localhost:4001/graphql`. The gateway / supergraph available on `/graphql` combines all the subgraphs, except for the drive subgraph (which is accessible via `/d/:driveId`). To get to the endpoint open your localhost by starting the reactor and adding `graphql` to the end of the url. The following commands explain how you can test & try the supergraph. 
+
+- Start the reactor:
+
+  ```bash
+  ph reactor
+  ```
+
+- Open the GraphQL editor in your browser:
+
+  ```
+  http://localhost:4001/graphql
+  ```
+
+The supergraph allows to both query & mutate data from the same endpoint. 
+
+- Create a todo document in the `powerhouse` drive using the `ToDo_createDocument` mutation.
+  ![ToDo_createDocument](https://i.ibb.co/GQTZr7Wk/Screenshot-2025-05-01-at-1-22-23-PM.png)
+
+- Get the document state using the `GetDocument` query.
+  ![GetDocument](https://i.ibb.co/v47cj4Q4/Screenshot-2025-05-01-at-1-22-41-PM.png)
+
+- In a different terminal, start connect:
+
+  ```bash
+  ph connect
+  ```
+
+- Open Connect and add the `powerhouse` drive:
+
+  ```
+  http://localhost:4001/d/powerhouse
+  ```
+
+- You should now see the todo document you've created earlier. Edit the todo document with the document editor and add a few tasks that you can query later. 
+
+- Go back to the GraphQL explorer and use the `GetDocument` query again — you should see the updated state.
+
+This is a quick example of how the supegraph can be used. 
+
+
+## Subgraphs are particularly useful for:
+
+1. **Cross-Document Interactions**: For example, connecting a ToDoList with an Invoice document model:
+   - When an invoice-related task is marked complete, update the invoice status
+   - When an invoice is paid, automatically check off related tasks
+
+2. **External Integrations**: 
+   - Sync tasks with external project management tools
+   - Connect with notification systems
+   - Integrate with analytics platforms
+
+3. **Custom Business Logic**:
+   - Implement complex task prioritization
+   - Add automated task assignments
+   - Create custom reporting functionality
+
+### Prebuilt subgraphs
+
+Some subgraphs (e.g., System Subgraph, Drive Subgraph) already exist.  
+To integrate with them, register them via the Reactor API.
+
+### Future Enhancements
+
+Bridge Processors and Subgraphs – Currently, there's a gap in how processors and subgraphs interact. Powerhouse might improve this in future updates.
+
+
+
+
+
+
+

package/docs/academy/02-AdvancedTutorial/04-WorkWithData/03-WorkingWithSubgraphs/_category_.json

@@ -0,0 +1,8 @@
+{
+    "label": "Working With Subgraphs",
+    "position": 4,
+    "link": {
+      "type": "generated-index",
+      "description": "Learn how to work with graphQL, subgraphs, and processors in Powerhouse."
+    }
+}

package/docs/academy/02-AdvancedTutorial/04-WorkWithData/04-analytics-processor.md

@@ -0,0 +1,342 @@
+# Analytics Processors
+
+An Analytics Processor is an object that can track analytics for operations and state changes on a set of document models. These analytics can be used to generate bespoke dashboards and reports, specific to the type or implementation of the document model.
+
+## Generating an Analytics Processor with the CLI
+
+The `ph-cli` utility can be used to generate the scaffolding for an Analytics Processor.
+
+```
+npx @powerhousedao/ph-cli generate --processor-type analytics --document-models ./my-document-models
+```
+
+This will generate two files: a class that implements `IProcessor` and a `ProcessorFactory` function that creates an instance of your processor. We can start with the factory.
+
+### `ProcessorFactory`
+
+If one has not already been created, the generator will create an `index.ts` that contains a `processorFactory` function:
+
+```typescript
+export const processorFactory =
+  (module: any) =>
+  (driveId: string): ProcessorRecord[] => {
+    return [
+      {
+        processor: new MyAnalyticsProcessor(module.analyticsStore),
+        filter: {
+          branch: ["main"],
+          documentId: ["*"],
+          scope: ["*"],
+          documentType: ["*"],
+        },
+      },
+    ];
+  };
+```
+
+This function appears complicated at first, but provides for great flexibility.
+
+The outside function (`(module: any) => ProcessorFactory`) will be called by the host application (`Reactor`, `Connect`, etc) a single time at initialization. This is intended to resolve external dependencies through the `module` object. In the case of this processor, you can see how the `module` object contains the `analyticsStore`.
+
+Next, for each drive created, the returned function (`(driveId:string): ProcessorRecord[]`) will be called. This means that this custom function will be responsible for determining which processors are added for each drive. Applications are free to reuse processors or pass in application-specific dependencies in to each processor.
+
+The `filter` parameter allows a user to tune which updates the processor receives. Usage is straightfoward: each field of the filter parameter can receive one to many patterns. The array for each field describes an `OR` operator. That is, `["a", "b"]` would match `"a"` or `"b"`. However, matches across fields describe an `AND` operator. That is, an update must match on `branch` `AND` `documentId` `AND` `documentType` `AND` `scope`.
+
+Globs are accepted as input, but not regexes.
+
+```
+{
+  branch: ["main"],
+  documentId: ["*"],
+  documentType: ["doc-type-a", "doc-type-b"],
+  scope: ["global", "local"],
+}
+```
+
+This example would match updates for:
+
+```
+("main" branch) AND (any documentId) AND ("doc-type-a" OR "doc-type-b" documentType) AND ("global" OR "local" scope)
+```
+
+### onStrands
+
+The `onStrands` method is the meat of the processor. This is the function called for all the updates matching the filter. Here is what will be generated for you:
+
+```typescript
+async onStrands<TDocument extends PHDocument>(
+    strands: InternalTransmitterUpdate<TDocument>[]
+  ): Promise<void> {
+  // nothing to update
+  if (strands.length === 0) {
+    return;
+  }
+
+  const analyticsInputs: AnalyticsSeriesInput[] = [];
+  for (const strand of strands) {
+    if (strand.operations.length === 0) {
+      continue;
+    }
+
+    const firstOp = strand.operations[0];
+    const source = AnalyticsPath.fromString(
+      `ph/${strand.driveId}/${strand.documentId}/${strand.branch}/${strand.scope}`,
+    );
+
+    if (firstOp.index === 0) {
+      await this.clearSource(source);
+    }
+
+    for (const operation of strand.operations) {
+      console.log(">>> ", operation.type);
+
+      // add analytics to the analyticsInputs array
+    }
+  }
+
+  if (analyticsInputs.length > 0) {
+    // batch insert all analytics data
+    await this.analyticsStore.addSeriesValues(analyticsInputs);
+  }
+}
+```
+
+This function provides a list of `strand` objects, each with a list of document-model operations (`InternalOperationUpdate[]`) on them. Essentially, it is a list of lists. Each operation will have the previous state of the document and the next state. This allows analytics capture from new state, deltas, or the operations themselves.
+
+> Note that on the first operation (given by `firstOp.index === 0`), it is best practice to clear any previous analytics series for that source. This is so that we do not dual insert operations.
+
+Model-specific code will go where the `console.log` statement currently resides.
+
+> It is best practice to batch insert all updates to the analytics system. In this example, we add all updates to an array of inputs, then insert them all at once. This is optimal over `await`-ing each separate value.
+
+## Learn By Example: RwaAnalyticsProcessor
+
+In the `reactor-local` package, we have implemented a processor for the `makerdao/rwa-portfolio` document type. This is a document model that tracks MakerDAO's Real World Asset (RWA) portfolio. It was initially generated using the `ph-cli` utility.
+
+In the case of the RWA processor, we only want to process updates for the rwa-specific document type, but across all documents. This is why the filter looks like:
+
+```js
+{
+  branch: ["main"],
+  documentId: ["*"],
+  documentType: ["makerdao/rwa-portfolio"],
+  scope: ["global"],
+}
+```
+
+Inside of the `onStrands` function, past the boilerplate, we see what is essentially a giant switch statement.
+
+```typescript
+if (operation.type === "CREATE_GROUP_TRANSACTION") {
+  const groupTransaction = operation.input as CreateGroupTransactionInput;
+  if (
+    groupTransaction.type !== "AssetPurchase" &&
+    groupTransaction.type !== "AssetSale" &&
+    groupTransaction.type !== "PrincipalDraw" &&
+    groupTransaction.type !== "PrincipalReturn"
+  ) {
+    continue;
+  }
+
+  // elided
+}
+```
+
+Since we have knowledge of this specific document model, we can cast the operation input to a specific type. Then, since we only want to track analytics for operations that create transactions, and we can filter out a few transaction types.
+
+Below that, we can see how we are capturing analytics data:
+
+```typescript
+// create good dimensions that we will want to filter on later
+const dimensions = {
+  asset: AnalyticsPath.fromString(
+    `sky/rwas/assets/t-bills/${fixedIncomeTransaction.assetId}`,
+  ),
+  portfolio: AnalyticsPath.fromString(
+    `sky/rwas/portfolios/${documentId}`,
+  ),
+};
+
+// create the series values
+const args = {
+  dimensions,
+  metric: "AssetBalance",
+  source,
+  start: DateTime.fromISO(fixedIncomeTransaction.entryTime),
+  value:
+    groupTransaction.type === "AssetPurchase"
+      ? fixedIncomeTransaction.amount
+      : -fixedIncomeTransaction.amount,
+};
+
+analyticsInputs.push(args);
+```
+
+With this processor implementation, we can now write queries against processor analytics updates. For example, a GQL query might look like the following:
+
+```graphql
+query Analytics($filter: AnalyticsFilter) {
+  analytics {
+    series(filter: $filter) {
+      start
+      end
+      rows {
+        dimensions {
+          name
+          path
+        }
+        metric
+        value
+        unit
+      }
+    }
+  }
+}
+```
+
+With variables:
+
+```json
+{
+  "filter": {
+    "granularity": "annual",
+    "start": "2024-01-01",
+    "end": "2025-01-01",
+    "metrics": [
+      "AssetBalance",
+    ],
+    "dimensions": [
+      {
+        "name": "asset",
+        "select": "sky/rwas",
+        "lod": 2
+      }
+    ]
+  }
+}
+```
+
+## Learn By Example: Document Operations
+
+The RWA processor example pulls information from operation _inputs_ to insert analytics data. Another use case might be to capture meta-analytics from the states themselves.
+
+This processor listens to all documents of type `powerhouse/document-drive`, and since the `document-drive` is itself implemented on top of the document model core systems, this means that we can process all virtual file system operations.  This processor count basic usage metrics using document operations and states.
+
+```typescript
+import { IAnalyticsStore } from "@powerhousedao/reactor-api";
+import { AnalyticsPath } from "@powerhousedao/reactor-api";
+import { AnalyticsSeriesInput } from "@powerhousedao/reactor-api";
+import { InternalTransmitterUpdate, IProcessor } from "document-drive";
+import { AddFileInput, DeleteNodeInput } from "document-drive/drive-document-model/gen/types";
+import { PHDocument } from "document-model";
+import { DateTime } from "luxon";
+
+// iterates over state nodes and retrieves one by id
+const findNode = (state: any, id: string) => {
+  const { nodes } = state;
+  for (const node of nodes) {
+    if (node.id === id) {
+      return node;
+    }
+  }
+
+  return null;
+};
+
+export class DriveProcessorProcessor implements IProcessor {
+  constructor(private readonly analyticsStore: IAnalyticsStore) {
+    //
+  }
+  
+  async onStrands<TDocument extends PHDocument>(
+    strands: InternalTransmitterUpdate<TDocument>[]
+  ): Promise<void> {
+    if (strands.length === 0) {
+      return;
+    }
+
+    const values:AnalyticsSeriesInput[] = [];
+
+    for (const strand of strands) {
+      const operations = strand.operations;
+      await Promise.all(operations.map((operation) => {
+        const source = AnalyticsPath.fromString(`switchboard/default/${strand.driveId}`);
+    
+        const start = DateTime.fromISO(operation.timestamp);
+        const dimensions: any = {
+          documentType: AnalyticsPath.fromString(`document/type/powerhouse/document-drive`),
+        };
+
+        if (operation.index === 0) {
+          this.analyticsStore.clearSeriesBySource(source);
+        }
+    
+        switch (operation.type) {
+          case "ADD_FILE": {
+            // count documents of each type (ADD_FILE, input.documentType)
+    
+            // lookup node in state
+            const input = operation.input as AddFileInput;
+            const node = findNode(strand.state, input.id);
+            if (!node) {
+              return Promise.resolve();
+            }
+    
+            dimensions["kind"] = AnalyticsPath.fromString(`document/kind/${node.kind}`);
+    
+            // increment by adding a 1
+            values.push({
+              source,
+              start,
+              value: 1,
+              metric: "Count",
+              dimensions,
+            });
+            
+            break;
+          }
+          case "ADD_FOLDER": {
+            dimensions["kind"] = AnalyticsPath.fromString("document/kind/folder");
+    
+            // increment by adding a 1
+            values.push({
+              source,
+              start,
+              value: 1,
+              metric: "Count",
+              dimensions,
+            });
+            
+            break;
+          }
+          case "DELETE_NODE": {
+            // the operation only contains the id, so lookup deleted item type in previous state
+            const input = operation.input as DeleteNodeInput;
+            const node = findNode(operation.previousState, input.id);
+            if (!node) {
+              return Promise.resolve();
+            }
+    
+            dimensions["kind"] = AnalyticsPath.fromString(`document/kind/${node.kind}`);
+    
+            // decrement by adding a -1
+            values.push({
+              source,
+              start,
+              value: -1,
+              metric: "Count",
+              dimensions,
+            });
+    
+            break;
+          }
+        }
+      }));
+    }
+
+    await this.analyticsStore.addSeriesValues(values);
+  }
+
+  async onDisconnect() {}
+}
+```