datajunction-ui 0.0.31 → 0.0.34

package/TODO.md

@@ -0,0 +1,265 @@
+# UI Test Improvements TODO
+
+## Phase 1: Fix Test Warnings (Completed)
+
+### 1.1 Fix `act()` Warnings ✅
+
+- [x] Wrap async state updates in `waitFor()` properly
+- [x] Add proper cleanup/waiting in tests for components with async effects
+- [x] Fixed: SQLBuilderPage, NotificationsPage, CubeBuilderPage tests
+- [ ] Remaining: Some AddEditNodePage/AddEditTagPage warnings (minor, from Formik internals)
+
+### 1.2 Fix Key Prop Warnings ✅
+
+- [x] Fix `key={rowValue}` in SQLBuilderPage (line 185) - use index-based key
+- [x] Fix `key={rowValue}` in NodeValidateTab (line 346) - use index-based key
+- [x] Fix missing key in QueryInfo.jsx error mapping (line 125)
+- [x] Fix incorrect mock data in Collapse.test.jsx causing undefined keys
+- [x] Fix missing key in NodeInfoTab.jsx: nodeTags, primary_key, required_dimensions, owners
+
+### 1.3 Fix DOM/React Warnings ✅
+
+- [x] Fix `value` prop on `textarea` should not be null - changed `custom_metadata: null` to `''` in AddEditNodePage
+- [x] Fix invalid DOM property `class` -> `className` in AddEditTagPage
+- [x] Fix nested elements: changed `<div>` to `<span>` in NodeInfoTab nodeTags
+- [x] Fix nested `<p>` tags: changed outer `<p>` to `<div>` in ListGroupItem (Markdown can render `<p>`)
+
+---
+
+## Phase 2: GraphQL Type Safety (In Progress)
+
+### Goal
+
+Migrate all REST GET endpoints to GraphQL for:
+
+- Field selection (fetch only what's needed)
+- Type safety via GraphQL codegen
+- Single source of truth for read operations
+
+Keep REST for mutations (POST/PATCH/DELETE) since they're already implemented and mutations are simpler.
+
+### Completed ✅
+
+#### 2.1 Type Generation Setup
+
+- [x] Added `@graphql-codegen/cli`, `@graphql-codegen/typescript`, `@graphql-codegen/typescript-operations`
+- [x] Added `openapi-typescript` for REST type generation
+- [x] Created `codegen.yml` pointing to static `schema.graphql` (no live server needed)
+- [x] Added scripts: `yarn codegen`, `yarn codegen:graphql`, `yarn codegen:openapi`, `yarn codegen:check`
+- [x] Generated types:
+  - `src/types/graphql.ts` (GraphQL types + query operation types)
+  - `src/types/openapi.ts` (REST endpoint types)
+- [x] Server Makefile updated with `make generate-schemas` and `make check-schemas`
+
+#### 2.2 GraphQL Query Files
+
+- [x] Created `src/graphql/queries/` directory with 10 `.graphql` files:
+  - `listNodesForLanding.graphql`
+  - `listCubesForPreset.graphql`
+  - `cubeForPlanner.graphql`
+  - `getNodeForEditing.graphql`
+  - `getNodesByNames.graphql`
+  - `getMetric.graphql`
+  - `getCubeForEditing.graphql`
+  - `upstreamNodes.graphql`
+  - `downstreamNodes.graphql`
+  - `getNodeColumnsWithPartitions.graphql`
+
+#### 2.3 Typed GraphQL Service
+
+- [x] Created `src/app/services/DJGraphQLService.ts` - fully typed GraphQL service
+- [x] All 10 existing GraphQL methods are now typed
+- [x] DJService.js GraphQL methods now delegate to DJGraphQLService.ts
+- [x] Components continue to use djClient context pattern (no changes needed)
+
+### Usage
+
+Components continue to use the existing djClient pattern - no changes needed:
+
+```jsx
+// Existing pattern (still works, now backed by typed service)
+const djClient = useContext(DJClientContext).DataJunctionAPI;
+const node = await djClient.getNodeForEditing('my.node');
+```
+
+For direct typed access (optional):
+
+```typescript
+import { DJGraphQLService } from '../services/DJGraphQLService';
+import type { GetNodeForEditingQuery } from '../../types/graphql';
+
+const node = await DJGraphQLService.getNodeForEditing('my.node');
+// node.name, node.type, node.current.displayName are all typed
+```
+
+### Development Workflow
+
+**Schema files (committed to repo):**
+| File | Source |
+|------|--------|
+| `openapi.json` (monorepo root) | Server OpenAPI spec |
+| `datajunction-server/.../schema.graphql` | Server GraphQL schema |
+| `datajunction-ui/src/types/graphql.ts` | Generated GraphQL types |
+| `datajunction-ui/src/types/openapi.ts` | Generated REST types |
+
+**When you change the server API:**
+
+```bash
+cd datajunction-server
+make generate-schemas  # Regenerates openapi.json + schema.graphql
+git add ../openapi.json datajunction_server/api/graphql/schema.graphql
+```
+
+**When you change UI GraphQL queries or pull server changes:**
+
+```bash
+cd datajunction-ui
+yarn codegen           # Regenerates graphql.ts + openapi.ts
+git add src/types/
+```
+
+**CI validation:**
+
+- Server: `make check` includes `make check-schemas` → fails if schemas are stale
+- UI: `yarn codegen:check` → fails if generated types are stale
+
+### Current State
+
+**Typed GraphQL methods (in DJGraphQLService.ts):**
+| Method | Purpose |
+|--------|---------|
+| `listNodesForLanding` | Paginated node listing |
+| `listCubesForPreset` | Cube dropdown |
+| `cubeForPlanner` | Query planner page |
+| `getNodeForEditing` | Edit node form |
+| `getNodesByNames` | Batch node fetch |
+| `getMetric` | Metric details |
+| `getCubeForEditing` | Edit cube form |
+| `upstreamNodes` | Upstream nodes |
+| `downstreamNodes` | Downstream nodes |
+| `getNodeColumnsWithPartitions` | Column partitions |
+
+### REST GET Endpoints to Migrate (~45 methods)
+
+#### High Priority (core functionality, frequently used)
+
+- [ ] `node` - GET /nodes/{name}/ → Full node details page
+- [ ] `nodes` - GET /nodes/?prefix= → Node listings
+- [ ] `nodesWithType` - GET /nodes/?node_type= → Filtered node lists
+- [ ] `metric` - GET /metrics/{name}/ → Metric details (REST version)
+- [ ] `metrics` - GET /metrics/ → All metrics list
+- [ ] `cube` - GET /cubes/{name}/ → Full cube details
+- [ ] `commonDimensions` - GET /metrics/common/dimensions/ → Dimension lookups
+- [ ] `upstreams` - GET /nodes/{name}/upstream/ → (has GQL version, deprecate REST)
+- [ ] `downstreams` - GET /nodes/{name}/downstream/ → (has GQL version, deprecate REST)
+- [ ] `history` - GET /history?node= → Activity history
+- [ ] `revisions` - GET /nodes/{name}/revisions/ → Node revision history
+
+#### Medium Priority (namespace/catalog/tags)
+
+- [ ] `namespaces` - GET /namespaces/ → Namespace list
+- [ ] `namespace` - GET /namespaces/{nmspce} → Namespace details
+- [ ] `namespaceSources` - GET /namespaces/{namespace}/sources
+- [ ] `catalogs` - GET /catalogs → Catalog list
+- [ ] `engines` - GET /engines → Engine list
+- [ ] `dimensions` - GET /dimensions → All dimensions
+- [ ] `nodeDimensions` - GET /nodes/{nodeName}/dimensions
+- [ ] `nodesWithDimension` - GET /dimensions/{name}/nodes/
+- [ ] `attributes` - GET /attributes
+- [ ] `listTags` - GET /tags → Tag list
+- [ ] `getTag` - GET /tags/{tagName} → Tag details
+- [ ] `listNodesForTag` - GET /tags/{tagName}/nodes
+- [ ] `users` - GET /users?with_activity=true
+
+#### Medium Priority (materializations)
+
+- [ ] `materializations` - GET /nodes/{node}/materializations
+- [ ] `availabilityStates` - GET /nodes/{node}/availability/
+- [ ] `materializationInfo` - GET /materialization/info
+- [ ] `listPreaggs` - GET /preaggs/
+- [ ] `getPreagg` - GET /preaggs/{preaggId}
+- [ ] `getCubeDetails` - GET /cubes/{cubeName}
+
+#### Lower Priority (SQL/data - may need special handling)
+
+- [ ] `sql` - GET /sql/{metric_name}?... → Generated SQL
+- [ ] `sqls` - GET /sql/?metrics=&dimensions= → Multi-metric SQL
+- [ ] `measuresV3` - GET /sql/measures/v3/ → V3 measures SQL
+- [ ] `metricsV3` - GET /sql/metrics/v3/ → V3 metrics SQL
+- [ ] `data` - GET /data/?metrics=&dimensions= → Query execution
+- [ ] `nodeData` - GET /data/{nodeName}? → Node data query
+- [ ] `compiledSql` - GET /sql/{node}/ → Compiled SQL
+
+#### Lower Priority (system metrics/admin)
+
+- [ ] `querySystemMetric` - GET /system/data/{metric} → Dashboard analytics
+- [ ] `system.node_counts_by_*` - System metrics
+- [ ] `system.dimensions` - GET /system/dimensions
+- [ ] `nodeDetails` - GET /nodes/details/
+- [ ] `node_dag` - GET /nodes/{name}/dag/
+- [ ] `node_lineage` - GET /nodes/{name}/lineage/
+- [ ] `listDeployments` - GET /deployments
+- [ ] `listMetricMetadata` - GET /metrics/metadata
+
+#### Lower Priority (notifications/export)
+
+- [ ] `getNotificationPreferences` - GET /notifications/
+- [ ] `getSubscribedHistory` - GET /history/?only_subscribed=true
+- [ ] `listServiceAccounts` - GET /service-accounts
+- [ ] `clientCode` - GET /datajunction-clients/python/new_node/{name}
+- [ ] `notebookExportCube` - GET /datajunction-clients/python/notebook/?cube=
+- [ ] `notebookExportNamespace` - GET /datajunction-clients/python/notebook/?namespace=
+
+#### Keep as REST (streaming - EventSource)
+
+- `stream` - EventSource /stream/... (GraphQL subscriptions are different tech)
+- `streamNodeData` - EventSource /stream/{nodeName}...
+
+### Implementation Tasks
+
+1. **Server-side: Extend GraphQL schema**
+
+   - [ ] Ensure all required fields are available in GraphQL schema
+   - [ ] Add any missing queries to the GraphQL API
+   - [ ] Verify introspection is enabled for codegen
+
+2. **UI-side: Set up GraphQL codegen**
+
+   - [ ] Add `@graphql-codegen/cli` and `@graphql-codegen/typescript` as dev dependencies
+   - [ ] Create `codegen.yml` configuration
+   - [ ] Add npm script: `yarn generate-types`
+   - [ ] Generate types to `src/types/graphql.ts`
+
+3. **UI-side: Migrate DJService methods**
+
+   - [ ] Start with high-priority methods
+   - [ ] Create GraphQL queries in `src/graphql/queries/`
+   - [ ] Update DJService.js methods to use GraphQL
+   - [ ] Remove deprecated REST calls
+
+4. **Type the mutation payloads**
+   - [ ] Keep OpenAPI types for REST mutations
+   - [ ] Add `openapi-typescript` for mutation request/response types
+   - [ ] Generate to `src/types/api-mutations.ts`
+
+### Benefits
+
+- **Field selection**: UI fetches only needed data
+- **Type safety**: Compile-time errors when code doesn't match schema
+- **Single type source for reads**: GraphQL schema is authoritative
+- **Performance**: Already proven with `cubeForPlanner`, `listNodesForLanding`
+
+---
+
+## Phase 3: Integration Tests (Long-term)
+
+### Goal
+
+Add integration test suite that runs against a real DJ server to catch real integration issues.
+
+### Tasks
+
+- [ ] Add Docker Compose setup for test DJ server
+- [ ] Create separate test suite for integration tests
+- [ ] Add CI workflow to run integration tests
+- [ ] Consider using Playwright or Cypress for E2E tests

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "datajunction-ui",
-  "version": "0.0.31",
+  "version": "0.0.34",
   "description": "DataJunction Metrics Platform UI",
   "module": "src/index.tsx",
   "repository": {

package/src/app/components/ListGroupItem.jsx

@@ -9,14 +9,14 @@ export default class ListGroupItem extends Component {
         <div className="d-flex gap-2 w-100 justify-content-between py-3">
           <div>
             <h6 className="mb-0 w-100">{label}</h6>
-            <p
+            <div
               className="mb-0 opacity-75"
               role="dialog"
               aria-hidden="false"
               aria-label={label}
             >
               <Markdown>{value}</Markdown>
-            </p>
+            </div>
           </div>
         </div>
       </div>

package/src/app/components/QueryInfo.jsx

@@ -122,8 +122,9 @@ export default function QueryInfo({
         <li className={'query-info'}>
           <label>Logs</label>{' '}
           {errors?.length ? (
-            errors.map(error => (
+            errors.map((error, idx) => (
               <div
+                key={`error-${idx}`}
                 style={{
                   height: '800px',
                   width: '80%',

package/src/app/components/__tests__/__snapshots__/ListGroupItem.test.tsx.snap

@@ -13,7 +13,7 @@ exports[`<ListGroupItem /> should render and match the snapshot 1`] = `
       >
         Name
       </h6>
-      <p
+      <div
         aria-hidden="false"
         aria-label="Name"
         className="mb-0 opacity-75"
@@ -24,7 +24,7 @@ exports[`<ListGroupItem /> should render and match the snapshot 1`] = `
             Something
           </span>
         </Markdown>
-      </p>
+      </div>
     </div>
   </div>
 </div>

package/src/app/components/djgraph/__tests__/Collapse.test.jsx

@@ -35,10 +35,13 @@ describe('<Collapse />', () => {
         <Collapse
           {...defaultProps}
           data={{
+            name: 'test.transform',
             type: 'transform',
-            column_names: Array(11).fill(idx => {
-              return `column-${idx}`;
-            }),
+            column_names: Array.from({ length: 11 }, (_, idx) => ({
+              name: `column_${idx}`,
+              type: 'string',
+              order: idx,
+            })),
             primary_key: [],
           }}
         />

package/src/app/pages/AddEditNodePage/index.jsx

@@ -53,7 +53,7 @@ export function AddEditNodePage({ extensions = {} }) {
     primary_key: '',
     mode: 'published',
     owners: [],
-    custom_metadata: null,
+    custom_metadata: '',
   };
 
   const validator = values => {

package/src/app/pages/AddEditTagPage/index.jsx

@@ -90,7 +90,7 @@ export function AddEditTagPage() {
                             name="display_name"
                             id="display_name"
                             placeholder="Display Name"
-                            class="FullNameField"
+                            className="FullNameField"
                           />
                         </div>
                         <br />

package/src/app/pages/CubeBuilderPage/__tests__/index.test.jsx

@@ -185,30 +185,48 @@ describe('CubeBuilderPage', () => {
     jest.clearAllMocks();
   });
 
-  it('renders without crashing', () => {
+  it('renders without crashing', async () => {
     render(
       <DJClientContext.Provider value={{ DataJunctionAPI: mockDjClient }}>
         <CubeBuilderPage />
       </DJClientContext.Provider>,
     );
+
+    // Wait for async effects to complete
+    await waitFor(() => {
+      expect(mockDjClient.metrics).toHaveBeenCalled();
+    });
+
     expect(screen.getByText('Cube')).toBeInTheDocument();
   });
 
-  it('renders the Metrics section', () => {
+  it('renders the Metrics section', async () => {
     render(
       <DJClientContext.Provider value={{ DataJunctionAPI: mockDjClient }}>
         <CubeBuilderPage />
       </DJClientContext.Provider>,
     );
+
+    // Wait for async effects to complete
+    await waitFor(() => {
+      expect(mockDjClient.metrics).toHaveBeenCalled();
+    });
+
     expect(screen.getByText('Metrics *')).toBeInTheDocument();
   });
 
-  it('renders the Dimensions section', () => {
+  it('renders the Dimensions section', async () => {
     render(
       <DJClientContext.Provider value={{ DataJunctionAPI: mockDjClient }}>
         <CubeBuilderPage />
       </DJClientContext.Provider>,
     );
+
+    // Wait for async effects to complete
+    await waitFor(() => {
+      expect(mockDjClient.metrics).toHaveBeenCalled();
+    });
+
     expect(screen.getByText('Dimensions *')).toBeInTheDocument();
   });
 
@@ -237,22 +255,31 @@ describe('CubeBuilderPage', () => {
     }
     fireEvent.click(screen.getAllByText('Dimensions *')[0]);
 
-    expect(mockDjClient.commonDimensions).toHaveBeenCalled();
+    // Wait for commonDimensions to be called and state to update
+    await waitFor(() => {
+      expect(mockDjClient.commonDimensions).toHaveBeenCalled();
+    });
 
     const selectDimensions = screen.getAllByTestId('select-dimensions')[0];
     expect(selectDimensions).toBeDefined();
     expect(selectDimensions).not.toBeNull();
-    expect(
-      screen.getByText(
-        'default.repair_order_details.repair_order_id → default.repair_order.hard_hat_id → default.hard_hat.birth_date',
-      ),
-    ).toBeInTheDocument();
+
+    await waitFor(() => {
+      expect(
+        screen.getByText(
+          'default.repair_order_details.repair_order_id → default.repair_order.hard_hat_id → default.hard_hat.birth_date',
+        ),
+      ).toBeInTheDocument();
+    });
 
     const selectDimensionsDate = screen.getAllByTestId(
       'dimensions-default.date_dim',
     )[0];
 
     fireEvent.keyDown(selectDimensionsDate.firstChild, { key: 'ArrowDown' });
+    await waitFor(() => {
+      expect(screen.getByText('Day')).toBeInTheDocument();
+    });
     fireEvent.click(screen.getByText('Day'));
     fireEvent.click(screen.getByText('Month'));
     fireEvent.click(screen.getByText('Year'));
@@ -264,9 +291,8 @@ describe('CubeBuilderPage', () => {
     })[0];
     expect(createCube).toBeInTheDocument();
 
-    await waitFor(() => {
-      fireEvent.click(createCube);
-    });
+    fireEvent.click(createCube);
+
     await waitFor(() => {
       expect(mockDjClient.createCube).toHaveBeenCalledWith(
         '',
@@ -322,22 +348,31 @@ describe('CubeBuilderPage', () => {
 
     fireEvent.click(screen.getAllByText('Dimensions *')[0]);
 
-    expect(mockDjClient.commonDimensions).toHaveBeenCalled();
+    // Wait for commonDimensions to be called and state to update
+    await waitFor(() => {
+      expect(mockDjClient.commonDimensions).toHaveBeenCalled();
+    });
 
     const selectDimensions = screen.getAllByTestId('select-dimensions')[0];
     expect(selectDimensions).toBeDefined();
     expect(selectDimensions).not.toBeNull();
-    expect(
-      screen.getByText(
-        'default.repair_order_details.repair_order_id → default.repair_order.hard_hat_id → default.hard_hat.birth_date',
-      ),
-    ).toBeInTheDocument();
+
+    await waitFor(() => {
+      expect(
+        screen.getByText(
+          'default.repair_order_details.repair_order_id → default.repair_order.hard_hat_id → default.hard_hat.birth_date',
+        ),
+      ).toBeInTheDocument();
+    });
 
     const selectDimensionsDate = screen.getAllByTestId(
       'dimensions-default.date_dim',
     )[0];
 
     fireEvent.keyDown(selectDimensionsDate.firstChild, { key: 'ArrowDown' });
+    await waitFor(() => {
+      expect(screen.getByText('Day')).toBeInTheDocument();
+    });
     fireEvent.click(screen.getByText('Day'));
     fireEvent.click(screen.getByText('Month'));
     fireEvent.click(screen.getByText('Year'));
@@ -349,9 +384,8 @@ describe('CubeBuilderPage', () => {
     })[0];
     expect(createCube).toBeInTheDocument();
 
-    await waitFor(() => {
-      fireEvent.click(createCube);
-    });
+    fireEvent.click(createCube);
+
     await waitFor(() => {
       expect(mockDjClient.patchCube).toHaveBeenCalledWith(
         'default.repair_orders_cube',

package/src/app/pages/NodePage/NodeInfoTab.jsx

@@ -37,9 +37,13 @@ export default function NodeInfoTab({ node }) {
   const [metricInfo, setMetricInfo] = useState(null);
 
   const nodeTags = node?.tags.map(tag => (
-    <div className={'badge tag_value'}>
+    <span
+      key={tag.name}
+      className={'badge tag_value'}
+      style={{ marginRight: '4px' }}
+    >
       <a href={`/tags/${tag.name}`}>{tag.display_name}</a>
-    </div>
+    </span>
   ));
   const djClient = useContext(DJClientContext).DataJunctionAPI;
 
@@ -314,13 +318,19 @@ export default function NodeInfoTab({ node }) {
         }
       >
         {node?.type !== 'metric'
-          ? node?.primary_key?.map(dim => (
-              <span className="rounded-pill badge bg-secondary-soft PrimaryKey">
+          ? node?.primary_key?.map((dim, idx) => (
+              <span
+                key={`pk-${idx}`}
+                className="rounded-pill badge bg-secondary-soft PrimaryKey"
+              >
                 <a href={`/nodes/${node?.name}`}>{dim}</a>
               </span>
             ))
-          : node?.required_dimensions?.map(dim => (
-              <span className="rounded-pill badge bg-secondary-soft PrimaryKey">
+          : node?.required_dimensions?.map((dim, idx) => (
+              <span
+                key={`rd-${idx}`}
+                className="rounded-pill badge bg-secondary-soft PrimaryKey"
+              >
                 <a href={`/nodes/${node?.upstream_node}`}>{dim.name}</a>
               </span>
             ))}
@@ -341,6 +351,7 @@ export default function NodeInfoTab({ node }) {
             <p className="mb-0 opacity-75">
               {node?.owners.map(owner => (
                 <span
+                  key={owner.username}
                   className="badge node_type__transform"
                   style={{ margin: '2px', fontSize: '16px', cursor: 'pointer' }}
                 >

package/src/app/pages/NodePage/NodeMaterializationTab.jsx

@@ -11,6 +11,11 @@ import AvailabilityStateBlock from './AvailabilityStateBlock';
 
 const cronstrue = require('cronstrue');
 
+/**
+ * Cube materialization tab - shows cube-specific materializations.
+ * For non-cube nodes, the parent component (index.jsx) renders
+ * NodePreAggregationsTab instead.
+ */
 export default function NodeMaterializationTab({ node, djClient }) {
   const [rawMaterializations, setRawMaterializations] = useState([]);
   const [selectedRevisionTab, setSelectedRevisionTab] = useState(null);