comfy-pr 0.2.20 → 0.2.24

package/README.md

@@ -1,225 +1,225 @@
-# Comfy-PR
-
-Make PRs that publishes ComfyUI Custom Nodes to [ComfyUI Registry](https://www.comfyregistry.org/).
-
-## Project Goals: (aka roadmap)
-
-### Cli usage:
-
-- [x] fork repo
-- [x] clone repo locally
-- [x] create pyproject branch, run comfy node init . Push branch.
-- [x] create publish branch, create in a Github workflow file. Push branch.
-- [x] create PR to original repository with template PR description.
-- [x] Submit PR
-- [x] Clean local debris before clone
-- [/] DOING: Export PR status into csv for @haohao
-
-### Github Actions Worker
-
-- [x] Fetch repos from CM & CR list
-- [x] Make diff
-- [x] Notify to slack channel
-- [x] Fetch repo status (private or archived or ...)
-- [x] Fetch pr status (open / merged / closed) + comments
-- [x] Fetch pr comments
-- [x] Automaticaly find candidates, and do the cli does
-- [x] Mention related prs in dashboard https://github.com/drip-art/Comfy-Registry-PR/issues/1
-- [x] Analyze Totals
-- [ ] Follow-up prs by state
-  - [ ] Issues Comment
-  - [ ] Slack
-  - [ ] Email
-- [ ] Delete the forked repo which is Merged
-
-### Web Site https://comfy-pr.vercel.app
-
-- [x] A dashboard csv/yaml exporter site for @haohao
-- [ ]
-
-## Usages
-
-### CLI Usage: Get Started by
-
-```
-bunx comfy-pr [...GITHUB_REPO_URLS]
-```
-
-### 1. Setup Envs
-
-A demo .env should be sth like:
-
-```sh
-# your github token
-GH_TOKEN=ghp_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
-
-# the pr source organization/ leave it blank to use yourself's account.
-FORK_OWNER="ComfyNodePRs"
-
-# PR prefix
-FORK_PREFIX="PR-"
-```
-
-#### Github API Token (GH_TOKEN)
-
-GO https://github.com/settings/tokens?type=beta to get an Github Access key
-
-Check 3 permissions for all of your repositories
-
-- Pull requests Access: Read and write
-- Workflows Access: Read and write
-- Metadata Access: Read-only
-
-And save your GH_TOKEN into .env file
-
-#### Github SSH Key (.ssh/id_rsa, .ssh/id_rsa.pub)
-
-Must provide to push code automaticaly, btw prob. you've already setup.
-
-Run `ssh-keygen`, got `id_rsa.pub`, Then add into here https://github.com/settings/keys
-
-### 2. Run!
-
-Ways to run this script
-
-1. Local run
-2. Docker run (also local)
-3. Docker run at cloud (TODO)
-
-#### 1. Launch by Docker Compose
-
-After configured your .env file, run docker compose build and up.
-
-```sh
-git clone https://github.com/drip-art/Comfy-Registry-PR
-cd Comfy-Registry-PR
-docker compose build
-docker compose up
-```
-
-#### 2. Docker usage (not stable)
-
-```sh
-docker run --rm -it \
-    -v $HOME/.ssh:/root/.ssh:ro \
-    -e GH_TOKEN=ghp_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW \
-    -e REPO=https://github.com/snomiao/ComfyNode-Registry-test \
-    snomiao/comfy-registry-pr
-```
-
-#### 3. Run native in Unix/Linux/MacOS/WSL
-
-```sh
-git clone https://github.com/drip-art/Comfy-Registry-PR
-
-# setup comfy-cli environment
-cd Comfy-Registry-PR
-python3 -m venv .venv
-chmod +x ./.venv/bin/*
-source ./.venv/bin/activate
-pip3 install comfy-cli
-
-
-
-# setup bun for js-script
-curl -fsSL https://bun.sh/install | bash
-bun i
-
-# and
-bun src/cli.ts [REPO_PATH_NEED_TO_PR]
-# for example
-bun src/cli.ts https://github.com/snomiao/ComfyNode-Registry-test
-
-```
-
-#### 4. Run natively in Windows
-
-```bat
-
-git clone https://github.com/drip-art/Comfy-Registry-PR
-
-@REM setup comfy-cli environment
-cd Comfy-Registry-PR
-python3 -m venv .venv
-.\.venv\Scripts\activate
-pip3 install comfy-cli
-
-@REM run with tsx
-npx -y cross-env REPO=https://github.com/snomiao/ComfyNode-Registry-test npx -y tsx src/cli.ts
-
-```
-
-#### Other Configurations in dockerfile
-
-Don't change it unless you know what you are doing.
-
-```dockerfile
-
-ENV FORK_OWNER=drip-art
-ENV FORK_PREFIX=PR-
-
-# Unset it into current authorized user's name and email (from your github api token).
-ENV GIT_USEREMAIL=comfy-ci@drip.art
-ENV GIT_USERNAME=comfy-ci
-```
-
-## Development
-
-### Cli
-
-```sh
-# Create comfy pr dir and go into it
-mkdir comfy-pr
-cd comfy-pr
-
-# Prepare code and environments
-git clone https://github.com/drip-art/Comfy-Registry-PR .
-
-# Prepare bun
-# go here - [Installation \| Bun Docs]( https://bun.sh/docs/installation )
-
-# Install project
-bun i
-
-# Prepare bun
-bun i
-```
-
-### Github Action Worker & server
-
-1. Setup envs in the usages section above (plz check bun src/cli.ts runnable first)
-
-2. Run mongodb with docker compose
-
-```sh
-docker compose -f docker-compose.mongodb.yml up
-```
-
-```yaml
-services:
-  mongdb:
-    restart: always
-    image: mongo
-    ports: ["27017:27017"]
-    volumes: [./data/mongodb:/data/db]
-```
-
-And fill URI into env
-
-```env
-MONGODB_URI=mongodb://localhost:27017
-```
-
-3. Play with codes...
-
-```sh
-# To initialize your database, run:
-bun src/index.ts
-
-# To start develop in any of other scripts:
-# Feel free to run any scripts in src/, they are safe to re-run and stop in any time.
-bun src/THAT_FILE_YOU_WANT_TO_RUN.ts
-
-# To check if you didn't break anything?
-bun test --watch
-```
+# Comfy-PR
+
+Make PRs that publishes ComfyUI Custom Nodes to [ComfyUI Registry](https://www.comfyregistry.org/).
+
+## Project Goals: (aka roadmap)
+
+### Cli usage:
+
+- [x] fork repo
+- [x] clone repo locally
+- [x] create pyproject branch, run comfy node init . Push branch.
+- [x] create publish branch, create in a Github workflow file. Push branch.
+- [x] create PR to original repository with template PR description.
+- [x] Submit PR
+- [x] Clean local debris before clone
+- [/] DOING: Export PR status into csv for @haohao
+
+### Github Actions Worker
+
+- [x] Fetch repos from CM & CR list
+- [x] Make diff
+- [x] Notify to slack channel
+- [x] Fetch repo status (private or archived or ...)
+- [x] Fetch pr status (open / merged / closed) + comments
+- [x] Fetch pr comments
+- [x] Automaticaly find candidates, and do the cli does
+- [x] Mention related prs in dashboard https://github.com/drip-art/Comfy-Registry-PR/issues/1
+- [x] Analyze Totals
+- [ ] Follow-up prs by state
+  - [ ] Issues Comment
+  - [ ] Slack
+  - [ ] Email
+- [ ] Delete the forked repo which is Merged
+
+### Web Site https://comfy-pr.vercel.app
+
+- [x] A dashboard csv/yaml exporter site for @haohao
+- [ ]
+
+## Usages
+
+### CLI Usage: Get Started by
+
+```
+bunx comfy-pr [...GITHUB_REPO_URLS]
+```
+
+### 1. Setup Envs
+
+A demo .env should be sth like:
+
+```sh
+# your github token
+GH_TOKEN=ghp_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
+
+# the pr source organization/ leave it blank to use yourself's account.
+FORK_OWNER="ComfyNodePRs"
+
+# PR prefix
+FORK_PREFIX="PR-"
+```
+
+#### Github API Token (GH_TOKEN)
+
+GO https://github.com/settings/tokens?type=beta to get an Github Access key
+
+Check 3 permissions for all of your repositories
+
+- Pull requests Access: Read and write
+- Workflows Access: Read and write
+- Metadata Access: Read-only
+
+And save your GH_TOKEN into .env file
+
+#### Github SSH Key (.ssh/id_rsa, .ssh/id_rsa.pub)
+
+Must provide to push code automaticaly, btw prob. you've already setup.
+
+Run `ssh-keygen`, got `id_rsa.pub`, Then add into here https://github.com/settings/keys
+
+### 2. Run!
+
+Ways to run this script
+
+1. Local run
+2. Docker run (also local)
+3. Docker run at cloud (TODO)
+
+#### 1. Launch by Docker Compose
+
+After configured your .env file, run docker compose build and up.
+
+```sh
+git clone https://github.com/drip-art/Comfy-Registry-PR
+cd Comfy-Registry-PR
+docker compose build
+docker compose up
+```
+
+#### 2. Docker usage (not stable)
+
+```sh
+docker run --rm -it \
+    -v $HOME/.ssh:/root/.ssh:ro \
+    -e GH_TOKEN=ghp_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW \
+    -e REPO=https://github.com/snomiao/ComfyNode-Registry-test \
+    snomiao/comfy-registry-pr
+```
+
+#### 3. Run native in Unix/Linux/MacOS/WSL
+
+```sh
+git clone https://github.com/drip-art/Comfy-Registry-PR
+
+# setup comfy-cli environment
+cd Comfy-Registry-PR
+python3 -m venv .venv
+chmod +x ./.venv/bin/*
+source ./.venv/bin/activate
+pip3 install comfy-cli
+
+
+
+# setup bun for js-script
+curl -fsSL https://bun.sh/install | bash
+bun i
+
+# and
+bun src/cli.ts [REPO_PATH_NEED_TO_PR]
+# for example
+bun src/cli.ts https://github.com/snomiao/ComfyNode-Registry-test
+
+```
+
+#### 4. Run natively in Windows
+
+```bat
+
+git clone https://github.com/drip-art/Comfy-Registry-PR
+
+@REM setup comfy-cli environment
+cd Comfy-Registry-PR
+python3 -m venv .venv
+.\.venv\Scripts\activate
+pip3 install comfy-cli
+
+@REM run with tsx
+npx -y cross-env REPO=https://github.com/snomiao/ComfyNode-Registry-test npx -y tsx src/cli.ts
+
+```
+
+#### Other Configurations in dockerfile
+
+Don't change it unless you know what you are doing.
+
+```dockerfile
+
+ENV FORK_OWNER=drip-art
+ENV FORK_PREFIX=PR-
+
+# Unset it into current authorized user's name and email (from your github api token).
+ENV GIT_USEREMAIL=comfy-ci@drip.art
+ENV GIT_USERNAME=comfy-ci
+```
+
+## Development
+
+### Cli
+
+```sh
+# Create comfy pr dir and go into it
+mkdir comfy-pr
+cd comfy-pr
+
+# Prepare code and environments
+git clone https://github.com/drip-art/Comfy-Registry-PR .
+
+# Prepare bun
+# go here - [Installation \| Bun Docs]( https://bun.sh/docs/installation )
+
+# Install project
+bun i
+
+# Prepare bun
+bun i
+```
+
+### Github Action Worker & server
+
+1. Setup envs in the usages section above (plz check bun src/cli.ts runnable first)
+
+2. Run mongodb with docker compose
+
+```sh
+docker compose -f docker-compose.mongodb.yml up
+```
+
+```yaml
+services:
+  mongdb:
+    restart: always
+    image: mongo
+    ports: ["27017:27017"]
+    volumes: [./data/mongodb:/data/db]
+```
+
+And fill URI into env
+
+```env
+MONGODB_URI=mongodb://localhost:27017
+```
+
+3. Play with codes...
+
+```sh
+# To initialize your database, run:
+bun src/index.ts
+
+# To start develop in any of other scripts:
+# Feel free to run any scripts in src/, they are safe to re-run and stop in any time.
+bun src/THAT_FILE_YOU_WANT_TO_RUN.ts
+
+# To check if you didn't break anything?
+bun test --watch
+```

package/next-env.d.ts

@@ -1,5 +1,5 @@
-/// <reference types="next" />
-/// <reference types="next/image-types/global" />
-
-// NOTE: This file should not be edited
-// see https://nextjs.org/docs/basic-features/typescript for more information.
+/// <reference types="next" />
+/// <reference types="next/image-types/global" />
+
+// NOTE: This file should not be edited
+// see https://nextjs.org/docs/basic-features/typescript for more information.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "comfy-pr",
-  "version": "0.2.20",
+  "version": "0.2.24",
   "description": "Make PRs that publishes ComfyUI Custom Nodes to [ComfyUI Registry]( https://www.comfyregistry.org/ ).",
   "keywords": [],
   "license": "ISC",
@@ -21,6 +21,8 @@
   ],
   "scripts": {
     "_postinstall": "bun ./src/muteInstances.ts",
+    "vercel:dev": "vercel dev",
+    "vercel:build": "vercel build",
     "build": "next build",
     "dev": "next dev",
     "lint": "next lint",
@@ -34,13 +36,15 @@
     "@ctrl/mac-address": "^3.0.3",
     "@monaco-editor/react": "^4.6.0",
     "@slack/web-api": "^7.2.0",
-    "@snomiao/die": "^1.0.9",
+    "@snomiao/die": "^1.1.0",
     "@trpc/server": "^10.45.2",
     "@types/d3": "^7.4.3",
+    "@types/file-saver": "^2.0.7",
     "d3": "^7.9.0",
     "daisyui": "^4.12.7",
     "dotenv": "^16.4.5",
     "enhanced-ms": "^3.0.0",
+    "file-saver": "^2.0.5",
     "globby": "^14.0.1",
     "js-yaml": "^4.1.0",
     "json-stable-stringify": "^1.1.1",
@@ -62,10 +66,14 @@
     "react-api-doc": "^0.0.9",
     "react-hook-form": "^7.52.0",
     "react-markdown": "^9.0.1",
+    "snoflow": "^1.6.0",
+    "sparse-bitfield": "^3.0.3",
     "toml": "^3.0.0",
     "trpc-openapi": "^1.2.0",
     "ts-pattern": "^5.2.0",
+    "unwind-array": "^1.1.4",
     "use-swr-component": "^0.0.5",
+    "vercel": "^34.3.0",
     "yaml": "^2.4.5",
     "zod": "^3.23.8",
     "zx": "^8.1.3"

package/src/Authors.ts

@@ -0,0 +1,48 @@
+import { type Task } from "@/packages/mongodb-pipeline-ts/Task";
+import console from "console";
+import { db } from "./db";
+import { gh } from "./gh";
+import type { AwaitedReturnType } from "./types/AwaitedReturnType";
+import { updateAuthors } from "./updateAuthors";
+export type Author = {
+  email?: string;
+  // cold down
+  githubMtime?: Date;
+  instagramMtime?: Date;
+  discordMtime?: Date;
+  twitterMtime?: Date;
+
+  // id
+  githubId?: string;
+  instagramId?: string;
+  discordId?: string;
+  twitterId?: string;
+
+  // repos on cm and cr
+  cm: number; // comfy node manager
+  cr: number; // comfy registry
+
+  // common info
+  nicknames?: string[];
+  bios?: string[];
+  links?: string[]; // one could be url or markdown style link: [name](link)
+  blogs?: string[]; // one could be url or markdown style link: [name](link)
+  companies?: string[];
+  locations?: string[];
+  avatars?: string[];
+  hireable?: boolean; // github
+};
+
+export const Authors = db.collection<Author>("Authors");
+await Authors.createIndex('githubId')
+await Authors.createIndex('email')
+export const GithubUsers = db.collection<
+  { username: string } & Task<AwaitedReturnType<typeof gh.users.getByUsername>["data"]>
+>("GithubUsers");
+
+if (import.meta.main) {
+  // collect github id from cn repos
+  await updateAuthors();
+  console.log("done");
+}
+

package/src/CMNodes.ts

@@ -1,60 +1,60 @@
-import jsonStableStringify from "json-stable-stringify";
-import md5 from "md5";
-import type { ObjectId } from "mongodb";
-import { db } from "./db";
-import { fetchCMNodes } from "./fetchCMNodes";
-import { type SlackMsg } from "./slack/SlackMsgs";
-import { updateCMNodesDuplicationWarnings } from "./updateCMNodesDuplicationWarnings";
-
-// Raw version maybe duplicated with id or reference
-type CMNodeRaw = Awaited<ReturnType<typeof fetchCMNodesWithHash>>[number];
-export type CMNode = CMNodeRaw & {
-  repo_id?: ObjectId;
-  duplicated?: {
-    [k: string]: { hashes: string[]; slackNotification: SlackMsg };
-  };
-};
-export const CMNodes = db.collection<CMNode>("CMNodes");
-await CMNodes.createIndex({ mtime: -1 });
-await CMNodes.createIndex({ hash: 1 }, { unique: true }).catch(() => {});
-
-if (import.meta.main) {
-  console.time("CMNodes");
-  console.log(await updateCMNodes());
-  // check dups
-  console.timeLog("CMNodes");
-  const all = await CMNodes.countDocuments({});
-  console.timeLog("CMNodes");
-  const dups = await CMNodes.countDocuments({ duplicated: { $exists: true } });
-  console.timeEnd("CMNodes");
-  console.log("CMNodes updated, duplicates:", dups, "all:", all);
-}
-export async function updateCMNodes() {
-  const nodes = await fetchCMNodesWithHash();
-  console.log("CMNodes updating (" + nodes.length + " nodes)");
-
-  // updating nodes
-  const updateResult = await CMNodes.bulkWrite(
-    nodes.flatMap((node) => ({
-      updateOne: {
-        filter: { hash: node.hash },
-        update: { $set: node },
-        upsert: true,
-      },
-    })),
-    { ordered: false },
-  );
-  // delete outdated
-  // await CMNodes.deleteMany({ hash: { $nin: nodes.map(prop("hash")) } });
-
-  await updateCMNodesDuplicationWarnings(nodes);
-
-  return [updateResult];
-}
-
-async function fetchCMNodesWithHash() {
-  return (await fetchCMNodes()).map((e) => ({
-    ...e,
-    hash: md5("SALT=Bvxmh8mGYh6qzLGE " + jsonStableStringify(e)),
-  }));
-}
+import jsonStableStringify from "json-stable-stringify";
+import md5 from "md5";
+import type { ObjectId } from "mongodb";
+import { db } from "./db";
+import { fetchCMNodes } from "./fetchCMNodes";
+import { type SlackMsg } from "./slack/SlackMsgs";
+import { updateCMNodesDuplicationWarnings } from "./updateCMNodesDuplicationWarnings";
+
+// Raw version maybe duplicated with id or reference
+type CMNodeRaw = Awaited<ReturnType<typeof fetchCMNodesWithHash>>[number];
+export type CMNode = CMNodeRaw & {
+  repo_id?: ObjectId;
+  duplicated?: {
+    [k: string]: { hashes: string[]; slackNotification: SlackMsg };
+  };
+};
+export const CMNodes = db.collection<CMNode>("CMNodes");
+await CMNodes.createIndex({ mtime: -1 });
+await CMNodes.createIndex({ hash: 1 }, { unique: true }).catch(() => {});
+
+if (import.meta.main) {
+  console.time("CMNodes");
+  console.log(await updateCMNodes());
+  // check dups
+  console.timeLog("CMNodes");
+  const all = await CMNodes.countDocuments({});
+  console.timeLog("CMNodes");
+  const dups = await CMNodes.countDocuments({ duplicated: { $exists: true } });
+  console.timeEnd("CMNodes");
+  console.log("CMNodes updated, duplicates:", dups, "all:", all);
+}
+export async function updateCMNodes() {
+  const nodes = await fetchCMNodesWithHash();
+  console.log("CMNodes updating (" + nodes.length + " nodes)");
+
+  // updating nodes
+  const updateResult = await CMNodes.bulkWrite(
+    nodes.flatMap((node) => ({
+      updateOne: {
+        filter: { hash: node.hash },
+        update: { $set: node },
+        upsert: true,
+      },
+    })),
+    { ordered: false },
+  );
+  // delete outdated
+  // await CMNodes.deleteMany({ hash: { $nin: nodes.map(prop("hash")) } });
+
+  await updateCMNodesDuplicationWarnings(nodes);
+
+  return [updateResult];
+}
+
+async function fetchCMNodesWithHash() {
+  return (await fetchCMNodes()).map((e) => ({
+    ...e,
+    hash: md5("SALT=Bvxmh8mGYh6qzLGE " + jsonStableStringify(e)),
+  }));
+}