comfy-pr 0.2.27 → 1.0.1

package/README.md

@@ -1,247 +1,348 @@
 # Comfy-PR
 
-Make PRs that helps comfy community grow with [Comfy.org](https://comfy.org/).
+A comprehensive automation platform for [ComfyUI Custom Node](https://registry.comfy.org/) development and publishing. Comfy-PR helps Custom Node authors streamline their publishing workflow by automating repository setup, pull request creation, and ongoing maintenance.
 
-## Project Goals: (aka roadmap)
+## Features
 
-### Cli usage:
+- 🔄 **Automated PR Creation**: Clone repos, update `pyproject.toml`, initialize GitHub Actions, and create PRs
+- 📊 **Analytics Dashboard**: Web-based dashboard with statistics, CSV/YAML exports, and repository insights
+- 🔗 **GitHub Webhook Integration**: Real-time monitoring of issues, PRs, and comments
+- ⚡ **Performance Optimized**: Cached GitHub API client with SQLite storage (5min TTL)
+- 📋 **Task Management**: Automated task execution with progress tracking
+- 🎯 **Rule-Based Follow-ups**: Configurable automation rules for different PR states
 
-- [x] fork repo
-- [x] clone repo locally
-- [x] create pyproject branch, run comfy node init . Push branch.
-- [x] create publish branch, create in a Github workflow file. Push branch.
-- [x] create PR to original repository with template PR description.
-- [x] Submit PR
-- [x] Clean local debris before clone
-- [x] DOING: Export PR status into csv for @haohao
+## Comfy-PR Project Goals
 
-### Github Actions Workerds
+The Comfy-PR project aims to support and streamline the process for Custom Node Authors to publish their work in the Comfy Registry. Here's why this initiative is essential:
 
-- [x] Fetch repos from CM & CR list
-- [x] Make diff
-- [x] Notify to slack channel
-- [x] Fetch repo status (private or archived or ...)
-- [x] Fetch pr status (open / merged / closed) + comments
-- [x] Fetch pr comments
-- [x] Automaticaly find candidates, and do the cli does
-- [x] Mention related prs in dashboard https://github.com/drip-art/Comfy-Registry-PR/issues/1
-- [x] Analyze Totals
-- [x] license schema updator
-- [x] bypass repo
-- [ ] Follow-up prs by state
-  - [ ] Issues Comment
-  - [ ] Slack
-  - [ ] Email
-- [ ] Delete the forked repo which is Merged
+1. **Simplify Node Publishing**: Provide tools and assistance to make publishing Custom Nodes straightforward, allowing authors to concentrate on development rather than the complexities of the publishing process.
+2. **Expand Node Availability**: Streamlined publishing will increase the number of Custom Nodes in the Comfy Registry, enriching the ecosystem and offering more options to users.
+3. **Encourage Collaboration**: Scanning GitHub repositories and providing follow-up support fosters collaboration, knowledge-sharing, and a stronger sense of community among Custom Node Authors and users.
+4. **Ensure Quality and Compliance**: Automate checks and provide guidance to maintain high-quality standards and compliance within the Comfy Registry.
+5. **Resolve Publishing Issues Promptly**: Address Custom Node Authors' issues during the publishing process quickly, reducing frustration and improving the overall user experience.
+6. **Strengthen the Comfy Community**: Help solve users' problems with Custom Node uploading and publishing, contributing to a more vibrant, supportive, and engaged community.
+7. **Promote Innovation**: Lower barriers to publishing Custom Nodes to encourage innovation and creativity within the community, leading to the development of novel and exciting nodes.
 
-### Web Site https://comfy-pr.vercel.app
+Through these efforts, Comfy-PR seeks to create an environment where Custom Node Authors can thrive and users can access a diverse and high-quality array of Custom Nodes.
 
-- [x] A dashboard csv/yaml exporter site for @haohao
-- [x] pr dashboard
+## Architecture Overview
 
-## Admin
+### Core Components
 
-### Changing PR Owner
+1. **CLI Tool** (`src/cli.ts`): Command-line interface for processing individual repositories
+2. **Main Service** (`src/index.ts`): Orchestrates batch processing of repositories
+3. **Web Dashboard** (`app/`): Next.js application with analytics and management UI
+4. **Webhook Service** (`run/index.ts`): Real-time GitHub event monitoring
+5. **Task System** (`app/tasks/`): Automated background tasks for various operations
 
-If you wish to change which Github account the Pull Requests come from, then you need to place a Github token into the **[Actions Secrets](https://github.com/drip-art/Comfy-Registry-PR/settings/secrets/actions )** 
+### Key Features
 
-`GH_TOKEN_COMFY_PR = ************`
+#### CLI Operations
 
-## Usages
+- ✅ Repository forking and local cloning
+- ✅ Automated `pyproject.toml` setup via `comfy node init`
+- ✅ GitHub Actions workflow creation and publishing
+- ✅ Template-based PR creation with descriptions
+- ✅ Clean workspace management
 
-### CLI Usage: Get Started by
+#### Analytics & Monitoring
 
-```
-bunx comfy-pr [...GITHUB_REPO_URLS]
-```
+- ✅ Repository status tracking (private/archived/active)
+- ✅ PR status monitoring (open/merged/closed) with comments
+- ✅ Statistical analysis and reporting
+- ✅ CSV/YAML data exports
+- ✅ Related PR cross-referencing
+
+#### Automation Engine
+
+- ✅ Rule-based follow-up actions
+- ✅ Slack notifications
+- ✅ Email task management
+- ✅ License schema updates
+- ✅ Repository bypass logic
+- 🔄 Auto-cleanup of merged forks (in progress)
+
+### Web Dashboard
 
-### 1. Setup Envs
+Access the dashboard at https://comfy-pr.vercel.app
 
-A demo .env should be sth like:
+- 📊 Repository statistics and analytics
+- 📈 Interactive charts and visualizations
+- 📋 Task management interface
+- 📤 Data export tools (CSV/YAML)
+- 🔍 PR status monitoring and filtering
 
-```sh
-# your github token
-GH_TOKEN=ghp_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
+## Configuration
 
-# the pr source organization/ leave it blank to use yourself's account.
+### Environment Variables
+
+#### Core Settings
+
+```bash
+# GitHub API token (required)
+GH_TOKEN=ghp_your_github_token_here
+
+# PR source organization (optional - defaults to your account)
 FORK_OWNER="ComfyNodePRs"
 
-# PR prefix
+# PR branch prefix (optional)
 FORK_PREFIX="PR-"
+
+# MongoDB connection (for dashboard/analytics)
+MONGODB_URI=mongodb://localhost:27017
 ```
 
-#### Github API Token (GH_TOKEN)
+#### Webhook Service (Optional)
 
-GO https://github.com/settings/tokens?type=beta to get an Github Access key
+```bash
+# Enable real-time webhook monitoring
+USE_WEBHOOKS=true
+GITHUB_WEBHOOK_SECRET=your_secure_webhook_secret
+WEBHOOK_BASE_URL=https://your-domain.com
+PORT=8080
+```
 
-Check 3 permissions for all of your repositories
+### GitHub Token Setup
 
-- Pull requests Access: Read and write
-- Workflows Access: Read and write
-- Metadata Access: Read-only
+1. Go to [GitHub Personal Access Tokens](https://github.com/settings/tokens?type=beta)
+2. Create a fine-grained token with these permissions:
+   - **Pull requests**: Read and write
+   - **Workflows**: Read and write
+   - **Metadata**: Read-only
+   - **Repository hooks**: Read and write (for webhooks)
 
-And save your GH_TOKEN into .env file
+### SSH Key Setup
 
-#### Github SSH Key (.ssh/id_rsa, .ssh/id_rsa.pub)
+Required for automated git operations:
 
-Must provide to push code automaticaly, btw prob. you've already setup.
+```bash
+ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
+cat ~/.ssh/id_rsa.pub
+# Add the public key to https://github.com/settings/keys
+```
 
-Run `ssh-keygen`, got `id_rsa.pub`, Then add into here https://github.com/settings/keys
+## Quick Start
 
-### 2. Run!
+### CLI Usage
 
-Ways to run this script
+```bash
+# Install and run for specific repositories
+bunx comfy-pr [GITHUB_REPO_URLS...]
 
-1. Local run
-2. Docker run (also local)
-3. Docker run at cloud (TODO)
+# Process from a file list
+bunx comfy-pr --repolist repos.txt
 
-#### 1. Launch by Docker Compose
+# Use environment variable
+REPO=https://github.com/owner/repo bunx comfy-pr
+```
 
-After configured your .env file, run docker compose build and up.
+### Examples
 
-```sh
-git clone https://github.com/drip-art/Comfy-Registry-PR
-cd Comfy-Registry-PR
-docker compose build
-docker compose up
-```
+```bash
+# Process a single repository
+bunx comfy-pr https://github.com/example/my-comfy-node
 
-#### 2. Docker usage (not stable)
+# Process multiple repositories
+bunx comfy-pr \
+  https://github.com/user1/node-a \
+  https://github.com/user2/node-b
 
-```sh
-docker run --rm -it \
-    -v $HOME/.ssh:/root/.ssh:ro \
-    -e GH_TOKEN=ghp_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW \
-    -e REPO=https://github.com/snomiao/ComfyNode-Registry-test \
-    snomiao/comfy-registry-pr
+# Use a repository list file
+echo "https://github.com/example/repo1" > repos.txt
+echo "https://github.com/example/repo2" >> repos.txt
+bunx comfy-pr --repolist repos.txt
 ```
 
-#### 3. Run native in Unix/Linux/MacOS/WSL
+## Installation & Setup
 
-```sh
-git clone https://github.com/drip-art/Comfy-Registry-PR
+### Prerequisites
 
-# setup comfy-cli environment
-cd Comfy-Registry-PR
-python3 -m venv .venv
-chmod +x ./.venv/bin/*
-source ./.venv/bin/activate
-pip3 install comfy-cli
+- [Bun](https://bun.sh) runtime
+- Python 3.x with `comfy-cli` installed
+- Git with SSH key configured
+- GitHub Personal Access Token
+
+### Local Development
+
+```bash
+# Clone the repository
+git clone https://github.com/Comfy-Org/Comfy-PR
+cd Comfy-PR
 
+# Install dependencies
+bun install
+
+# Setup Python environment for comfy-cli
+python3 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+pip install comfy-cli
 
+# Configure environment
+cp .env.example .env
+# Edit .env with your settings
 
-# setup bun for js-script
-curl -fsSL https://bun.sh/install | bash
-bun i
+# Run CLI tool
+bun src/cli.ts https://github.com/example/my-comfy-node
 
-# and
-bun src/cli.ts [REPO_PATH_NEED_TO_PR]
-# for example
-bun src/cli.ts https://github.com/snomiao/ComfyNode-Registry-test
+# Run main service (batch processing)
+bun src/index.ts
 
+# Start web dashboard
+bun run dev  # Available at http://localhost:3000
 ```
 
-#### 4. Run natively in Windows
+### Production Deployment
 
-```bat
+#### Docker Compose (Recommended)
 
-git clone https://github.com/drip-art/Comfy-Registry-PR
+```bash
+# Configure environment
+cp .env.example .env
+# Edit .env with production settings
 
-@REM setup comfy-cli environment
-cd Comfy-Registry-PR
-python3 -m venv .venv
-.\.venv\Scripts\activate
-pip3 install comfy-cli
+# Deploy with Docker Compose
+docker compose up -d
+```
 
-@REM run with tsx
-npx -y cross-env REPO=https://github.com/snomiao/ComfyNode-Registry-test npx -y tsx src/cli.ts
+#### Manual Docker
 
+```bash
+docker run --rm -it \
+  -v $HOME/.ssh:/root/.ssh:ro \
+  -e GH_TOKEN=your_github_token \
+  -e MONGODB_URI=mongodb://localhost:27017 \
+  comfy-org/comfy-pr
 ```
 
-#### Other Configurations in dockerfile
+#### Cloud Deployment
+
+The webhook service can be deployed to:
 
-Don't change it unless you know what you are doing.
+- **Vercel**: `vercel --prod`
+- **Google Cloud Run**: `cd run && ./deploy.sh`
+- **Railway**: `railway deploy`
+- **Heroku**: Standard Node.js deployment
 
-```dockerfile
+## Development
 
-ENV FORK_OWNER=drip-art
-ENV FORK_PREFIX=PR-
+### Project Structure
 
-# Unset it into current authorized user's name and email (from your github api token).
-ENV GIT_USEREMAIL=comfy-ci@drip.art
-ENV GIT_USERNAME=comfy-ci
+```
+Comfy-PR/
+├── src/                      # Core utilities and business logic
+│   ├── cli.ts               # Command-line interface
+│   ├── index.ts             # Main service orchestrator
+│   ├── ghc.ts               # Cached GitHub API client
+│   ├── db/                  # Database models and utilities
+│   ├── gh/                  # GitHub API wrappers
+│   └── utils/               # Shared utilities
+├── app/                      # Next.js web dashboard
+│   ├── (dashboard)/         # Dashboard pages and components
+│   ├── api/                 # API routes and tRPC
+│   └── tasks/               # Background task implementations
+├── run/                      # Production services
+│   ├── index.ts             # GitHub webhook service
+│   └── deploy.sh            # Cloud deployment scripts
+├── gh-service/              # Legacy webhook service
+├── templates/               # PR and workflow templates
+└── packages/                # Internal packages
+    └── mongodb-pipeline-ts/ # MongoDB aggregation utilities
 ```
 
-## Development
+### Development Workflow
 
-### Cli
+```bash
+# Install dependencies
+bun install
 
-```sh
-# Create comfy pr dir and go into it
-mkdir comfy-pr
-cd comfy-pr
+# Start MongoDB (required for dashboard)
+docker compose up mongodb -d
 
-# Prepare code and environments
-git clone https://github.com/drip-art/Comfy-Registry-PR .
+# Initialize database
+bun src/index.ts
 
-# Prepare bun
-# go here - [Installation \| Bun Docs]( https://bun.sh/docs/installation )
+# Development modes:
+bun run dev          # Web dashboard (http://localhost:3000)
+bun run dev:tsc      # TypeScript compiler watch mode
+bun run gh-service   # Webhook service
 
-# Install project
-bun i
+# Testing
+bun test             # Run test suite
+bun test --watch     # Watch mode
 
-# Prepare bun
-bun i
+# Linting and building
+bun run lint         # ESLint
+bun run build        # Next.js build
 ```
 
-### Github Action Worker & server
+### Running Individual Components
 
-1. Setup envs in the usages section above (plz check bun src/cli.ts runnable first)
+```bash
+# Process specific repositories
+bun src/cli.ts https://github.com/example/repo
 
-2. Run mongodb with docker compose
+# Run specific tasks
+bun app/tasks/coreping/coreping.ts
+bun app/tasks/gh-bounty/gh-bounty.ts
 
-```sh
-docker compose -f docker-compose.mongodb.yml up
+# Update repository data
+bun src/updateCNRepos.ts
+bun src/updateAuthors.ts
 ```
 
-```yaml
-services:
-  mongdb:
-    restart: always
-    image: mongo
-    ports: ["27017:27017"]
-    volumes: [./data/mongodb:/data/db]
-```
+## Advanced Usage
 
-And fill URI into env
+### Webhook Integration
 
-```env
-MONGODB_URI=mongodb://localhost:27017
+For real-time GitHub event monitoring:
+
+```bash
+# Generate secure webhook secret
+export GITHUB_WEBHOOK_SECRET=$(openssl rand -hex 32)
+
+# Enable webhook mode
+export USE_WEBHOOKS=true
+export WEBHOOK_BASE_URL=https://your-domain.com
+
+# Start webhook service
+bun run gh-service
 ```
 
-3. Play with codes...
+See [WEBHOOK_SETUP.md](./WEBHOOK_SETUP.md) for detailed webhook configuration.
 
-```sh
-# To initialize your database, run:
-bun src/index.ts
+### Task System
 
-# To start develop in any of other scripts:
-# Feel free to run any scripts in src/, they are safe to re-run and stop in any time.
-bun src/THAT_FILE_YOU_WANT_TO_RUN.ts
+The platform includes several automated tasks:
 
-# To check if you didn't break anything?
-bun test --watch
+- **CorePing** (`app/tasks/coreping/`): Repository health monitoring
+- **GitHub Bounty** (`app/tasks/gh-bounty/`): Bounty management
+- **Design Tasks** (`app/tasks/gh-design/`): Design-related automation
+- **Contributor Analysis** (`app/tasks/github-contributor-analyze/`): Contribution statistics
+- **Action Updates** (`app/tasks/github-action-update/`): GitHub Actions maintenance
+
+### Database Inspection
+
+```bash
+# Inspect production database (read-only)
+echo 'MONGODB_URI_INSPECT=mongodb://prod-readonly-uri' > .env.development.local
+echo 'MONGODB_URI=$MONGODB_URI_INSPECT' >> .env.development.local
+
+# Run inspection scripts
+bun src/checkPRsFailures.ts
+bun src/analyzeTotals.ts
 ```
 
+### Performance Optimization
 
-## DB Inspecting
+The project uses a cached GitHub API client (`src/ghc.ts`) that:
 
-Make .env.development.local as 
+- Stores responses in SQLite for 5 minutes
+- Reduces API rate limiting
+- Improves response times for repeated requests
 
-```sh
-MONGODB_URI_INSPECT_HAOHAO={{that readonly db link sno gave you}}
-MONGODB_URI=$MONGODB_URI_INSPECT_HAOHAO
-```
+```typescript
+import { ghc } from "./src/ghc";
 
-And inspect db with script, e.g. `src/checkPRsFailures.ts`
+// Use ghc instead of gh for automatic caching
+const repo = await ghc.repos.get({ owner, repo });
+```

package/bot/IdleWaiter.ts

@@ -0,0 +1,31 @@
+/**
+ * A utility class to wait for idle periods based on activity pings.
+ *
+ * @example
+ * const idleWaiter = new IdleWaiter();
+ *
+ * // Somewhere in your code, when activity occurs:
+ * idleWaiter.ping();
+ *
+ * // To wait for an idle period of 5 seconds:
+ * await idleWaiter.wait(5000);
+ * console.log('System has been idle for 5 seconds');
+ */
+export class IdleWaiter {
+  lastActivityTime = Date.now();
+  checkInterval = 100; // Default check interval in milliseconds
+
+  constructor() {
+    this.ping();
+  }
+
+  ping() {
+    this.lastActivityTime = Date.now();
+    return this;
+  }
+
+  async wait(ms: number) {
+    while (this.lastActivityTime >= Date.now() - ms)
+      await new Promise((resolve) => setTimeout(resolve, this.checkInterval));
+  }
+}