ghcp-llm-council 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +42 -0
- package/README.md +221 -0
- package/bin/install.js +89 -0
- package/package.json +43 -0
- package/skills/llm-council/SKILL.md +256 -0
- package/skills/llm-council/bin/council.cjs +616 -0
- package/skills/llm-council/defaults/council.json +38 -0
- package/skills/llm-council/prompts/design/chairman.md +26 -0
- package/skills/llm-council/prompts/design/councillor.md +24 -0
- package/skills/llm-council/prompts/general/chairman.md +19 -0
- package/skills/llm-council/prompts/general/councillor.md +11 -0
- package/skills/llm-council/prompts/plan/chairman.md +26 -0
- package/skills/llm-council/prompts/plan/councillor.md +24 -0
- package/skills/llm-council/prompts/ranker.md +28 -0
- package/skills/llm-council/prompts/research/chairman.md +25 -0
- package/skills/llm-council/prompts/research/councillor.md +24 -0
- package/skills/llm-council/prompts/review/chairman.md +25 -0
- package/skills/llm-council/prompts/review/councillor.md +25 -0
- package/skills/llm-council/server/lib/config.cjs +112 -0
- package/skills/llm-council/server/lib/health.cjs +36 -0
- package/skills/llm-council/server/lib/http.cjs +273 -0
- package/skills/llm-council/server/lib/ids.cjs +11 -0
- package/skills/llm-council/server/lib/models.cjs +44 -0
- package/skills/llm-council/server/lib/ranking.cjs +36 -0
- package/skills/llm-council/server/lib/store.cjs +139 -0
- package/skills/llm-council/server/lib/ws.cjs +66 -0
- package/skills/llm-council/server/public/app.css +251 -0
- package/skills/llm-council/server/public/app.js +647 -0
- package/skills/llm-council/server/public/index.html +211 -0
- package/skills/llm-council/server/public/vendor/CHECKSUMS +8 -0
- package/skills/llm-council/server/public/vendor/LICENSES/Apache-2.0.txt +201 -0
- package/skills/llm-council/server/public/vendor/LICENSES/MPL-2.0.txt +373 -0
- package/skills/llm-council/server/public/vendor/LICENSES/marked-MIT.txt +44 -0
- package/skills/llm-council/server/public/vendor/README.md +33 -0
- package/skills/llm-council/server/public/vendor/marked.min.js +6 -0
- package/skills/llm-council/server/public/vendor/purify.min.js +3 -0
- package/skills/llm-council/server/server.cjs +90 -0
- package/skills/llm-council/server/start.cjs +75 -0
- package/skills/llm-council/server/stop.cjs +34 -0
- package/skills/llm-council/version.json +1 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Vivekananda Gavini and ghcp-llm-council contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
22
|
+
|
|
23
|
+
================================================================================
|
|
24
|
+
Third-party components bundled with this software
|
|
25
|
+
================================================================================
|
|
26
|
+
|
|
27
|
+
The frontend bundles two third-party libraries under their respective
|
|
28
|
+
licenses. Their license texts and notices are reproduced verbatim in
|
|
29
|
+
`skills/llm-council/server/public/vendor/LICENSES/`.
|
|
30
|
+
|
|
31
|
+
* marked v12.0.0 — MIT License
|
|
32
|
+
Copyright (c) 2018+ MarkedJS, Copyright (c) 2011-2018 Christopher Jeffrey
|
|
33
|
+
See: skills/llm-council/server/public/vendor/LICENSES/marked-MIT.txt
|
|
34
|
+
|
|
35
|
+
* DOMPurify v3.2.4 — Apache License 2.0 OR Mozilla Public License 2.0
|
|
36
|
+
Copyright (c) Cure53 and other contributors
|
|
37
|
+
See: skills/llm-council/server/public/vendor/LICENSES/Apache-2.0.txt
|
|
38
|
+
skills/llm-council/server/public/vendor/LICENSES/MPL-2.0.txt
|
|
39
|
+
|
|
40
|
+
The original copyright notices in those minified files are preserved
|
|
41
|
+
unchanged.
|
|
42
|
+
|
package/README.md
ADDED
|
@@ -0,0 +1,221 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="assets/logo.svg" width="80" alt="LLM Council" />
|
|
3
|
+
</p>
|
|
4
|
+
|
|
5
|
+
<h1 align="center">LLM Council</h1>
|
|
6
|
+
|
|
7
|
+
<p align="center">
|
|
8
|
+
Multi-model deliberation for GitHub Copilot CLI.<br/>
|
|
9
|
+
Five models debate. One chairman synthesises. You watch it live.
|
|
10
|
+
</p>
|
|
11
|
+
|
|
12
|
+
<p align="center">
|
|
13
|
+
<a href="https://www.npmjs.com/package/ghcp-llm-council"><img src="https://img.shields.io/npm/v/ghcp-llm-council?color=blue" alt="npm version" /></a>
|
|
14
|
+
<a href="./LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License" /></a>
|
|
15
|
+
<img src="https://img.shields.io/badge/node-%3E%3D20-brightgreen" alt="Node 20+" />
|
|
16
|
+
<a href="https://github.com/thevgavini/ghcp-llm-council/stargazers"><img src="https://img.shields.io/github/stars/thevgavini/ghcp-llm-council?style=social" alt="GitHub stars" /></a>
|
|
17
|
+
</p>
|
|
18
|
+
|
|
19
|
+
<p align="center">
|
|
20
|
+
<a href="#install">Install</a> · <a href="#how-it-works">How it works</a> · <a href="#usage">Usage</a> · <a href="#configuration">Configuration</a> · <a href="#development">Development</a>
|
|
21
|
+
</p>
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## What is this?
|
|
26
|
+
|
|
27
|
+
LLM Council is a **skill** (plugin) for [GitHub Copilot CLI](https://docs.github.com/en/copilot/github-copilot-in-the-cli). Instead of getting one model's opinion, you get a structured debate between five frontier models — with peer review, anonymous ranking, and a chairman's synthesis.
|
|
28
|
+
|
|
29
|
+
Think of it as a panel of senior engineers reviewing your question. Each brings a different perspective (Anthropic, OpenAI, Meta, DeepSeek, Mistral), they critique each other's answers without knowing who said what, and a chairman distils the best reasoning into one definitive response.
|
|
30
|
+
|
|
31
|
+
Everything runs locally through your existing Copilot subscription and GitHub token. No extra API keys, no external services, no data leaving your machine beyond the model inference calls you're already making.
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
Question ──► 5 councillors answer in parallel (Stage 1)
|
|
37
|
+
│
|
|
38
|
+
▼
|
|
39
|
+
Responses anonymised (A, B, C, D, E)
|
|
40
|
+
│
|
|
41
|
+
▼
|
|
42
|
+
Each councillor ranks all responses (Stage 2)
|
|
43
|
+
│
|
|
44
|
+
▼
|
|
45
|
+
Rankings aggregated (Borda count)
|
|
46
|
+
│
|
|
47
|
+
▼
|
|
48
|
+
Chairman reads everything, writes synthesis (Stage 3)
|
|
49
|
+
│
|
|
50
|
+
▼
|
|
51
|
+
Final answer ──► CLI + Browser UI + Disk
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
A local web UI shows every stage unfolding in real time. No external API keys required — uses your existing Copilot subscription and `gh auth` token.
|
|
55
|
+
|
|
56
|
+
### Architecture
|
|
57
|
+
|
|
58
|
+
<p align="center">
|
|
59
|
+
<img src="assets/architecture.svg" alt="System Architecture" width="100%" />
|
|
60
|
+
</p>
|
|
61
|
+
|
|
62
|
+
### Deliberation flow
|
|
63
|
+
|
|
64
|
+
<p align="center">
|
|
65
|
+
<img src="assets/deliberation-flow.svg" alt="Deliberation Flow" width="100%" />
|
|
66
|
+
</p>
|
|
67
|
+
|
|
68
|
+
## Default council
|
|
69
|
+
|
|
70
|
+
| Model | Vendor | Role | Backend |
|
|
71
|
+
|-------|--------|------|---------|
|
|
72
|
+
| Claude Sonnet 4.6 | Anthropic | Councillor | Copilot inference (`task`) |
|
|
73
|
+
| GPT-5.2 | OpenAI | Councillor | Copilot inference (`task`) |
|
|
74
|
+
| Llama 3.3 70B | Meta | Councillor | GitHub Models |
|
|
75
|
+
| DeepSeek V3 | DeepSeek | Councillor | GitHub Models |
|
|
76
|
+
| Mistral Medium 2505 | Mistral | Councillor | GitHub Models |
|
|
77
|
+
| **Claude Opus 4.7** | **Anthropic** | **Chairman** | Copilot inference (`task`) |
|
|
78
|
+
|
|
79
|
+
Two inference paths, zero extra credentials:
|
|
80
|
+
- **`task` backend** — models dispatched through Copilot's built-in inference (your subscription covers it)
|
|
81
|
+
- **`github-models` backend** — models called via `models.github.ai` using your `gh auth token`
|
|
82
|
+
|
|
83
|
+
## Install
|
|
84
|
+
|
|
85
|
+
### One-liner (npx)
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
npx ghcp-llm-council install
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### Manual
|
|
92
|
+
|
|
93
|
+
```powershell
|
|
94
|
+
# 1. Install GitHub CLI (if you don't have it)
|
|
95
|
+
winget install GitHub.cli
|
|
96
|
+
|
|
97
|
+
# 2. Authenticate (required for model inference)
|
|
98
|
+
gh auth login
|
|
99
|
+
|
|
100
|
+
# 3. Clone and install the skill
|
|
101
|
+
git clone https://github.com/thevgavini/ghcp-llm-council.git
|
|
102
|
+
cd ghcp-llm-council
|
|
103
|
+
.\install-skill.ps1
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
### Uninstall
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
npx ghcp-llm-council uninstall
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Restart your Copilot CLI session after installing. Requires Node 20+ and GitHub Copilot CLI.
|
|
113
|
+
|
|
114
|
+
## Usage
|
|
115
|
+
|
|
116
|
+
In any Copilot CLI session:
|
|
117
|
+
|
|
118
|
+
```
|
|
119
|
+
ask the council: should we use a monorepo or polyrepo for our microservices?
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
The skill:
|
|
123
|
+
1. Asks you to pick a mode (general, review, design, plan, research)
|
|
124
|
+
2. Spins up a local server, opens the browser UI
|
|
125
|
+
3. Dispatches all councillors in parallel
|
|
126
|
+
4. Runs peer review + ranking
|
|
127
|
+
5. Chairman synthesises a final answer
|
|
128
|
+
6. Returns the synthesis in chat and persists to disk
|
|
129
|
+
|
|
130
|
+
### Follow-ups
|
|
131
|
+
|
|
132
|
+
```
|
|
133
|
+
follow up: what about the CI/CD complexity tradeoff?
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Runs a fresh 3-stage deliberation on the existing conversation thread.
|
|
137
|
+
|
|
138
|
+
### File context
|
|
139
|
+
|
|
140
|
+
```
|
|
141
|
+
council: review src/auth.cjs for security holes
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
The skill auto-detects file references and passes them to every councillor inline.
|
|
145
|
+
|
|
146
|
+
## Modes
|
|
147
|
+
|
|
148
|
+
| Mode | Best for | Trigger words |
|
|
149
|
+
|------|----------|---------------|
|
|
150
|
+
| `general` | Open Q&A, opinions, explanations | _(default)_ |
|
|
151
|
+
| `review` | Code review, security audit | "review", "audit", "find bugs" |
|
|
152
|
+
| `design` | Architecture decisions, tech choices | "design", "should I use X or Y" |
|
|
153
|
+
| `plan` | Implementation roadmaps | "plan", "roadmap", "how would you build" |
|
|
154
|
+
| `research` | Deep dives, learning | "explain", "how does X work" |
|
|
155
|
+
|
|
156
|
+
Each mode has tuned prompts for councillors and chairman. Mode packs can override the council lineup (e.g., `review` mode uses Opus as both councillor and chairman).
|
|
157
|
+
|
|
158
|
+
## Configuration
|
|
159
|
+
|
|
160
|
+
Defaults: `skills/llm-council/defaults/council.json`
|
|
161
|
+
Per-project override: `<cwd>/.llm-council/state/council.json`
|
|
162
|
+
|
|
163
|
+
```json
|
|
164
|
+
{
|
|
165
|
+
"council": [
|
|
166
|
+
{"id": "claude-sonnet-4.6", "vendor": "Anthropic", "display": "Claude Sonnet 4.6", "backend": "task"},
|
|
167
|
+
{"id": "gpt-5.2", "vendor": "OpenAI", "display": "GPT-5.2", "backend": "task"},
|
|
168
|
+
{"id": "meta/llama-3.3-70b-instruct", "vendor": "Meta", "display": "Llama 3.3 70B", "backend": "github-models"}
|
|
169
|
+
],
|
|
170
|
+
"chairman": "claude-opus-4.7",
|
|
171
|
+
"chairman_backend": "task",
|
|
172
|
+
"min_responses_to_proceed": 2,
|
|
173
|
+
"councillor_timeout_seconds": 120
|
|
174
|
+
}
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Add any model supported by Copilot inference or GitHub Models. The council scales to any size.
|
|
178
|
+
|
|
179
|
+
## Project structure
|
|
180
|
+
|
|
181
|
+
```
|
|
182
|
+
skills/llm-council/
|
|
183
|
+
├── SKILL.md # Agent instructions (the skill contract)
|
|
184
|
+
├── bin/council.cjs # CLI helper (init, patch, advance, synthesize)
|
|
185
|
+
├── server/ # Zero-dependency HTTP + WebSocket server
|
|
186
|
+
│ ├── start.cjs # Entry point
|
|
187
|
+
│ └── public/ # Browser UI (vanilla JS, live cards)
|
|
188
|
+
├── defaults/council.json # Default model lineup
|
|
189
|
+
└── prompts/ # Mode-specific prompt templates
|
|
190
|
+
├── general/ # councillor.md + chairman.md
|
|
191
|
+
├── review/
|
|
192
|
+
├── design/
|
|
193
|
+
├── plan/
|
|
194
|
+
└── research/
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
State lives in `<cwd>/.llm-council/` — add to `.gitignore`.
|
|
198
|
+
|
|
199
|
+
## Development
|
|
200
|
+
|
|
201
|
+
```bash
|
|
202
|
+
node skills/llm-council/server/start.cjs # Launch server standalone for UI dev
|
|
203
|
+
node skills/llm-council/server/stop.cjs # Tear down
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Tests live on the `tests/suite` branch:
|
|
207
|
+
|
|
208
|
+
```bash
|
|
209
|
+
git checkout tests/suite
|
|
210
|
+
npm test
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
## Inspired by
|
|
214
|
+
|
|
215
|
+
[karpathy/llm-council](https://github.com/karpathy/llm-council) — the original multi-model deliberation concept. This project brings it into the Copilot CLI ecosystem with live visualisation, zero external credentials, domain-specific modes (review, design, plan, research) with tuned prompts, and a skill-native interface that runs entirely from your terminal.
|
|
216
|
+
|
|
217
|
+
## License
|
|
218
|
+
|
|
219
|
+
MIT — see [LICENSE](./LICENSE).
|
|
220
|
+
|
|
221
|
+
Bundles [marked](https://github.com/markedjs/marked) (MIT) and [DOMPurify](https://github.com/cure53/DOMPurify) (Apache-2.0 / MPL-2.0) in the frontend. Full notices in [`server/public/vendor/LICENSES/`](skills/llm-council/server/public/vendor/LICENSES/).
|
package/bin/install.js
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
'use strict';
|
|
3
|
+
|
|
4
|
+
const fs = require('fs');
|
|
5
|
+
const path = require('path');
|
|
6
|
+
const os = require('os');
|
|
7
|
+
|
|
8
|
+
const SKILL_NAME = 'llm-council';
|
|
9
|
+
|
|
10
|
+
function main() {
|
|
11
|
+
const command = process.argv[2] || 'install';
|
|
12
|
+
|
|
13
|
+
if (command === '--help' || command === '-h') {
|
|
14
|
+
console.log(`
|
|
15
|
+
ghcp-llm-council — Multi-model deliberation skill for GitHub Copilot CLI
|
|
16
|
+
|
|
17
|
+
Usage:
|
|
18
|
+
npx ghcp-llm-council install Install the skill (default)
|
|
19
|
+
npx ghcp-llm-council uninstall Remove the skill
|
|
20
|
+
npx ghcp-llm-council --help Show this help
|
|
21
|
+
`);
|
|
22
|
+
process.exit(0);
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
if (command === 'uninstall') {
|
|
26
|
+
uninstall();
|
|
27
|
+
} else {
|
|
28
|
+
install();
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
function getSkillDest() {
|
|
33
|
+
const home = os.homedir();
|
|
34
|
+
return path.join(home, '.copilot', 'skills', SKILL_NAME);
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
function install() {
|
|
38
|
+
const source = path.resolve(__dirname, '..', 'skills', SKILL_NAME);
|
|
39
|
+
const destRoot = path.join(os.homedir(), '.copilot', 'skills');
|
|
40
|
+
const dest = path.join(destRoot, SKILL_NAME);
|
|
41
|
+
|
|
42
|
+
if (!fs.existsSync(source)) {
|
|
43
|
+
console.error(`\x1b[31mError: Source skill not found at: ${source}\x1b[0m`);
|
|
44
|
+
process.exit(1);
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
fs.mkdirSync(destRoot, { recursive: true });
|
|
48
|
+
|
|
49
|
+
if (fs.existsSync(dest)) {
|
|
50
|
+
console.log('\x1b[33mReplacing existing installation...\x1b[0m');
|
|
51
|
+
fs.rmSync(dest, { recursive: true, force: true });
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
console.log(`\x1b[36mInstalling '${SKILL_NAME}' skill...\x1b[0m`);
|
|
55
|
+
copyDirSync(source, dest);
|
|
56
|
+
|
|
57
|
+
console.log(`\x1b[32m✓ Installed to: ${dest}\x1b[0m`);
|
|
58
|
+
console.log('\x1b[36mRestart your Copilot CLI session, then say: ask the council <your question>\x1b[0m');
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
function uninstall() {
|
|
62
|
+
const dest = getSkillDest();
|
|
63
|
+
|
|
64
|
+
if (!fs.existsSync(dest)) {
|
|
65
|
+
console.log('\x1b[33mSkill not installed — nothing to remove.\x1b[0m');
|
|
66
|
+
process.exit(0);
|
|
67
|
+
}
|
|
68
|
+
|
|
69
|
+
fs.rmSync(dest, { recursive: true, force: true });
|
|
70
|
+
console.log(`\x1b[32m✓ Removed skill from: ${dest}\x1b[0m`);
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
function copyDirSync(src, dest) {
|
|
74
|
+
fs.mkdirSync(dest, { recursive: true });
|
|
75
|
+
const entries = fs.readdirSync(src, { withFileTypes: true });
|
|
76
|
+
|
|
77
|
+
for (const entry of entries) {
|
|
78
|
+
const srcPath = path.join(src, entry.name);
|
|
79
|
+
const destPath = path.join(dest, entry.name);
|
|
80
|
+
|
|
81
|
+
if (entry.isDirectory()) {
|
|
82
|
+
copyDirSync(srcPath, destPath);
|
|
83
|
+
} else {
|
|
84
|
+
fs.copyFileSync(srcPath, destPath);
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
main();
|
package/package.json
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "ghcp-llm-council",
|
|
3
|
+
"version": "0.4.0",
|
|
4
|
+
"description": "A GitHub Copilot CLI skill that runs your question past a council of LLMs — multi-model deliberation with live UI.",
|
|
5
|
+
"author": "Vivekananda Gavini",
|
|
6
|
+
"license": "MIT",
|
|
7
|
+
"type": "commonjs",
|
|
8
|
+
"engines": { "node": ">=20" },
|
|
9
|
+
"bin": {
|
|
10
|
+
"ghcp-llm-council": "./bin/install.js"
|
|
11
|
+
},
|
|
12
|
+
"files": [
|
|
13
|
+
"bin/",
|
|
14
|
+
"skills/",
|
|
15
|
+
"LICENSE",
|
|
16
|
+
"README.md"
|
|
17
|
+
],
|
|
18
|
+
"keywords": [
|
|
19
|
+
"github-copilot",
|
|
20
|
+
"copilot-cli",
|
|
21
|
+
"copilot-skill",
|
|
22
|
+
"llm",
|
|
23
|
+
"multi-model",
|
|
24
|
+
"ai-council",
|
|
25
|
+
"deliberation",
|
|
26
|
+
"claude",
|
|
27
|
+
"gpt",
|
|
28
|
+
"llama",
|
|
29
|
+
"deepseek",
|
|
30
|
+
"mistral"
|
|
31
|
+
],
|
|
32
|
+
"scripts": {
|
|
33
|
+
"start": "node skills/llm-council/server/start.cjs"
|
|
34
|
+
},
|
|
35
|
+
"repository": {
|
|
36
|
+
"type": "git",
|
|
37
|
+
"url": "https://github.com/thevgavini/ghcp-llm-council.git"
|
|
38
|
+
},
|
|
39
|
+
"homepage": "https://github.com/thevgavini/ghcp-llm-council#readme",
|
|
40
|
+
"bugs": {
|
|
41
|
+
"url": "https://github.com/thevgavini/ghcp-llm-council/issues"
|
|
42
|
+
}
|
|
43
|
+
}
|
|
@@ -0,0 +1,256 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: llm-council
|
|
3
|
+
description: MUST USE whenever the user asks the council, asks a panel of LLMs, requests multiple model opinions on a question, says "ask the council", "council take", "get the council to weigh in", or wants several models to deliberate. Runs a 3-stage deliberation (parallel first opinions, anonymised peer review, chairman synthesis) and shows it live in a local web UI.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# llm-council
|
|
7
|
+
|
|
8
|
+
A panel of LLMs deliberates on the user's question across three stages — independent first opinions, anonymised peer review, and a chairman's synthesis — visualised live in a local web UI.
|
|
9
|
+
|
|
10
|
+
## When to use
|
|
11
|
+
|
|
12
|
+
Use this skill when the user:
|
|
13
|
+
|
|
14
|
+
- Says "ask the council X", "council: X", "get a council take on X", "multi-model opinion on X".
|
|
15
|
+
- Explicitly asks for several models to weigh in on a hard or judgment-heavy question.
|
|
16
|
+
- Says "follow up: X" or "ask the council a follow-up: X" — treat as a follow-up to the most recent conversation.
|
|
17
|
+
|
|
18
|
+
Do not use this skill for:
|
|
19
|
+
|
|
20
|
+
- Routine single-opinion questions.
|
|
21
|
+
- Tool-using tasks (code editing, file reading, command execution) — councillors are constrained to answer from their own knowledge only.
|
|
22
|
+
|
|
23
|
+
## The contract: you MUST complete all 3 stages before responding to the user
|
|
24
|
+
|
|
25
|
+
The single most common failure mode of this skill is the agent declaring "council convened" and stopping without actually running the deliberation. **Do not do this.** Every invocation must run all three stages end-to-end. The user is watching a browser UI; if you stop early, they see empty thinking-state cards forever.
|
|
26
|
+
|
|
27
|
+
## How it works — every step uses one helper
|
|
28
|
+
|
|
29
|
+
You do not compose HTTP calls or write to files directly. Use the bundled helper:
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
node <skill_dir>/bin/council.cjs <subcommand> [--flags]
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
`<skill_dir>` is the directory containing this `SKILL.md`. Every helper subcommand prints a single JSON line on stdout (success object or `{"error":"..."}`). Multi-line text payloads (councillor responses, ranker outputs, synthesis) are read from **stdin** to avoid shell escaping issues.
|
|
36
|
+
|
|
37
|
+
## Two backends per councillor
|
|
38
|
+
|
|
39
|
+
Each councillor (and the chairman) has a `backend` field:
|
|
40
|
+
|
|
41
|
+
- **`task`** — dispatched by you via the `task` tool with `model` set to the councillor's `id`. Use this for Anthropic + OpenAI models exposed through ghcp CLI. No extra credentials needed (uses the user's Copilot inference).
|
|
42
|
+
- **`github-models`** — dispatched via the helper's `call` subcommand, which posts to `models.github.ai/inference/chat/completions` using the user's `gh auth token`. No extra credentials beyond `gh auth login`. Use this for Meta Llama, DeepSeek, Mistral, Microsoft Phi, Cohere — vendors NOT exposed through `task`.
|
|
43
|
+
|
|
44
|
+
The default council mixes both backends. Inspect the `council` array from `init`'s output: each entry has `id`, `vendor`, `display`, `backend`.
|
|
45
|
+
|
|
46
|
+
## Modes — pick the right prompt set for the question
|
|
47
|
+
|
|
48
|
+
Five modes are available; each ships its own `councillor` and `chairman` prompts tuned for the question shape:
|
|
49
|
+
|
|
50
|
+
| Mode | Use when the user asks for… | Triggering language |
|
|
51
|
+
|---|---|---|
|
|
52
|
+
| `general` (default) | Open-ended Q&A, explanations, opinions | "ask the council X" with no other framing |
|
|
53
|
+
| `review` | Code or change review, security analysis | "review", "audit", "what's wrong with this code", "find bugs in", "security check" |
|
|
54
|
+
| `design` | Architecture or technology choice | "design", "should I use X or Y", "what's the best approach for", "architecture for" |
|
|
55
|
+
| `plan` | Step-by-step implementation roadmap | "plan", "roadmap", "how would you build", "implementation plan for" |
|
|
56
|
+
| `research` | Learn about a topic; explain without recommending | "explain", "how does X work", "what is the state of", "deep dive into" |
|
|
57
|
+
|
|
58
|
+
### MANDATORY: pre-flight mode confirmation in the CLI
|
|
59
|
+
|
|
60
|
+
Before calling `init`, **you must offer the user a mode choice using the `ask_user` tool** so they can pick from a list without typing. Don't skip this even if the question seems obvious — the user might disagree with your read.
|
|
61
|
+
|
|
62
|
+
1. Classify the user's question into one of the five modes using the triggering language above. When the question doesn't fit any specialist mode, recommend `general`.
|
|
63
|
+
2. Use the `ask_user` tool with a `choices` array. Put the recommended mode first with "(Recommended)" appended. Example:
|
|
64
|
+
|
|
65
|
+
```
|
|
66
|
+
question: "Which council mode for this question?"
|
|
67
|
+
choices: ["design (Recommended)", "general", "review", "plan", "research"]
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
3. Map the user's selection to the mode name (strip the "(Recommended)" suffix if present). If the user provides freeform text that isn't a mode name, treat it as a clarification of the question — re-classify and ask again.
|
|
71
|
+
|
|
72
|
+
4. Once confirmed, proceed to Step 0 with `--mode <chosen>`.
|
|
73
|
+
|
|
74
|
+
For follow-ups (`follow up: ...`), skip this step — the conversation already has a mode and the helper inherits it automatically. If the user explicitly says "follow up but as X", pass `--mode X` on `follow-up`.
|
|
75
|
+
|
|
76
|
+
## File context — `--files` on init / follow-up
|
|
77
|
+
|
|
78
|
+
When the question is about specific code (especially in `review` mode), pass the files directly so every councillor sees the same source:
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
node <skill_dir>/bin/council.cjs init --mode review --files src/auth.cjs,src/middleware.cjs --question "Review these for security holes"
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
The helper reads each file, caps it at 50 KB (200 KB total across all `--files`), prepends a `--- FILE: <path> ---` block to the question, and stores the augmented question. The councillors then see the file contents inline. Don't paste files into the question text yourself — let the helper do it so size caps and the file-block framing are consistent.
|
|
85
|
+
|
|
86
|
+
If the user mentioned filenames in their question (e.g., "review src/auth.cjs"), grep them out and pass them via `--files` automatically — don't make the user re-type them.
|
|
87
|
+
|
|
88
|
+
## The mandatory loop
|
|
89
|
+
|
|
90
|
+
### Step 0 — Initialise
|
|
91
|
+
|
|
92
|
+
```
|
|
93
|
+
node <skill_dir>/bin/council.cjs init --question "<the user's exact question>" [--mode review|design|plan|research|general] [--files path1,path2,...]
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Parse JSON output: `url`, `conversation_id`, `turn_id`, `mode`, `prompts_dir`, `council`, `chairman`, `chairman_backend`, `councillor_timeout_seconds`, `min_responses_to_proceed`.
|
|
97
|
+
|
|
98
|
+
Tell the user once: **"Council convened at \<url\>. Watch live as the deliberation unfolds."**
|
|
99
|
+
|
|
100
|
+
> If the user said "follow up", use `follow-up --question "..." --cid <previous conversation id>` instead.
|
|
101
|
+
|
|
102
|
+
### Step 1 — First opinions
|
|
103
|
+
|
|
104
|
+
**For each councillor in `council`, do one of two things based on its `backend`:**
|
|
105
|
+
|
|
106
|
+
**Backend = `task`** (parallel, agent-driven):
|
|
107
|
+
- Dispatch a `task` sub-agent in **background** mode.
|
|
108
|
+
- `model` = the councillor's `id`. `agent_type` = `general-purpose`.
|
|
109
|
+
- `prompt` = entire contents of `<skill_dir>/<prompts_dir>/councillor.md` + a blank line + the user's exact question (where `<prompts_dir>` came from init's output, e.g. `prompts/review`).
|
|
110
|
+
- Group all `task`-backend councillor dispatches in a single response (parallel tool calls).
|
|
111
|
+
|
|
112
|
+
**Backend = `github-models`** (synchronous, helper-driven):
|
|
113
|
+
- For each one, run (on Windows PowerShell):
|
|
114
|
+
```powershell
|
|
115
|
+
$prompt = (Get-Content <skill_dir>/<prompts_dir>/councillor.md -Raw) + "`n`n" + $userQuestion
|
|
116
|
+
$resp = $prompt | node <skill_dir>/bin/council.cjs call --backend github-models --model <councillor.id>
|
|
117
|
+
```
|
|
118
|
+
- On bash / zsh:
|
|
119
|
+
```bash
|
|
120
|
+
printf '%s\n\n%s' "$(cat <skill_dir>/<prompts_dir>/councillor.md)" "$userQuestion" \
|
|
121
|
+
| node <skill_dir>/bin/council.cjs call --backend github-models --model <councillor.id>
|
|
122
|
+
```
|
|
123
|
+
- The helper prints the response text on stdout (latency metadata on stderr).
|
|
124
|
+
- These can be run sequentially (each takes 1-4s).
|
|
125
|
+
|
|
126
|
+
**As each councillor's response arrives (task or github-models), immediately PATCH the server:**
|
|
127
|
+
|
|
128
|
+
```powershell
|
|
129
|
+
$response | node <skill_dir>/bin/council.cjs patch-councillor `
|
|
130
|
+
--tid <turn_id> --cid <conversation_id> `
|
|
131
|
+
--id <councillor_id> --status ok --latency-ms <elapsed>
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
If a councillor times out, returns garbage, or errors, PATCH with `--status timeout|error|empty` and pipe the error message to stdin.
|
|
135
|
+
|
|
136
|
+
Once `min_responses_to_proceed` councillors have succeeded, advance:
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
node <skill_dir>/bin/council.cjs advance --tid <turn_id> --cid <conversation_id> --stage 2
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
If fewer than `min_responses_to_proceed` succeed, advance to `--stage -1`, tell the user, and stop.
|
|
143
|
+
|
|
144
|
+
### Step 2 — Anonymised peer review
|
|
145
|
+
|
|
146
|
+
Build label map: `Response A` → 1st surviving councillor's id, `Response B` → 2nd, etc.
|
|
147
|
+
|
|
148
|
+
```
|
|
149
|
+
node <skill_dir>/bin/council.cjs set-label-map --tid <turn_id> --cid <conversation_id> --map '{"Response A":"claude-sonnet-4.6", "Response B":"gpt-5.2", ...}'
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
**For each surviving councillor (acting as a ranker), dispatch in its native backend** (same task vs github-models split):
|
|
153
|
+
|
|
154
|
+
The ranker prompt is `<skill_dir>/prompts/ranker.md` (shared across modes — the anonymised ranking process is the same regardless of question type) with `{{QUESTION}}` and `{{RESPONSES}}` substituted (responses labelled A, B, C, D etc joined by blank lines).
|
|
155
|
+
|
|
156
|
+
As each ranking returns:
|
|
157
|
+
|
|
158
|
+
```powershell
|
|
159
|
+
$rankingText | node <skill_dir>/bin/council.cjs patch-ranking `
|
|
160
|
+
--tid <turn_id> --cid <conversation_id> `
|
|
161
|
+
--ranker <model_id>
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
(The helper parses `FINAL RANKING:` automatically.)
|
|
165
|
+
|
|
166
|
+
Once all rankings are in:
|
|
167
|
+
|
|
168
|
+
```
|
|
169
|
+
node <skill_dir>/bin/council.cjs aggregate --tid <turn_id> --cid <conversation_id>
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
### Step 3 — Chairman synthesis
|
|
173
|
+
|
|
174
|
+
Single dispatch using `chairman` + `chairman_backend` from init's output.
|
|
175
|
+
|
|
176
|
+
The chairman prompt is `<skill_dir>/<prompts_dir>/chairman.md` (mode-specific, from init's output) with `{{QUESTION}}`, `{{STAGE1}}` (each councillor's response prefixed `Model: <id>\nResponse: <text>\n\n`), and `{{STAGE2}}` (each ranker's raw text prefixed `Model: <id>\nRanking: <raw>\n\n`).
|
|
177
|
+
|
|
178
|
+
If `chairman_backend == task`, dispatch a `task` sub-agent. If `github-models`, use the `call` helper.
|
|
179
|
+
|
|
180
|
+
When the chairman returns:
|
|
181
|
+
|
|
182
|
+
```powershell
|
|
183
|
+
$synthesis | node <skill_dir>/bin/council.cjs synthesize `
|
|
184
|
+
--tid <turn_id> --cid <conversation_id> `
|
|
185
|
+
--model <chairman_id>
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
### Step 4 — Respond to the user
|
|
189
|
+
|
|
190
|
+
After Step 3, tell the user:
|
|
191
|
+
|
|
192
|
+
> "Synthesis ready at \<url\>. Final answer below."
|
|
193
|
+
|
|
194
|
+
Then paste the chairman's synthesis as markdown in the chat.
|
|
195
|
+
|
|
196
|
+
**You are NOT done before this step.**
|
|
197
|
+
|
|
198
|
+
## Follow-ups
|
|
199
|
+
|
|
200
|
+
When the user says "follow up: X" or similar, treat as a new turn on the most recent conversation:
|
|
201
|
+
|
|
202
|
+
```
|
|
203
|
+
node <skill_dir>/bin/council.cjs follow-up --question "X" --cid <previous_conversation_id>
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Then run Steps 1–4 exactly as above. Each turn is independent — councillors do not see prior turns' content. (This matches Karpathy's reference design.)
|
|
207
|
+
|
|
208
|
+
## Reading browser-side events (optional, advanced)
|
|
209
|
+
|
|
210
|
+
If you want to support a future browser-initiated input mode, you can periodically read the events file:
|
|
211
|
+
|
|
212
|
+
```
|
|
213
|
+
node <skill_dir>/bin/council.cjs read-events
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
Returns `{"events":[...]}` and drains the file atomically. The browser has no composer yet — this returns empty in normal operation; the subcommand is kept for future iteration.
|
|
217
|
+
|
|
218
|
+
## Failure handling
|
|
219
|
+
|
|
220
|
+
- **All councillors fail Stage 1.** Advance to `--stage -1` with an error PATCH on the turn. Tell the user; ask whether to retry.
|
|
221
|
+
- **Some councillors fail.** Continue with survivors as long as `min_responses_to_proceed` is met. Browser shows failed cards with retry buttons (browser-initiated retry isn't wired up — the user can re-ask in the CLI).
|
|
222
|
+
- **A ranker's text is unparseable** (no `FINAL RANKING:` section + no fallback matches). The helper's parser handles this — it returns an empty array, which the aggregator silently excludes. The raw text stays visible in the UI.
|
|
223
|
+
- **Chairman fails.** Re-dispatch once. If it fails twice, synthesise a short notice ("Chairman failed; raw councillor responses and rankings are available above") and PATCH that as the synthesis text so the conversation can still persist.
|
|
224
|
+
- **Server died mid-loop.** `init` will detect and restart on next call. For in-flight state loss, re-init with the same question (a duplicate conversation is acceptable; the user can delete from the sidebar later — that UX doesn't exist yet, so just acknowledge in chat).
|
|
225
|
+
- **Config references a `task` model you don't support.** PATCH the councillor with `--status error` and a message; continue with the rest. Tell the user to edit `<cwd>/.llm-council/state/council.json` (or `<skill_dir>/defaults/council.json`).
|
|
226
|
+
|
|
227
|
+
## Quick command reference
|
|
228
|
+
|
|
229
|
+
| Helper | Purpose |
|
|
230
|
+
|---|---|
|
|
231
|
+
| `init --question "..." [--mode general\|review\|design\|plan\|research] [--files a,b,c]` | Start server (if needed), create conversation + turn, return ids + `mode` + `prompts_dir` + council config |
|
|
232
|
+
| `follow-up --question "..." --cid <cid> [--mode ...] [--files ...]` | New turn on existing conversation; inherits the conversation's mode unless overridden |
|
|
233
|
+
| `doctor [--deep] [--json] [--timeout-ms N]` | Probe every councillor + chairman. github-models backends get a real ping; task backends are listed as `~ skipped` (only the agent can probe those). Prints a table + a JSON sentinel. Run before init when diagnosing missing-councillor issues. |
|
|
234
|
+
| `call --backend github-models --model <id> [--max-tokens N] [--timeout-ms N]` (stdin: prompt) | Synchronously call a GitHub Models model. Returns response text on stdout, `{latency_ms, usage}` on stderr |
|
|
235
|
+
| `patch-councillor --tid X --cid Y --id <model> --status ok --latency-ms N` (stdin: response) | Push one councillor's response |
|
|
236
|
+
| `advance --tid X --cid Y --stage N` | Transition turn to next stage |
|
|
237
|
+
| `set-label-map --tid X --cid Y --map '{...}'` | Set anonymisation map |
|
|
238
|
+
| `patch-ranking --tid X --cid Y --ranker <model>` (stdin: raw text) | Push one ranking; parser runs automatically |
|
|
239
|
+
| `aggregate --tid X --cid Y` | Compute + PATCH aggregate rankings |
|
|
240
|
+
| `synthesize --tid X --cid Y --model <chairman>` (stdin: synthesis md) | PATCH stage 3, persists to disk |
|
|
241
|
+
| `read-events` | Drain browser-side events file (future use) |
|
|
242
|
+
| `status` | Show server info + conversation count |
|
|
243
|
+
|
|
244
|
+
## Per-session state lives under `<cwd>/.llm-council/`
|
|
245
|
+
|
|
246
|
+
```
|
|
247
|
+
.llm-council/
|
|
248
|
+
state/
|
|
249
|
+
server-info (URL, port, pid)
|
|
250
|
+
server.pid
|
|
251
|
+
server.log
|
|
252
|
+
events (JSONL, browser-side events for future use)
|
|
253
|
+
council.json (runtime config override)
|
|
254
|
+
conversations/
|
|
255
|
+
<conv-id>.json (persisted on PATCH stage:3)
|
|
256
|
+
```
|