@tiledesk/tiledesk-server 3.0.1 → 3.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (140) hide show
  1. package/.circleci/config.yml +1 -1
  2. package/CHANGELOG.md +47 -0
  3. package/Dockerfile +3 -2
  4. package/Dockerfile-en +3 -2
  5. package/Dockerfile-jobs +3 -2
  6. package/Dockerfile-profiler +3 -2
  7. package/README.md +15 -0
  8. package/app.js +12 -1
  9. package/bin/quota-reset +68 -0
  10. package/bin/quota-restore +86 -0
  11. package/channels/chat21/chat21Handler.js +50 -34
  12. package/channels/chat21/chat21JoinTracker.js +43 -0
  13. package/config/database.js +5 -5
  14. package/config/native_mcp/native_mcp_servers.json +1 -0
  15. package/controllers/chatbot.controller.js +80 -27
  16. package/controllers/intents.controller.js +42 -0
  17. package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-chat21-ionic.md +468 -0
  18. package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-design-studio.md +1349 -0
  19. package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-server.md +1694 -0
  20. package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-tybot.md +789 -0
  21. package/docs/superpowers/plans/2026-09-29-hitl-known-limits-server.md +83 -0
  22. package/docs/superpowers/plans/2026-09-29-hitl-known-limits-tybot.md +76 -0
  23. package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-chat21-ionic.md +97 -0
  24. package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-design-studio.md +72 -0
  25. package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-server.md +223 -0
  26. package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-tybot.md +121 -0
  27. package/docs/superpowers/plans/2026-09-29-hitl-pre-branch-ports.md +65 -0
  28. package/docs/superpowers/plans/2026-09-30-flow-start-points-design-studio.md +88 -0
  29. package/docs/superpowers/plans/2026-09-30-flow-start-points-server.md +103 -0
  30. package/docs/superpowers/plans/2026-09-30-flow-start-points-tybot.md +43 -0
  31. package/docs/superpowers/plans/2026-10-01-scheduled-start-point-design-studio.md +80 -0
  32. package/docs/superpowers/plans/2026-10-01-scheduled-start-point-runtime.md +39 -0
  33. package/docs/superpowers/plans/2026-10-01-scheduled-start-point-server.md +106 -0
  34. package/docs/superpowers/specs/2026-09-15-webhook-chatbot-hitl-design.md +715 -0
  35. package/docs/superpowers/specs/2026-09-30-flow-start-points-design.md +156 -0
  36. package/docs/superpowers/specs/2026-10-01-scheduled-start-point-design.md +303 -0
  37. package/docs/superpowers/todo/2026-09-14-webhook-chatbot-shared-code-fixes.md +191 -0
  38. package/event/botEvent.js +12 -1
  39. package/event/emailEvent.js +3 -0
  40. package/event/messageEvent.js +44 -0
  41. package/jobs.js +10 -0
  42. package/jobsManager.js +19 -0
  43. package/middleware/errorHandler.js +6 -2
  44. package/models/analyticResult.js +1 -1
  45. package/models/channelConstants.js +3 -2
  46. package/models/dataTable.js +16 -0
  47. package/models/location.js +6 -12
  48. package/models/quotaUsage.js +71 -0
  49. package/models/request.js +71 -61
  50. package/models/requestSnapshot.js +12 -0
  51. package/models/routerLogger.js +1 -1
  52. package/models/webhook.js +44 -0
  53. package/package.json +9 -7
  54. package/pubmodules/emailNotification/requestNotification.js +14 -22
  55. package/pubmodules/events/test/eventRoute.js +7 -6
  56. package/pubmodules/rules/conciergeBot.js +9 -4
  57. package/routes/auth.js +1 -12
  58. package/routes/campaigns.js +0 -13
  59. package/routes/dataTable.js +23 -3
  60. package/routes/faq_kb.js +12 -1
  61. package/routes/messagesRoot.js +0 -1
  62. package/routes/quotes.js +38 -1
  63. package/routes/request.js +81 -5
  64. package/routes/users.js +0 -1
  65. package/routes/webhook.js +160 -8
  66. package/routes/webhooks.js +276 -6
  67. package/routes/widgetLoader.js +6 -3
  68. package/services/QuoteManager.js +77 -16
  69. package/services/aiService.js +1 -1
  70. package/services/chatbotService.js +55 -1
  71. package/services/dataTableService.js +436 -42
  72. package/services/departmentService.js +60 -43
  73. package/services/dolphinSchedulerClient.js +116 -0
  74. package/services/emailService.js +6 -1
  75. package/services/hitlParticipantListener.js +260 -0
  76. package/services/participantInviteService.js +302 -0
  77. package/services/participantRemoveService.js +79 -0
  78. package/services/quotaSnapshotService.js +1167 -0
  79. package/services/requestService.js +102 -144
  80. package/services/scheduledStart/dolphinBackend.js +246 -0
  81. package/services/scheduledStart/schedulerBackend.js +101 -0
  82. package/services/scheduledStart/schedulerServiceClient.js +41 -0
  83. package/services/scheduledStartPublishListener.js +186 -0
  84. package/services/scheduledStartRules.js +173 -0
  85. package/services/scheduledStartService.js +130 -0
  86. package/services/startPointCleanupListener.js +30 -0
  87. package/services/startPointService.js +158 -0
  88. package/services/testWsService.js +1 -4
  89. package/services/updateRequestSnapshotQueued.js +8 -9
  90. package/services/webhookConversationService.js +185 -0
  91. package/services/webhookSettingsValidator.js +61 -0
  92. package/template/email/checkpointReachedEmail.html +5 -0
  93. package/test/agentPromptsRoute.test.js +3 -2
  94. package/test/agentTokensQuota.js +237 -0
  95. package/test/botEventHitlFallback.js +19 -0
  96. package/test/chat21HandlerJoinWiring.js +59 -0
  97. package/test/dataTablesRoute.test.js +649 -2
  98. package/test/dateUtils.test.js +0 -3
  99. package/test/departmentService.js +8 -12
  100. package/test/{fileRoute.js → deprecated/_deprecated_fileRoute} +2 -2
  101. package/test/{kbsettingsRoute.js → deprecated/_deprecated_kbsettingsRoute} +3 -3
  102. package/test/faqRoute.js +6 -1
  103. package/test/faqkbRoute.js +14 -10
  104. package/test/faqkbV3Route.js +1 -1
  105. package/test/filepRoute.js +4 -50
  106. package/test/hitlParticipantListener.js +446 -0
  107. package/test/hitlQueueWiring.js +191 -0
  108. package/test/hitlWaitingReset.js +124 -0
  109. package/test/inviteParticipantsRoute.js +492 -0
  110. package/test/kbRoute.js +1 -1
  111. package/test/messageRoute.js +33 -21
  112. package/test/nativemcp.test.js +7 -1
  113. package/test/openaiRoute.js +2 -8
  114. package/test/publishFailLoud.test.js +414 -0
  115. package/test/quotaSnapshots.js +1044 -0
  116. package/test/removeParticipantsRoute.js +186 -0
  117. package/test/requestService.js +24 -16
  118. package/test/requestServiceRemoveParticipant.js +144 -0
  119. package/test/requestVisibility.test.js +186 -0
  120. package/test/scheduledStartFacade.js +184 -0
  121. package/test/scheduledStartHandover.js +95 -0
  122. package/test/scheduledStartPublish.js +320 -0
  123. package/test/scheduledStartRoute.js +163 -0
  124. package/test/scheduledStartRules.js +177 -0
  125. package/test/scheduledStartRun.js +299 -0
  126. package/test/scheduledStartSchedulerBackend.js +116 -0
  127. package/test/scheduledStartService.js +290 -0
  128. package/test/startPointsCleanup.js +69 -0
  129. package/test/startPointsRegression.js +169 -0
  130. package/test/startPointsRoute.js +158 -0
  131. package/test/webhookConversationRoute.js +347 -0
  132. package/test/webhookConversationService.js +261 -0
  133. package/test/webhookConversationTestUtils.js +55 -0
  134. package/utils/aiUtils.js +95 -1
  135. package/utils/dataTableUtils.js +99 -9
  136. package/utils/intentFieldNames.js +192 -0
  137. package/utils/quotaCliUtil.js +43 -0
  138. package/utils/requestVisibility.js +355 -0
  139. package/websocket/webSocketServer.js +36 -31
  140. /package/test/{imageRoute.js → deprecated/_deprecated_imageRoute} +0 -0
@@ -0,0 +1,715 @@
1
+ # Webhook-started conversational chatbots with human-in-the-loop (HITL) — Design
2
+
3
+ - **Status:** approved design (brainstorming), v1 and §11 implemented; §12 (known-limit corrections and pre-branch ports) approved 2026-09-29, ready for implementation planning
4
+ - **Date:** 2026-09-15
5
+ - **Feature branch (all repos):** `feature/webhook-chatbot-hitl`, created from each repo's default branch
6
+ - **Related:** [`../todo/2026-09-14-webhook-chatbot-shared-code-fixes.md`](../todo/2026-09-14-webhook-chatbot-shared-code-fixes.md) (out-of-scope fixes found during design)
7
+
8
+ | Repo | Base branch | Verified at |
9
+ |---|---|---|
10
+ | tiledesk-server | `master` | `5d240683` (2.24.2) |
11
+ | tiledesk-chatbot (tybot, `tybotRoute/`) | `main` | `ce06bf8e` (2.1.13) |
12
+ | design-studio | `master` | `c5972d94` (1.40.14) |
13
+ | chat21-ionic | `master` | `a7894025` (3.4.35) |
14
+ | tiledesk-dashboard | — | not modified |
15
+
16
+ All `file:line` references below were verified on these commits.
17
+
18
+ ---
19
+
20
+ ## 1. Problem
21
+
22
+ Today only **automation** chatbots (`faq_kb.subtype: 'webhook'`) can be started by a webhook. They run
23
+ headless (`automation-request-…` ids, tybot `/block` endpoint) and cannot hold a conversation with a human.
24
+
25
+ We need a **conversational** chatbot (`subtype: 'chatbot'`) that can be started **either** by an end user
26
+ through the web widget **or** by an external event through a webhook, and that — once started by the
27
+ webhook — can **pause and ask a platform user** (project agent/admin, not an external lead) for
28
+ human-in-the-loop feedback in the platform chat, then **resume** with that answer.
29
+
30
+ Reference use case (AIOps): Grafana alert → webhook → flow enriches and triages → invites an on-call
31
+ engineer → asks "Restart checkout-api? [Approve] [Reject]" → engineer answers in the agent chat → flow
32
+ remediates or escalates.
33
+
34
+ ### Why the current platform can't do it
35
+ - **Agent Handoff removes the bot.** `requestService.routeInternal` replaces the participant list with the
36
+ routed agent (`services/requestService.js:166-240`), so the flow can never resume.
37
+ - **The support chat assumes an external requester (lead)** and a bot-owned conversation notifies nobody.
38
+ - **Auto-close kills waiting conversations.** On stage both close jobs are enabled with a 20-minute timeout
39
+ (`tiledesk-config`: `CLOSE_*_UNRESPONSIVE_REQUESTS_AFTER_TIMEOUT=1200000`).
40
+
41
+ ## 2. Goals / non-goals
42
+
43
+ **Goals**
44
+ 1. Webhook option on the **start block** of a conversational chatbot; the webhook call starts the flow
45
+ exactly like the widget does (start block, same runtime path).
46
+ 2. A flow action **Invite human** that adds platform user(s) to the conversation **while the bot stays**.
47
+ 3. Platform users answer in **chat21-ionic**; the flow resumes with their answer (free text or buttons).
48
+ 4. Conversations waiting for a human are **not auto-closed**.
49
+ 5. Design Studio support: toggle, URLs, action, and a way to test the webhook start.
50
+
51
+ **Non-goals (v1)**
52
+ - Changes to automation (`subtype: 'webhook'`) behaviour, sync `web_response` for chatbots.
53
+ - Deduplication/correlation of repeated webhook calls (every call opens a new conversation).
54
+ - A dedicated all-in-one "Ask human" action, a "release human" action, lead mapping from payload.
55
+ - Changes to the dashboard Monitor view, push notifications, canned responses / Copilot behaviour.
56
+ - The shared-code fixes listed in the TODO file.
57
+
58
+ ## 3. Decisions (from brainstorming)
59
+
60
+ | # | Decision |
61
+ |---|---|
62
+ | D1 | HITL surface = **chat21-ionic** (stage runs `chat21/chat21-ionic:3.4.35-rc2` at `/chat/`; dashboard `CHAT_BASE_URL=/chat/`). Monitor stays for context/history. `tiledesk-agent-chat` (`/agent/`) is the agent-runtime UI and is not involved. |
63
+ | D2 | Webhook is an option **on the start block**; enabling it creates the webhook endpoint if missing. The call starts the **normal flow as the widget** would (system `/start` → start block). |
64
+ | D3 | Requester = **synthetic source lead** per webhook (no external user). |
65
+ | D4 | Who answers: **department + explicit override**. Department → **one** available agent. Explicit `members` → **all** named people. |
66
+ | D5 | Flow building block: **new "Invite human" action + reuse** Reply V2 / Capture User Reply for the question. |
67
+ | D6 | Architecture: **server starts the conversation and performs invites**; tybot only calls the server. |
68
+ | D7 | Webhook calls for chatbots are **async only** (`202 {request_id}`). |
69
+ | D8 | Waiting conversations are **protected from auto-close** via `workingStatus: 'pending'` until a human answers. |
70
+ | D9 | Shared-code fixes (department dropped on create, `hasBot` forced false, tybot `main` lock issues, pre-existing webhook bugs) are **not** part of v1; tracked in the TODO file. |
71
+ | D10 | Development starts from each repo's default branch on `feature/webhook-chatbot-hitl`. |
72
+ | D11 | **Backward compatibility:** the conversation path is selected by an explicit `webhook.mode === 'conversation'` (set by the start-block toggle), **not** by the bot subtype. Webhooks without `mode` keep today's automation behaviour for every bot subtype (existing chatbot-subtype webhooks, e.g. the AIOps PoC, and test `run-published-webhook` rely on it). |
73
+
74
+ ## 4. Architecture and end-to-end flow
75
+
76
+ ```
77
+ Design Studio: start block → "Webhook" toggle ON
78
+ → POST /:projectid/webhooks {chatbot_id, block_id: <start intent_id>, mode: 'conversation'} (only if none exists)
79
+ → shows production + development URLs
80
+
81
+ External system ── POST /webhook/:webhook_id {payload}
82
+ ▼ tiledesk-server
83
+ 1. load webhook → mode 'conversation' (bot subtype 'chatbot') → conversation path
84
+ (no mode → existing automation path, unchanged for every bot subtype)
85
+ 2. enabled check, rate limit ('webhook' bucket)
86
+ 3. upsert source lead lead_id 'webhook_<webhook_id>'
87
+ 4. requestService.create: request_id support-group-<project>-<uuid>
88
+ participants ['bot_<chatbot_id>'] → status ASSIGNED (s_invite_bot_01 does not fire)
89
+ channel {name:'webhook'} (channelOutbound defaults to chat21 → chat21 group created)
90
+ attributes {webhook_id, payload:{...body, webhook_http_method, webhook_query_params}}
91
+ dev: attributes.sourcePage contains '&td_draft=true' → draft bot, quota skipped
92
+ 5. set request.department (targeted update; create drops it when participants are passed)
93
+ 6. messageService.send('system','Bot',request_id,'/start',id_project,null,{subtype:'info',updateconversation:false})
94
+ 7. 202 {success:true, request_id}
95
+ ▼ normal path: botEvent → BotSubscriptionNotifier → tybot /ext → start block
96
+ flow … Invite human {department and/or members}
97
+ ▼ tiledesk-server POST /:projectid/requests/:request_id/participants/invite
98
+ pick agent(s) → addParticipant (bot stays) → chat21 group join → email
99
+ workingStatus 'pending' + attributes.hitl_waiting
100
+ ▼ flow: Reply V2 "Restart? [Approve] [Reject]" (locks, optional no-input timeout)
101
+ platform user in chat21-ionic clicks/types
102
+ → message → botEvent → /ext → Reply V2 resumes (lastUserMessage.sender = responder)
103
+ → server resets workingStatus 'open' (human participant message while hitl_waiting)
104
+ ```
105
+
106
+ Existing mechanics this relies on (verified):
107
+ - Widget start: trigger `s_invite_bot_01` (`pubmodules/trigger/default.js:115-135`) → action
108
+ `request.department.bot.launch` → reroute + system `/start` info message (`pubmodules/trigger/rulesTrigger.js:717-785`, send at `:765-773`).
109
+ - `botEvent` passes system messages only for `\start` / `/start`; bot resolved from `participants`
110
+ (`event/botEvent.js:50-56`, `getBotId` `:212`).
111
+ - tybot resolves `/start` by intent name `start` (`tybotRoute/engine/TiledeskChatbot.js:135-140`,
112
+ `engine/MongodbBotsDataSource.js:93-101`); `/ext` does not skip info messages (`tybotRoute/index.js:45-237`).
113
+ - tybot merges `request.attributes.payload` into flow variables: each key + `payload`
114
+ (`tybotRoute/utils/TiledeskChatbotUtil.js:850-869`).
115
+ - Draft routing: `BotSubscriptionNotifier` switches to the draft bot when `request.attributes.sourcePage`
116
+ contains `&td_draft=true` (`services/BotSubscriptionNotifier.js:30-34`); `requestService.create` skips
117
+ quota for `td_draft=true` (`services/requestService.js:583`).
118
+ - chat21 group creation gated on `channelOutbound.name === 'chat21'` (`channels/chat21/chat21Handler.js:527-532`);
119
+ participant join → `chat21.groups.join` (`:813-842`); messages carry `attributes.request_channel` (`:329`).
120
+
121
+ **Distinguishing webhook vs widget start inside the flow:** both enter the start block. Webhook starts
122
+ have the `webhook_http_method` (and `payload`) variables set; widget starts don't. Flow designers branch
123
+ on `webhook_http_method` (documented in the DS start-block webhook section).
124
+
125
+ ## 5. tiledesk-server
126
+
127
+ ### 5.1 Webhook record and management — `models/webhook.js`, `routes/webhooks.js`
128
+ - New optional fields on `WebhookSchema` (`models/webhook.js:5-44`):
129
+ - `mode: String`, enum `['conversation']` — absent = automation (today's behaviour). Only `'conversation'`
130
+ selects the new path (D11).
131
+ - `department_id: String` — department for created conversations (default: project default department).
132
+ - `source_name: String` — display name of the source lead (default: webhook `name`).
133
+ - `POST /` (`routes/webhooks.js:67-93`): validate that `chatbot_id` exists in `req.projectid` and is not
134
+ trashed (404 otherwise). Accept `mode`, `department_id`, `source_name`. When `mode === 'conversation'`:
135
+ the bot must be subtype `chatbot` (422 otherwise); if `block_id` is omitted resolve it to the intent named
136
+ `start` (422 if not found); `async` is ignored. Without `mode` the behaviour is unchanged.
137
+ - `PUT /:chatbot_id` (`:182-212`) and `PUT /update/:webhook_id` (`:214-245`): also accept `mode`,
138
+ `block_id`, `department_id` and `source_name` (same `mode` validation). Keep using `findOneAndUpdate`
139
+ (do **not** use `.save()` on existing documents — the pre-save hook at `models/webhook.js:51-66` never
140
+ calls `next()` for non-new docs).
141
+ - `POST /preload/:webhook_id` (`:95-112`): when `webhook.mode === 'conversation'`, generate
142
+ `support-group-<project>-<uuid>` instead of `automation-request-…` (same Redis key/TTL).
143
+ - One webhook per bot remains (unique index `{id_project, chatbot_id}`, `models/webhook.js:47`).
144
+
145
+ ### 5.2 Webhook call — `routes/webhook.js`
146
+ - `router.all('/:webhook_id')` (`:263-322`): after the existing checks (404, 422 disabled, 429 rate limit),
147
+ branch on `webhook.mode`:
148
+ - no `mode` → existing `webhookService.run(...)` unchanged (any bot subtype).
149
+ - `mode === 'conversation'` → load the bot (404 if missing/trashed) →
150
+ `webhookConversationService.start(webhook, chatbot, payload, {dev:false})` → `202 {success:true, request_id}`.
151
+ - `router.all('/:webhook_id/dev')` (`:324-357`), `mode === 'conversation'` branch only: require the preload (reuse
152
+ `NO_PRELOADED_DEV_REQUEST` → 422), **apply the `enabled` check and rate limit** (the automation dev route
153
+ currently skips both), use the preloaded request id and set draft `sourcePage`. **The preload is single-use:
154
+ the dev call consumes it** (deletes the Redis key right after reading it, before starting the conversation),
155
+ so a second dev call without a fresh preload gets `NO_PRELOADED_DEV_REQUEST` again instead of reusing the
156
+ same `request_id` and failing on the unique index.
157
+ - Analytics: emit `webhook.triggered` (prod only) with extra `mode: 'conversation'` and the real
158
+ `request_id` (`pubmodules/analytics-publisher/index.js:386-404` consumer tolerates extra fields).
159
+
160
+ ### 5.3 Conversation starter — new `services/webhookConversationService.js`
161
+ `start(webhook, chatbot, payload, { dev, request_id })`:
162
+ 1. **Lead:** `leadService.createIfNotExistsWithLeadId('webhook_' + webhook.webhook_id, webhook.source_name || webhook.name, undefined, webhook.id_project, 'system', { webhook_id })`
163
+ (signature `services/leadService.js:59`).
164
+ 2. **Request:** `requestService.create({ request_id: request_id || 'support-group-' + project + '-' + uuid, lead_id, lead, id_project, first_text: '/start', departmentid: webhook.department_id, participants: ['bot_' + webhook.chatbot_id], channel: { name: 'webhook' }, preflight: false, createdBy: 'system', attributes: { webhook_id, payload: { ...payload, webhook_http_method, webhook_query_params }, sourcePage: dev ? 'webhook://' + webhook.webhook_id + '?dev=true&td_draft=true' : undefined } })`.
165
+ The root bot id is used; its `url` points to the published copy (same as the widget). The dev
166
+ `sourcePage` must contain `&td_draft=true` (with the `&`): `BotSubscriptionNotifier` matches
167
+ `indexOf("&td_draft=true")` and `requestService.create` matches `includes("td_draft=true")`.
168
+ If `webhook.department_id` does not exist, `getOperators` rejects → 422 (§5.6).
169
+ MongoDB rejects document keys containing `.` or starting with `$`, which Kubernetes/Grafana-style
170
+ alert labels routinely have (e.g. `app.kubernetes.io/name`); before storage, payload keys are
171
+ rewritten recursively with `sanitizePayloadKeys` — every `.` becomes `_` and a leading `$` becomes
172
+ `_` — and if two keys collapse onto the same sanitized name, the later one wins.
173
+ 3. **Department (v1 workaround):** `requestService.create` leaves `department` undefined when participants
174
+ are passed (`services/requestService.js:609-627`, saved at `:661`). Resolve the department
175
+ (`webhook.department_id` or the project default) and set `department` on the created request with a
176
+ targeted `Request.findOneAndUpdate` before sending `/start`. The proper fix is TODO item 1.
177
+ 4. **Start:** `messageService.send('system', 'Bot', request_id, '/start', id_project, null, { subtype: 'info', updateconversation: false })`
178
+ (`services/messageService.js:17`).
179
+ 5. Return `{ request_id }`. If lead/request creation fails, do not send `/start`.
180
+
181
+ ### 5.4 Invite participants — `POST /:projectid/requests/:request_id/participants/invite`
182
+ Route in `routes/request.js` (mounted with `hasRoleOrTypes('agent', ['bot','subscription'])`, `app.js:621`;
183
+ bot tokens restricted to their own project by `middleware/has-role.js:43-69`). Logic in a new
184
+ `services/participantInviteService.js`.
185
+
186
+ Body: `{ department_id?: string, members?: string[] }` (ids or emails).
187
+
188
+ 1. Load request by `request_id` + project → 404 if missing; 409 if `status === 1000` (closed); 422 if the
189
+ request's channel is not `webhook` (Invite human is only for webhook-started conversations in v1).
190
+ 2. Select invitees:
191
+ - **members** given → resolve each to an **active** project user of agent role type (by `id_user` or
192
+ user email); invite **all** resolved. If none resolve → fall back to department.
193
+ - **department** (explicit `department_id`, otherwise `request.department`, otherwise default) →
194
+ `departmentService.getOperators(dept, project, true /* nobot */)` (`services/departmentService.js:186`):
195
+ - routing `assigned` → the single selected operator (round robin / advanced routing apply);
196
+ - routing `pooled` → **one** agent via `getRandomAvailableOperator(available_agents)` (`:604`),
197
+ excluding agents already participants before picking.
198
+ 3. Invitees already in `request.participants` are reported with `already_participant: true` and count as
199
+ invited (lets a flow "wait again" for the same human before a second question).
200
+ 4. No invitees → `200 { request_id, invited: [], reason: 'no_available_agents' }`.
201
+ 5. For each new invitee → `requestService.addParticipantByRequestId(request_id, project, id_user, options)`
202
+ (`services/requestService.js:1722-1830`). The bot stays. Known v1 side effect: `hasBot` becomes `false`
203
+ (`:1768`; TODO item 2) — message delivery is unaffected (`botEvent` resolves the bot from `participants`).
204
+ 6. **Waiting protection:** set `workingStatus: 'pending'` and `attributes.hitl_waiting: true` on the request,
205
+ emit `request.workingStatus.update` and `request.update`.
206
+ 7. **Email** each newly added invitee with `emailService.sendNewAssignedRequestNotification(email, request, project)`
207
+ (`services/emailService.js:385`), applying the same per-user/per-project notification opt-outs as
208
+ `sendAgentEmail` (`pubmodules/emailNotification/requestNotification.js:885`; factor the checks into a helper).
209
+ (`addParticipantByRequestId` does not emit `request.participants.update`, so no email is sent today.)
210
+ 8. Response (always a non-empty JSON body): `200 { request_id, department_id, invited: [{ id_user, fullname, already_participant }] }`.
211
+
212
+ ### 5.5 Waiting-protection reset — `event/messageEvent.js`
213
+ Add a listener next to the existing requester rule (`event/messageEvent.js:183-203`) on message creation:
214
+ if `request.attributes.hitl_waiting === true` **and** the sender is in `request.participantsAgents` **and**
215
+ (`message.attributes.subtype !== 'info'` **or** `message.attributes.action` is set) → set
216
+ `workingStatus: 'open'`, unset `attributes.hitl_waiting`, emit `request.workingStatus.update` + `request.update`.
217
+
218
+ Rationale: both close jobs exclude `workingStatus: 'pending'`
219
+ (`pubmodules/scheduler/tasks/closeBotUnresponsiveRequestTask.js:73-77`,
220
+ `closeAgentUnresponsiveRequestTask.js:65-69`); the existing reset only reacts to the requester, which in
221
+ these conversations is the source lead that never writes. chat21-ionic action buttons with
222
+ `show_reply=false` send an info message carrying `attributes.action` (chat21-ionic
223
+ `conversation-detail.page.ts:1871-1883`), hence the `action` clause. The routing-queue only refreshes
224
+ agent counters on `workingStatus` changes (`pubmodules/routing-queue/listenerQueued.js:296-316`).
225
+
226
+ ### 5.6 Errors
227
+ | Case | Response |
228
+ |---|---|
229
+ | Webhook not found / disabled / rate-limited | 404 / 422 / 429 |
230
+ | Bot missing or trashed | 404 |
231
+ | `mode: 'conversation'` on a bot that is not subtype `chatbot`, or no `start` block | 422 |
232
+ | Configured `department_id` not found | 422 `webhook misconfigured` |
233
+ | Request quota exceeded (`requestService.create` throws "Requests limits reached") | 403 |
234
+ | Dev call without preload | 422 `NO_PRELOADED_DEV_REQUEST` |
235
+ | Lead/request creation failure | 500, `/start` not sent |
236
+ | Invite: request missing / closed / foreign bot token | 404 / 409 / 403 |
237
+ | Invite: request's channel is not `webhook` | 422 |
238
+
239
+ ## 6. tybot — tiledesk-chatbot `main`
240
+
241
+ ### 6.1 Action JSON (saved by Design Studio)
242
+ ```json
243
+ {
244
+ "_tdActionType": "invite_human",
245
+ "_tdActionId": "<uuid>",
246
+ "departmentId": "<department _id>",
247
+ "members": "mario@acme.it, {{oncall_email}}",
248
+ "trueIntent": "#<intent_id>",
249
+ "falseIntent": "#<intent_id>",
250
+ "trueIntentAttributes": {},
251
+ "falseIntentAttributes": {}
252
+ }
253
+ ```
254
+ `departmentId` and `members` are optional; with both empty the server uses the conversation's department.
255
+
256
+ ### 6.2 Registration
257
+ - `tybotRoute/tiledeskChatbotPlugs/directives/Directives.js`: `static INVITE_HUMAN = "invite_human";`
258
+ - New `tybotRoute/tiledeskChatbotPlugs/directives/DirInviteHuman.js`.
259
+ - `tybotRoute/tiledeskChatbotPlugs/DirectivesChatbotPlug.js` handlers map (`:250-314`):
260
+ `[Directives.INVITE_HUMAN]: DirInviteHuman`.
261
+
262
+ ### 6.3 `DirInviteHuman` behaviour
263
+ Modelled on `DirIfOnlineAgentsV2` (`tiledeskChatbotPlugs/directives/DirIfOnlineAgentsV2.js`: `httpUtils.request`
264
+ with `JWT <token>` at `:181-221`, jumps via `DirIntent.intentDirectiveFor` + `callback(true)`), without
265
+ copying its bug at `:140`.
266
+ 1. No `trueIntent` and no `falseIntent` → log error, `callback()`.
267
+ 2. `requestId` starts with `automation-request-` → set `flowError` = `(Invite human) requires a conversation`,
268
+ jump to `falseIntent`, no HTTP call.
269
+ 3. Resolve `members`: `TiledeskChatbot.allParametersStatic(tdcache, requestId)` (`engine/TiledeskChatbot.js:571`)
270
+ + `new Filler().fill(members, variables)` (`tiledeskChatbotPlugs/Filler.js:7`); array-valued variables
271
+ expand to items; split on `,` `;` newline; trim; dedupe; drop empties.
272
+ 4. `POST ${API_ENDPOINT}/${projectId}/requests/${requestId}/participants/invite`
273
+ body `{ department_id: departmentId, members }` (`utils/HttpUtils.js:9-54` supports `json` bodies).
274
+ 5. Success → set variables `invited_humans` (array) and `invited_humans_count` (number);
275
+ count ≥ 1 → jump `trueIntent`, else `falseIntent`.
276
+ 6. HTTP error/timeout → flow log error, set `flowError`, jump `falseIntent`.
277
+ 7. Every jump ends with `callback(true)` (stop the rest of the block).
278
+ 8. Flow logs: `[Invite Human] Invited: <names> (department <name>)`, `[Invite Human] No human available`,
279
+ `[Invite Human] Error: …`.
280
+
281
+ ### 6.4 Not in the directive
282
+ - No waiting: the next block's **Reply V2** (buttons, `noInputIntent`/`noInputTimeout`,
283
+ `DirReplyV2.js:84-116`) or **Capture User Reply** waits.
284
+ - Responder identity: `lastUserMessage` (filled by `TiledeskChatbotUtil.lastUserMessageFrom`,
285
+ `utils/TiledeskChatbotUtil.js:663-678`) carries `sender` / `senderFullname`.
286
+ - Flow-design rule on `main` (see TODO item 3): a pending lock captures every incoming message, including
287
+ internal jumps, and Capture User Reply does not stop the block. Therefore: **Invite human before the
288
+ question block**, prefer **Reply V2 with buttons**, and put Capture User Reply **last** in its block.
289
+
290
+ ## 7. chat21-ionic — `master`
291
+
292
+ Conversations are identified by `attributes.request_channel === 'webhook'` (set server-side at
293
+ `channels/chat21/chat21Handler.js:329`; copied onto agent messages at chat21-ionic
294
+ `conversation-detail.page.ts:1157`).
295
+
296
+ 1. **Buttons stay clickable (required).** `isLastMessage` (`src/chat21-core/utils/utils-message.ts:78-83`)
297
+ compares against the literal last message, so the "X added" info line hides text/action buttons
298
+ (`message-attachment.component.html:5, :26`; rendered via
299
+ `ion-conversation-detail.component.html:207`). Change to "last message that is not an info message".
300
+ Applies to all conversations (also fixes widget-started ones).
301
+ 2. **Channel icon** for `webhook`: `ion-list-conversations.component.html:111-117`,
302
+ `header-conversation-detail.component.html:50-56`, new svg asset.
303
+ 3. **Archive confirmation** for `request_channel === 'webhook'` before closing
304
+ (`conversations-list.page.ts:1060-1147`): "This conversation was started by an automated flow. Closing it
305
+ stops the flow. Continue?"
306
+ 4. **Null guards** on `this.conversation` / `this.conversation.attributes` in `getLeadDetail`
307
+ (`conversation-detail.page.ts:1004-1044`, `:1012`) and `updateLeadInfo` (`:1365-1395`).
308
+
309
+ Notification: invite → chat21 group join + email; the bot's question after the invite triggers the
310
+ existing sound/tab blink (non-info message from another sender, `app.component.ts:1131-1149`).
311
+
312
+ ## 8. design-studio — `master`
313
+ Paths: `C = src/app/chatbot-design-studio`, `CV = C/cds-dashboard/cds-canvas`.
314
+
315
+ ### 8.1 Webhook service — `C/services/webhook-service.service.ts`
316
+ - `createStartWebhook(chatbot_id, block_id, opts?: { department_id?, source_name? })` → `POST /webhooks/`
317
+ with body `{ chatbot_id, block_id, mode: 'conversation', ...opts }`.
318
+ - `updateWebhookSettings(chatbot_id, patch: { enabled?, mode?, block_id?, department_id?, source_name? })` → `PUT /webhooks/:chatbot_id`.
319
+ - Do **not** use `deleteWebhook` (`:97`, it stops a dev preload) for the off toggle.
320
+
321
+ ### 8.2 Start block webhook section (subtype `chatbot` only)
322
+ - `CV/cds-panel-intent-detail/cds-panel-intent-detail.component.ts`: extend `initializeStart()` (`:100-103`)
323
+ to load the webhook (`getWebhook` `:129-140`, guarding a missing `webhook_id`).
324
+ - New start-only HTML section (not under `isWebhook`, which hides "Available to agents" at `:16-37`):
325
+ **Webhook** toggle; production + development URLs with copy/regenerate (reuse `copyText` `:213-235`,
326
+ `onRegenerateWebhook` `:166-182`); department select (`dashboardService.departments`); source name input;
327
+ hint about branching on `webhook_http_method`.
328
+ - Toggle state = webhook exists **and** `mode === 'conversation'` **and** `enabled`.
329
+ - Toggle ON: no webhook → `createStartWebhook(chatbot_id, startIntent.intent_id)`; webhook exists →
330
+ `updateWebhookSettings({ enabled: true, mode: 'conversation', block_id: startIntent.intent_id })`. If the
331
+ existing webhook had no `mode` (headless automation webhook bound to another block), show a confirmation
332
+ first: "This chatbot already has a webhook that runs a block without a conversation. Switching it will make
333
+ calls start conversations from the start block. Continue?". OFF: `updateWebhookSettings({ enabled: false })`
334
+ (mode kept).
335
+ - Leave untouched: lazy auto-create for blocks named `webhook` (`CV/cds-intent/cds-intent.component.ts:265-272`).
336
+ - Small webhook badge on the start block header when enabled (`panel-intent-header`, name-based logic `:72-81`).
337
+
338
+ ### 8.3 "Invite human" action (template: `ifonlineagentsv2`)
339
+ | Step | Location |
340
+ |---|---|
341
+ | `TYPE_ACTION.INVITE_HUMAN = 'invite_human'` | `C/utils-actions.ts` enum (`ONLINE_AGENTSV2` at `:33`) |
342
+ | `ACTIONS_LIST.INVITE_HUMAN` entry, `chatbot_types: [TYPE_CHATBOT.CHATBOT]`, category, icon, `status:'active'`, doc key | `C/utils-actions.ts` (template entry `:141`). **Enum key and ACTIONS_LIST key must be identical** (`CV/cds-intent/cds-intent.component.ts:751-766`). |
343
+ | `ActionInviteHuman` model (`departmentId?`, `members?`, `trueIntent`, `falseIntent`, `*IntentAttributes?`) | `src/app/models/action-model.ts` (template `:78-95`) |
344
+ | Factory branch | `C/services/intent.service.ts` `createNewAction` `:951` (template `:1025-1027`) |
345
+ | Component `cds-action-invite-human` (copy `CV/actions/list/cds-action-online-agents-v2/cds-action-online-agents.component.*`; drop radio/operating hours; keep department select, `/true` `/false` connectors `:118-119`, `onChangeSelect`/`onResetBlockSelect` `:187-215`) | new folder under `CV/actions/list/` |
346
+ | Members field: `cds-textarea` with `setAttributeBtn` (variable picker inserting `{{var}}`) | `CV/base-elements/textarea/textarea.component.ts:11, :29` |
347
+ | Module declarations/exports | `CV/actions/list/cds-actions.module.ts` (`:78`, `:120`, `:242`) |
348
+ | Block preview switch case | `CV/cds-intent/cds-intent.component.html:212-218` |
349
+ | Detail panel switch case | `CV/cds-panel-action-detail/cds-panel-action-detail.component.html:84-91` |
350
+ | Connector redraw | `C/services/connector.service.ts:522-548` |
351
+ | Icon | `C/services/icon.service.ts:229-232` + `src/assets/images/actions/invite_human.svg` |
352
+ | i18n (name, doc recommending "Invite human → question block", labels) | `src/assets/i18n/en.json` (`CDSActionList` `:861`, `NAME` `:862`, `DOC` `:918`); other locales have no `CDSActionList` |
353
+
354
+ Menu filtering by subtype needs no change (`src/app/utils/project-utils.ts:139-141`,
355
+ `CV/cds-panel-elements/cds-panel-elements.component.ts:125`).
356
+
357
+ ### 8.4 Test the webhook start
358
+ - New play menu item "Test webhook start" (`C/utils-menu.ts:28-31`), visible for chatbots with the webhook enabled.
359
+ - Start: `preloadWebhook(webhook_id)` → `support-group-…` `request_id`; show dev URL box (widen
360
+ `cds-header.component.html:110-121`); `getToken` (`cds-header.component.ts:519-537`) → `logService.starterLog`;
361
+ open **only** the logs panel for that `request_id` (do not go through the widget path at
362
+ `CV/cds-canvas.component.ts:1538-1544`; set `mesage_request_id`, `cds-canvas.component.html:107-112`);
363
+ highlight the start block (`CV/cds-intent/cds-intent.component.ts:196-201` currently only `webhook`).
364
+ - Close: `stopWebhook()` (`cds-header.component.ts:539-548`); extend the `isWebhook` guard at `:417`.
365
+
366
+ ## 9. Testing
367
+
368
+ ### 9.1 tiledesk-server (mocha + chai-http; `npm test` starts MongoDB; Redis for the rate limiter)
369
+ - New `test/webhookConversationRoute.js`: `mode:'conversation'` webhook create (block_id resolved from `start`;
370
+ missing/foreign bot 404; non-chatbot bot 422); **backward compatibility: a chatbot-subtype webhook without
371
+ `mode` still returns the automation response**; PUT settings (mode, block_id, department_id, source_name);
372
+ run → 202 + request (participants `bot_<id>`, channel `webhook`, department, `attributes.payload`,
373
+ `webhook_id`) + lead `webhook_<id>` + system `/start` info message; disabled 422; rate limit 429; dev without
374
+ preload 422; dev with preload → preloaded `support-group-…` id + draft; dev enforces enabled + rate limit.
375
+ Existing automation tests unchanged.
376
+ - New `test/inviteParticipantsRoute.js`: members (all valid invited, unknown ignored, fallback to department);
377
+ assigned → one; pooled → one; none → `invited: []` + reason; already participant flag; bot stays;
378
+ `workingStatus 'pending'` + `hitl_waiting`; closed 409; foreign-project bot token 403.
379
+ - Waiting reset: human participant text → `open` + marker removed; info without action → unchanged;
380
+ info with `attributes.action` → `open`; requester rule unchanged; close-task queries exclude `pending`.
381
+
382
+ ### 9.2 tybot (`tybotRoute`, mocha; in-process tilebot on 10001 + mock Tiledesk API on 10002 + bot fixture,
383
+ as in `test/conversation-replyv2_button_match_test.js`)
384
+ New `test/conversation-invite_human_bot.js` + `test/conversation-invite_human_test.js`:
385
+ invited → trueIntent (mock receives filled `members` and `department_id`); `invited: []` → falseIntent;
386
+ HTTP 500 → falseIntent + `flowError`; `automation-request-…` → falseIntent, no HTTP call; HITL round trip:
387
+ Invite → Reply V2 buttons → agent message "Approve" → approve block with `lastUserMessage.sender` = agent id.
388
+
389
+ ### 9.3 design-studio (`ng test`; CI does not run tests — run locally, plus `ng build`)
390
+ Specs: `webhook-service.service.spec.ts` (HTTP shape of new methods via `HttpTestingController`);
391
+ invite-human component (connectors, department selection); start panel toggle (create vs `enabled` update,
392
+ never `deleteWebhook`). `strictTemplates` is off, so template bindings need manual review.
393
+
394
+ ### 9.4 chat21-ionic (`ng test`)
395
+ New `src/chat21-core/utils/utils-message.spec.ts`: `isLastMessage` ignores trailing info messages.
396
+ Manual: `webhook` icon; archive confirmation only for `request_channel === 'webhook'`.
397
+
398
+ ### 9.5 End-to-end acceptance (local stack, or stage when deployment is decided)
399
+ | # | Scenario | Expected |
400
+ |---|---|---|
401
+ | A | AIOps happy path: `curl` prod URL with Grafana-like payload; flow start → branch `webhook_http_method` → enrich → Invite human (SRE) → Reply V2 [Approve][Reject] | 202 `request_id`; agent emailed; conversation in chat21-ionic with webhook icon; still open after > 20 min; Approve resumes the flow; `workingStatus` back to `open` |
402
+ | B | Same bot from the widget | Start block runs as before; `webhook_http_method` not set |
403
+ | C | No available agent in department | "No human available" branch |
404
+ | D | DS "Test webhook start" (dev URL) | Draft bot runs; logs panel shows the run; close stops preload |
405
+ | E | Existing automation webhook | Unchanged, including sync `web_response` |
406
+ | F | Disabled webhook / rate limit | 422 / 429 |
407
+
408
+ ## 10. Known limitations (v1)
409
+ - `hasBot` becomes `false` after an invite (TODO 2): affects `hasBot` consumers (analytics "with bot", which
410
+ close job would apply once the conversation is not `pending`), not message delivery.
411
+ - The `request.create` event/websocket CREATE is published before the department is set (TODO 1 workaround).
412
+ - Every webhook call opens a new conversation; no dedup.
413
+ - A conversation nobody answers is never auto-closed while `pending`; the flow's no-input timeout must
414
+ escalate/close.
415
+ - A second question later in the flow needs **Invite human** again right before it to restore protection.
416
+ - Payload keys become flow variables and can overwrite same-named variables (same as automation webhooks).
417
+ - tybot `main` lock/Capture User Reply behaviour (TODO 3) constrains flow design (§6.4).
418
+ - chat21-ionic `request_channel` on the list item comes from the latest conversation snapshot attributes;
419
+ a message without `request_channel` could reset the list icon (the detail page re-reads it from the request).
420
+ - Invite human is rejected (422) in conversations not started by a webhook; tybot takes the "no human
421
+ available" branch with `flowError`.
422
+
423
+ ---
424
+
425
+ ## 11. Participant removal and reassignment (v2 extension, 2026-09-29)
426
+
427
+ Status: approved in brainstorming on 2026-09-29; extends §5–§9. Repo bases: the feature branches after merging
428
+ the default branches on 2026-09-29 (tiledesk-server 2.26.0, tiledesk-chatbot 2.2.0, design-studio 1.40.15,
429
+ chat21-ionic 3.4.35).
430
+
431
+ ### 11.1 Goal and decisions
432
+ Goal: a chatbot started by a webhook interacts with a back-office operator, and people leave the conversation
433
+ cleanly at the right moment, without leaving the flow stuck.
434
+
435
+ Decisions (user, 2026-09-29):
436
+ 1. **The flow removes humans** once it has what it needed, with a new `remove_human` action whose scope is
437
+ either *only the humans this flow invited* or *all human participants*.
438
+ 2. **Operators can leave** a webhook conversation from chat21-ionic; leaving counts as a removal.
439
+ 3. **If the human the flow needs leaves or is removed, the server reassigns automatically** to another
440
+ available agent of the same department. If nobody is available, **the flow is told**: it jumps to the
441
+ "No human available" branch of the `invite_human` action that brought the human in, with a specific
442
+ `flowError`.
443
+ 4. **The bot is never removed automatically.** Full ownership ("pure human handoff") uses the existing
444
+ **Remove current bot** action (`removecurrentbot`) after `invite_human`— today reachable in Design Studio only as the `\removecurrentbot` text directive in a Reply (there is no dedicated editor action yet); pairing is what `invite_human`
445
+ already does. Deliberate removal of the bot by a person (dashboard, API) stays allowed.
446
+ 5. **Webhook-only boundary:** every flow-driven conversation enters through the webhook conversation start
447
+ (web interfaces, schedulers and integrations are callers of it), so invite, remove, reassignment and Leave
448
+ apply only to conversations with `request.channel.name === 'webhook'`. Widget conversations are untouched.
449
+ 6. **All new REST APIs are project scoped:** under `/:projectid/...`, behind the existing authentication and
450
+ role middleware; every read and write filters on `id_project` (conversations, departments, project users);
451
+ a conversation of another project answers 404.
452
+ 7. **Start points** (several start types per chatbot: web, scheduled, integration apps such as Google mail)
453
+ are a **separate sub-project** with its own design. This extension only prepares for it with
454
+ `attributes.start_type` (§11.2).
455
+
456
+ Approach: the server owns removal handling (one listener for every removal path, agent picking next to the
457
+ existing invite logic); tybot only calls endpoints and receives the fallback jump.
458
+
459
+ ### 11.2 Conversation data (`request.attributes`)
460
+ | Field | Written by | Meaning |
461
+ |---|---|---|
462
+ | `start_type` | conversation start (§5.3) | how the flow was started; always `'webhook'` in this version, reserved for start points |
463
+ | `hitl_waiting` | invite (§5.4), unchanged | a question is waiting for a human; cleared by the reply listener (§5.5) |
464
+ | `hitl_invited` | invite, reassignment | user ids invited by the flow that are still participants |
465
+ | `hitl_department_id` | invite | the resolved department used to pick agents (explicit, conversation or default) |
466
+ | `hitl_fallback` | invite, only when ≥1 human invited and the action has a "No human available" branch | `{ intent: "#<blockId>" }`, where the flow resumes when no human can be found |
467
+ | `hitl_left` | removal listener | user ids that left or were removed by someone other than the flow while the flow was waiting for a human; never picked again automatically for this conversation (department invites and replacements) |
468
+
469
+ `hitl_invited` includes invitees reported as `already_participant`. The reply listener (§5.5) clears only
470
+ `hitl_waiting`: the flow still "holds" its humans until it removes them.
471
+
472
+ ### 11.3 API (tiledesk-server)
473
+ 1. **Invite** — `POST /:projectid/requests/:request_id/participants/invite` gains an optional
474
+ `fallback_intent` (string, `#<blockId>`; 422 if not a string). When at least one human is invited the server
475
+ stores `hitl_invited` (union with the existing list), `hitl_department_id` and, if given, `hitl_fallback`.
476
+ Response unchanged.
477
+ 2. **Remove (new)** — `POST /:projectid/requests/:request_id/participants/remove`, same router and middleware
478
+ as invite (`app.js:621`, `hasRoleOrTypes('agent', ['bot','subscription'])`).
479
+ - Body: `{ scope: 'invited' | 'all_humans' }` (default `'invited'`; any other value → 422).
480
+ - 404 unknown conversation (in this project), 409 closed, 422 channel not `webhook`.
481
+ - First clears `hitl_waiting`, `hitl_invited`, `hitl_fallback` (atomic update filtered by `request_id` +
482
+ `id_project`), sets `workingStatus: 'open'` if it was `'pending'`, then removes each human in scope with
483
+ `requestService.removeParticipantByRequestId` (so chat21 group, activity and events behave as a normal
484
+ removal). The bot is never removed by this endpoint.
485
+ - Scope `invited` = the `hitl_invited` list read before clearing, intersected with current participants;
486
+ `all_humans` = every participant not prefixed `bot_`.
487
+ - Response `200 { request_id, removed: [{ id_user, fullname }] }`; an empty list is not an error.
488
+ 3. **Generic removal** — `DELETE /:projectid/requests/:request_id/participants/:participantid` keeps its API;
489
+ chat21-ionic Leave and the dashboard use it. Its new behaviour comes from the listener (§11.4). Removing the
490
+ bot through it stays allowed.
491
+
492
+ ### 11.4 Removal listener (new, e.g. `services/hitlParticipantListener.js`)
493
+ Subscribes to `request.participants.leave` (payload `{ member, request, actor, source }`,
494
+ `services/requestService.js`) and `request.participants.update` (`{ beforeRequest, removedParticipants,
495
+ addedParticipants }`), queue variants when `requestEvent.queueEnabled`. Acts only on webhook conversations that
496
+ are not closed.
497
+
498
+ **A human left** (`member` not prefixed `bot_`):
499
+ 1. Atomic `findOneAndUpdate({ request_id, id_project, 'attributes.hitl_invited': member },
500
+ { $pull: { 'attributes.hitl_invited': member }, $addToSet: { 'attributes.hitl_left': member } }, { new: true })`.
501
+ No match → stop (the human was not invited by the flow, or the flow's remove endpoint already released it).
502
+ 2. `hitl_invited` still non-empty → stop.
503
+ 3. Reassignment is attempted whether or not `hitl_fallback` is set. (Amended by §12.2.1: only while `hitl_waiting === true`; otherwise the leave just releases the human.)
504
+ 4. **Reassign**: pick one available agent of `hitl_department_id` with the invite service's picker
505
+ (§5.4 step 2), excluding `hitl_left` and current participants.
506
+ - **Found** → add as participant, email as an invite, `$addToSet` into `hitl_invited`; `hitl_waiting` and
507
+ `workingStatus` are left as they are (a pending question stays pending).
508
+ - **Not found** → clear `hitl_waiting`, `hitl_invited`, `hitl_fallback`, set `workingStatus: 'open'`, emit
509
+ `request.workingStatus.update` + `request.update`; then, only when `hitl_fallback` is set (otherwise only the
510
+ waiting state is cleared), send a system info message to the conversation with
511
+ `attributes.action = '#<blockId>{"flowError":"(Invite human) The invited human left and no replacement is available"}'`
512
+ (tybot's explicit-action syntax, `TiledeskChatbotUtil.parseIntent`; the text must contain no `'`, which
513
+ tybot rewrites to `"`). A send failure is logged; the conversation stays open and not waiting, so the
514
+ auto-close jobs can end it.
515
+ The message is sent with sender `system` and `attributes.hitl_fallback: true`; `event/botEvent.js` forwards
516
+ system messages to the bot only for `/start` today, so it must also forward a system message whose
517
+ `attributes.hitl_fallback === true` and that carries `attributes.action` (no other system message changes).
518
+
519
+ **Dashboard "replace participants"** (`participants.update` with added humans): the added humans are
520
+ `$addToSet` into `hitl_invited`, removed ones go through steps 1–2 only (no automatic pick, no fallback).
521
+
522
+ **The bot left**: clear `hitl_waiting`, `hitl_invited`, `hitl_fallback`, set `workingStatus: 'open'`; no message
523
+ to the bot. The remaining humans own the conversation.
524
+
525
+ Concurrency: step 1 is atomic, so when two invited humans leave together only the last one out reaches step 4.
526
+ A reply just before leaving clears only `hitl_waiting`; reassignment still runs (intended).
527
+
528
+ ### 11.5 tybot (tiledesk-chatbot)
529
+ - **`invite_human`** sends `fallback_intent: falseIntent` (as `#<blockId>`) in the invite body when the action
530
+ has a `falseIntent`. Nothing else changes.
531
+ - **New `remove_human`** directive (`Directives.REMOVE_HUMAN = 'remove_human'`), action JSON:
532
+ `{ _tdActionType: 'remove_human', scope: 'invited' | 'all_humans' (default 'invited'), trueIntent?, falseIntent?,
533
+ trueIntentAttributes?, falseIntentAttributes? }`.
534
+ - Calls `POST /:projectid/requests/:request_id/participants/remove` with the bot token and the same timeout
535
+ as `invite_human` (`INVITE_TIMEOUT_MS`, 30 s).
536
+ - Sets `removed_humans` (array of `{ id_user, fullname }`) and `removed_humans_count`.
537
+ - Success (also when nobody was removed) → `trueIntent`; HTTP error → `falseIntent` with
538
+ `flowError = "(Remove human) An error occurred: " + <server error text or message>`; automation request id →
539
+ `falseIntent` with `flowError = "(Remove human) requires a conversation"`. With no branch configured the
540
+ block continues with the next action.
541
+ - Logs only the error summary (`{ message, status, data }`), never the raw HTTP error (same rule as §6.3).
542
+ - **Fallback jump**: no engine change — a message with `attributes.action` unlocks the locked intent and runs the
543
+ named block with its parameters (`engine/TiledeskChatbot.js`, explicit action path).
544
+ - The lock/Capture User Reply fix (TODO item 3) is planned alongside as a separate change.
545
+
546
+ ### 11.6 chat21-ionic
547
+ - **Leave conversation**: an icon button on the conversation list item (visible on desktop and mobile; the archive button next to it is hidden on desktop since upstream 6460bede), only for `isWebhookConversation(conversation)` and not archived conversations.
548
+ - Confirmation alert: "You will leave this conversation. If a flow invited you, it will look for another operator or continue without you. Continue?" with Cancel / Leave; new keys in all 15 locale files.
549
+ - `TiledeskService.removeParticipant(requestId, userId, projectId)` → `DELETE /:projectid/requests/:requestId/participants/:userId`
550
+ with the logged-in user's id and token; on error a toast shows the server reason. The conversation leaves the
551
+ list when chat21 removes the member from the group.
552
+ - Archive (close) is unchanged.
553
+
554
+ ### 11.7 design-studio
555
+ - **"Remove human" action**, same pattern as "Invite human" (§8.3): `ActionRemoveHuman` (`scope`, `trueIntent`,
556
+ `falseIntent`, `trueIntentAttributes?`, `falseIntentAttributes?`), `TYPE_ACTION.REMOVE_HUMAN = 'remove_human'`
557
+ and `ACTIONS_LIST.REMOVE_HUMAN` (subtype `chatbot`, Most used), factory branch, component + spec, module,
558
+ both switch cases, connector redraw, icon `person_remove`, `en.json`.
559
+ - Panel: scope select ("Only humans invited by this flow" default, "All human participants"), outputs
560
+ "Removed" and "Error". Preview shows the scope and both connectors.
561
+ - Help texts: Invite human — "If the invited human leaves, another available agent is invited; if nobody is
562
+ available, the flow continues from No human available."; Remove human — "Removes human participants from the conversation while the chatbot keeps running: only the humans this flow invited, or all of them. Place it at the end of the part of the flow that needs a human: until they are removed, invited humans who leave are replaced automatically. The Removed output is also taken when there was nobody to remove. Available only in conversations started by a webhook. To hand the conversation over completely to the invited humans, remove the bot instead with a Reply whose text is \removecurrentbot."
563
+
564
+ ### 11.8 Testing
565
+ - **tiledesk-server** (mocha, test MongoDB + Redis): invite stores `fallback_intent`/`hitl_invited`/
566
+ `hitl_department_id` only when someone is invited, `start_type` set at start; remove endpoint scopes, bot kept,
567
+ fields cleared, no reassignment, 404/409/422, other-project conversation → 404; listener: reassign to another
568
+ agent (excluding `hitl_left`), no agent → fallback message with the exact `action`, other invited human remains
569
+ → no-op, non-invited human → no-op, dashboard replace with an added human → no pick, bot removed → state
570
+ cleared, concurrent leaves → one reassignment; existing webhook/invite/HITL suites still pass.
571
+ - **tybot**: `remove_human` unit (scope, variables, error summary without token, automation) and conversation
572
+ tests (removed → trueIntent, error → falseIntent with reason, `invite_human` sends `fallback_intent`, the
573
+ fallback `action` message unlocks a paused Reply V2 and lands in the fallback block with `flowError`).
574
+ - **chat21-ionic**: Leave visible only for open webhook conversations; `removeParticipant` URL; locale JSON valid;
575
+ build.
576
+ - **design-studio**: `remove_human` component spec; build.
577
+ - **End-to-end** (extends §9.5 scenario A), local stack with a department of agents A and B:
578
+ 1. webhook → flow invites A; A answers with a button; flow continues;
579
+ 2. flow invites again; A leaves from chat21-ionic → B invited automatically, answers, flow continues;
580
+ 3. flow `remove_human` scope `invited` → B removed, bot finishes;
581
+ 4. department with only A: A leaves → flow jumps to "No human available" with the new `flowError`;
582
+ 5. full ownership: `invite_human` then `removecurrentbot` → A owns the conversation, no longer pending.
583
+
584
+ ### 11.9 Limitations
585
+ - Reassignment picks from the invite's department only; explicit members are not re-invited individually.
586
+ - Without a "No human available" branch on `invite_human`, a failed reassignment only clears the waiting state
587
+ (the flow is not notified).
588
+ - Only the latest invite's fallback block is kept.
589
+ - `hitl_left` grows for the life of the conversation; a human who left is never picked again automatically
590
+ (a flow can still invite them explicitly by member).
591
+ - When the server's fallback message resumes the flow, its text becomes the flow's `lastUserText` (and `lastUserMessage.sender` is `system`, and it is appended to the transcript); AI Prompt / Ask KB actions in the fallback block that default to `{{lastUserText}}` receive that sentence.
592
+ - The server-driven fallback jump carries only `flowError`; the `invite_human` action's `falseIntentAttributes` are not applied on that path.
593
+ - An internal jump into the waiting action's own block is still consumed as its answer (lock ownership compares the action id, and the jumped-to block contains that same action).
594
+
595
+ ---
596
+
597
+ ## 12. Known-limit corrections and pre-branch ports (2026-09-29)
598
+
599
+ Status: approved in brainstorming on 2026-09-29; amends §11.4 (12.2.1) and adds pre-branch ports. Feature
600
+ branches at the start: tiledesk-server `c23e22ff`, tiledesk-chatbot `7d3e1a67`, chat21-ionic `15d4f61d`,
601
+ design-studio `9b9b80c2`.
602
+
603
+ ### 12.1 Decisions (user, 2026-09-29)
604
+ 1. **Replace only while the flow is waiting.** A departed invited human is replaced (or the flow is sent to its
605
+ fallback) only while the conversation waits for a human answer (`hitl_waiting === true`). Otherwise the
606
+ departure just releases that human. This supersedes §11.4 step 3 ("Reassignment is attempted whether or not
607
+ `hitl_fallback` is set" stays true, but only while waiting).
608
+ 2. **Pre-branch ports for all four repos.** Feature branches are merged separately into each repo's default
609
+ branch and its pre branch, so each repo gets a second local branch `feature/webhook-chatbot-hitl-PRE`, built
610
+ from the pre branch by cherry-picking the feature commits. Pre branches: tiledesk-server `master-COLLAUDO`,
611
+ tiledesk-chatbot `main_pre`, chat21-ionic `master-PRE`, design-studio `master-pre`.
612
+ 3. **Disk space is limited:** no new git worktrees; ports are done one repo at a time in the existing checkout.
613
+ 4. All the other known limits listed in 12.2–12.3 are corrected (no decision needed).
614
+
615
+ ### 12.2 tiledesk-server corrections
616
+ 1. **Replace only while waiting** (`services/hitlParticipantListener.js`). When the last invited human leaves:
617
+ - `hitl_waiting === true` → unchanged §11.4 behaviour (replacement from `hitl_department_id`, else clear the
618
+ waiting state and send the fallback if `hitl_fallback` is set);
619
+ - otherwise → the leaver is only pulled from `hitl_invited` (NOT added to `hitl_left`: it did not abandon a
620
+ pending question); when `hitl_invited` becomes empty, both `hitl_invited` and `hitl_fallback` are unset (the
621
+ flow's hold is released and tracking ends); no replacement, no message, no status change.
622
+ The decision reads `hitl_waiting` from the step-1 document (atomic with the leave), never from the event
623
+ payload.
624
+ 2. **Atomic participant removal** (`services/requestService.js` `removeParticipantByRequestId`). Replace the
625
+ load–modify–save with:
626
+ - one `findOneAndUpdate` filtered by `request_id`, `id_project` and `participants: member`, doing `$pull` on
627
+ `participants` and on `participantsBots` (bot member, without the `bot_` prefix) or `participantsAgents`
628
+ (human member), plus `$set` of `attributes.abandoned_by_project_users.<project_user_id>` (timestamp) for a
629
+ human member, and the `hasBot = false` change for a bot member as today; no match → resolve as the current
630
+ "already not found" branch does (no events);
631
+ - then, unless the request is closed, a second update setting `status` to assigned (participants remain) or
632
+ unassigned (none left), filtered on the resulting participant count so a concurrent change is not
633
+ overwritten;
634
+ - the emitted events (`request.update`, `request.update.comment`, `request.updated`,
635
+ `request.participants.leave`) and their payload shapes stay identical, built from the updated, populated
636
+ document.
637
+ The TODO entry "Concurrent participant removals can lose one removal" is marked done.
638
+ 3. **`clearWaiting` in one write per attempt** (`hitlParticipantListener.js`): two conditional updates, no
639
+ aggregation-pipeline update. Attempt A matches `workingStatus: 'pending'` and, in one write, unsets
640
+ `hitl_waiting`/`hitl_invited`/`hitl_fallback` and sets `workingStatus: 'open'`; attempt B (only when A did not
641
+ match) only unsets the three attributes. `request.workingStatus.update` is emitted only from A. The residual
642
+ gap needs a manual status change or an invite racing a bot removal, when no flow remains.
643
+ 4. **Replacement vs the flow's Remove human**: after `addParticipantByRequestId` for a replacement, the listener
644
+ re-reads `hitl_invited` atomically; if the replacement is no longer in it (the flow released its humans in the
645
+ meantime), the listener removes the replacement again via `removeParticipantByRequestId` and does not treat
646
+ that removal as a HITL leave (it is not in `hitl_invited`, so §11.4 step 1 already ignores it).
647
+
648
+ ### 12.3 tybot corrections
649
+ 1. **No token in logs** (`tybotRoute/utils/HttpUtils.js` `request`, TODO item 6). The error handler logs only
650
+ `{ message, code, status, url (without query string), data (response body) }`; it never logs the raw error,
651
+ `error.response`, `error.config` or `error.request`. The callback still receives the original error.
652
+ 2. **A lock belongs to the action that set it** (`DirCaptureUserReply`, `DirReplyV2.lockUnlock`). "Reply path"
653
+ only when the locked action id equals the action's own `_tdActionId`; a different locked action id means the
654
+ action was reached by an internal jump: it runs as a first pass (asks its question and takes over the lock).
655
+ 3. **Capture User Reply without block info** logs an error and stops the block (`callback(true)`) instead of
656
+ continuing past the question.
657
+ 4. **The no-input timer only fires for the action that owns the lock** (`DirReplyV2`). After the existing
658
+ `USER_INPUT === timeout_id` check the timer also requires `currentLockedAction === action._tdActionId`;
659
+ otherwise it logs at debug and does nothing (no unlock, no jump). A Reply V2 without `intent_info.intent_name`
660
+ logs an error and stops the block without locking.
661
+ 5. **Logger-level redaction** (`tybotRoute/utils/winston.js`, TODO item 7). Every transport runs a redaction
662
+ format before `simple()`: values of keys `authorization|token|x-api-key|apikey|api_key|gptkey|password|secret`
663
+ become `[REDACTED]`, and `JWT <token>` / `Bearer <token>` in any string, message included, become
664
+ `JWT [REDACTED]` / `Bearer [REDACTED]`. It works on copies (callers' objects are not mutated), is depth-limited
665
+ and cycle-safe, and also covers Error objects (message, stack, `config`, `response`).
666
+ 6. **Async `/block` error** (`tybotRoute/index.js`). On a `/ext` failure the async branch logs
667
+ `HttpUtils.errorSummary(err, url)` and answers `500 { success: false, error: <message> }`; the raw error (which
668
+ carries the bot token) is neither logged nor returned, so tiledesk-server forwarding the body is safe.
669
+
670
+ ### 12.4 Pre-branch ports
671
+ Per repo, one at a time, in the existing checkout:
672
+ 1. `git switch -c feature/webhook-chatbot-hitl-PRE origin/<pre branch>`; clean install from that branch's lockfile
673
+ into the existing `node_modules` (tiledesk-server `npm ci`; tiledesk-chatbot `npm ci` at the repo root;
674
+ chat21-ionic `npm ci --ignore-scripts` with Node 14; design-studio `npm ci` with Node 16).
675
+ 2. Cherry-pick, in order, every non-merge commit of the feature branch since it left the default branch (v1, v2
676
+ and the §12.2/§12.3 corrections), skipping the "Merge origin/<default> into feature/…" commits. Docs commits
677
+ exist only in tiledesk-server and are kept.
678
+ 3. Conflicts: keep the pre branch's code and add the feature's change; never drop pre-branch behaviour to make the
679
+ change fit. Each resolution is recorded in the plan ledger.
680
+ 4. tybot: `main_pre` sends internal jumps through `/exec` (`DirIntent` `tilebotService.executeBlock`) instead of
681
+ `/ext`; the lock fix (§11.5 / TODO item 3) and §12.3.2 are re-verified there with the same tests and adapted if
682
+ the behaviour differs; the adaptation is recorded.
683
+ 5. Tests: the feature suites, then a full-suite comparison against the pre branch (no failing test title only on
684
+ the `-PRE` branch, apart from the known port-10002 cascade in tybot with each file green alone); builds for
685
+ chat21-ionic and design-studio. Build outputs (`www/`, `dist/`) are deleted after the check.
686
+ 6. Switch back to `feature/webhook-chatbot-hitl` and reinstall from its lockfile, so the checkout is left exactly as
687
+ before. The user is told when each repo is back on its normal branch.
688
+ Nothing is pushed; each repo ends with two local branches (default-based and `-PRE`).
689
+
690
+ ### 12.5 Testing
691
+ - tiledesk-server: new listener tests for "not waiting → release, no replacement, fallback cleared when the last
692
+ invited human leaves"; `clearWaiting` status event only on change; replacement released when the flow removed its
693
+ humans concurrently; `removeParticipantByRequestId` concurrency (two removals of different members at once →
694
+ both removed; a removal concurrent with a HITL write → the HITL fields survive), plus every existing
695
+ request/participant/HITL suite; full-suite comparison with `master`.
696
+ - tybot: HttpUtils (closed port and timeout: captured logs never contain the Authorization value; callback gets
697
+ the error); lock ownership (waiting on question A, an internal jump reaches question B → B is asked and the next
698
+ user text answers B, A is not consumed); Capture without block info stops; Capture / Reply V2 (buttons,
699
+ no-input) / lock / invite / remove suites; full-suite comparison with `main`.
700
+ - chat21-ionic and design-studio: no code change on the default-based branches; their `-PRE` ports are tested per
701
+ 12.4.
702
+ - End-to-end (§11.8) remains pending (needs a running chat21 stack or stage).
703
+
704
+ ## 13. End-to-end corrections (2026-09-30)
705
+
706
+ ### 13.1 Invite waits for the chat21 join
707
+
708
+ - The live run showed the flow's next message reaching the chat21 group ~30 ms before the invited human was added
709
+ (the join ran asynchronously after the invite answered), so the human missed it (2 of 4 invites).
710
+ - When the conversation's `channelOutbound.name` is `chat21`, `POST /:projectid/requests/:request_id/participants/invite`
711
+ now answers only after chat21 has settled (joined or failed) the join of every newly added member. Members that were
712
+ already participants trigger no join and are not waited for; other outbound channels are unchanged.
713
+ - The wait is bounded by `HITL_CHAT21_JOIN_TIMEOUT_MS` (default 5000). On timeout or join error the invite still answers
714
+ 200 as before and logs a warn line. `chat21Handler` registers each join in `chat21JoinTracker` (keyed by group and
715
+ member); the `group.join` / `group.join.error` events are unchanged.