@tiledesk/tiledesk-server 3.0.1 → 3.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.circleci/config.yml +1 -1
- package/CHANGELOG.md +34 -0
- package/Dockerfile +3 -2
- package/Dockerfile-en +3 -2
- package/Dockerfile-jobs +3 -2
- package/Dockerfile-profiler +3 -2
- package/README.md +15 -0
- package/app.js +12 -1
- package/bin/quota-reset +68 -0
- package/bin/quota-restore +86 -0
- package/channels/chat21/chat21Handler.js +50 -34
- package/channels/chat21/chat21JoinTracker.js +43 -0
- package/config/database.js +5 -5
- package/config/native_mcp/native_mcp_servers.json +1 -0
- package/controllers/chatbot.controller.js +80 -27
- package/controllers/intents.controller.js +42 -0
- package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-chat21-ionic.md +468 -0
- package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-design-studio.md +1349 -0
- package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-server.md +1694 -0
- package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-tybot.md +789 -0
- package/docs/superpowers/plans/2026-09-29-hitl-known-limits-server.md +83 -0
- package/docs/superpowers/plans/2026-09-29-hitl-known-limits-tybot.md +76 -0
- package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-chat21-ionic.md +97 -0
- package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-design-studio.md +72 -0
- package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-server.md +223 -0
- package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-tybot.md +121 -0
- package/docs/superpowers/plans/2026-09-29-hitl-pre-branch-ports.md +65 -0
- package/docs/superpowers/plans/2026-09-30-flow-start-points-design-studio.md +88 -0
- package/docs/superpowers/plans/2026-09-30-flow-start-points-server.md +103 -0
- package/docs/superpowers/plans/2026-09-30-flow-start-points-tybot.md +43 -0
- package/docs/superpowers/plans/2026-10-01-scheduled-start-point-design-studio.md +80 -0
- package/docs/superpowers/plans/2026-10-01-scheduled-start-point-runtime.md +39 -0
- package/docs/superpowers/plans/2026-10-01-scheduled-start-point-server.md +106 -0
- package/docs/superpowers/specs/2026-09-15-webhook-chatbot-hitl-design.md +715 -0
- package/docs/superpowers/specs/2026-09-30-flow-start-points-design.md +156 -0
- package/docs/superpowers/specs/2026-10-01-scheduled-start-point-design.md +303 -0
- package/docs/superpowers/todo/2026-09-14-webhook-chatbot-shared-code-fixes.md +191 -0
- package/event/botEvent.js +12 -1
- package/event/emailEvent.js +3 -0
- package/event/messageEvent.js +44 -0
- package/jobs.js +10 -0
- package/jobsManager.js +19 -0
- package/middleware/errorHandler.js +6 -2
- package/models/analyticResult.js +1 -1
- package/models/channelConstants.js +3 -2
- package/models/quotaUsage.js +71 -0
- package/models/request.js +6 -0
- package/models/requestSnapshot.js +12 -0
- package/models/routerLogger.js +1 -1
- package/models/webhook.js +44 -0
- package/package.json +7 -5
- package/pubmodules/emailNotification/requestNotification.js +14 -22
- package/pubmodules/events/test/eventRoute.js +7 -6
- package/pubmodules/rules/conciergeBot.js +9 -4
- package/routes/auth.js +1 -12
- package/routes/campaigns.js +0 -13
- package/routes/dataTable.js +0 -3
- package/routes/faq_kb.js +12 -1
- package/routes/messagesRoot.js +0 -1
- package/routes/quotes.js +38 -1
- package/routes/request.js +81 -5
- package/routes/users.js +0 -1
- package/routes/webhook.js +160 -8
- package/routes/webhooks.js +276 -6
- package/routes/widgetLoader.js +6 -3
- package/services/QuoteManager.js +77 -16
- package/services/aiService.js +1 -1
- package/services/chatbotService.js +55 -1
- package/services/departmentService.js +60 -43
- package/services/dolphinSchedulerClient.js +116 -0
- package/services/emailService.js +6 -1
- package/services/hitlParticipantListener.js +260 -0
- package/services/participantInviteService.js +302 -0
- package/services/participantRemoveService.js +79 -0
- package/services/quotaSnapshotService.js +1167 -0
- package/services/requestService.js +102 -144
- package/services/scheduledStart/dolphinBackend.js +246 -0
- package/services/scheduledStart/schedulerBackend.js +101 -0
- package/services/scheduledStart/schedulerServiceClient.js +41 -0
- package/services/scheduledStartPublishListener.js +186 -0
- package/services/scheduledStartRules.js +173 -0
- package/services/scheduledStartService.js +130 -0
- package/services/startPointCleanupListener.js +30 -0
- package/services/startPointService.js +158 -0
- package/services/testWsService.js +1 -4
- package/services/updateRequestSnapshotQueued.js +8 -9
- package/services/webhookConversationService.js +185 -0
- package/services/webhookSettingsValidator.js +61 -0
- package/template/email/checkpointReachedEmail.html +5 -0
- package/test/agentPromptsRoute.test.js +3 -2
- package/test/agentTokensQuota.js +237 -0
- package/test/botEventHitlFallback.js +19 -0
- package/test/chat21HandlerJoinWiring.js +59 -0
- package/test/dataTablesRoute.test.js +10 -2
- package/test/dateUtils.test.js +0 -3
- package/test/departmentService.js +8 -12
- package/test/{fileRoute.js → deprecated/_deprecated_fileRoute} +2 -2
- package/test/{kbsettingsRoute.js → deprecated/_deprecated_kbsettingsRoute} +3 -3
- package/test/faqRoute.js +6 -1
- package/test/faqkbRoute.js +14 -10
- package/test/faqkbV3Route.js +1 -1
- package/test/filepRoute.js +4 -50
- package/test/hitlParticipantListener.js +446 -0
- package/test/hitlQueueWiring.js +191 -0
- package/test/hitlWaitingReset.js +124 -0
- package/test/inviteParticipantsRoute.js +492 -0
- package/test/kbRoute.js +1 -1
- package/test/messageRoute.js +33 -21
- package/test/nativemcp.test.js +7 -1
- package/test/openaiRoute.js +2 -8
- package/test/publishFailLoud.test.js +414 -0
- package/test/quotaSnapshots.js +1044 -0
- package/test/removeParticipantsRoute.js +186 -0
- package/test/requestService.js +24 -16
- package/test/requestServiceRemoveParticipant.js +144 -0
- package/test/requestVisibility.test.js +186 -0
- package/test/scheduledStartFacade.js +184 -0
- package/test/scheduledStartHandover.js +95 -0
- package/test/scheduledStartPublish.js +320 -0
- package/test/scheduledStartRoute.js +163 -0
- package/test/scheduledStartRules.js +177 -0
- package/test/scheduledStartRun.js +299 -0
- package/test/scheduledStartSchedulerBackend.js +116 -0
- package/test/scheduledStartService.js +290 -0
- package/test/startPointsCleanup.js +69 -0
- package/test/startPointsRegression.js +169 -0
- package/test/startPointsRoute.js +158 -0
- package/test/webhookConversationRoute.js +347 -0
- package/test/webhookConversationService.js +261 -0
- package/test/webhookConversationTestUtils.js +55 -0
- package/utils/aiUtils.js +95 -1
- package/utils/intentFieldNames.js +192 -0
- package/utils/quotaCliUtil.js +43 -0
- package/utils/requestVisibility.js +355 -0
- package/websocket/webSocketServer.js +36 -31
- /package/test/{imageRoute.js → deprecated/_deprecated_imageRoute} +0 -0
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
# Flow start points (web + webhook) — Design
|
|
2
|
+
|
|
3
|
+
Date: 2026-09-30 · Status: approved in conversation, pending written review
|
|
4
|
+
Repos: tiledesk-server, tiledesk-chatbot (tybot), design-studio. chat21-ionic unchanged.
|
|
5
|
+
Branch: continue on `feature/webhook-chatbot-hitl` (this changes the unreleased webhook-conversation feature,
|
|
6
|
+
spec `2026-09-15-webhook-chatbot-hitl-design.md`); `-PRE` ports afterwards as for the rest of that feature.
|
|
7
|
+
|
|
8
|
+
## 1. Goal
|
|
9
|
+
|
|
10
|
+
A conversational chatbot flow can have several **start points**, each a dedicated box on the canvas with its
|
|
11
|
+
own arrow into the flow. First delivery: **web** (the widget interaction, as today) and **webhook**.
|
|
12
|
+
Later sub-projects, each with its own spec: **scheduled** and **integration triggers** (e.g. a new Gmail email
|
|
13
|
+
via the connector triggers).
|
|
14
|
+
|
|
15
|
+
## 2. Decisions (user, 2026-09-30)
|
|
16
|
+
|
|
17
|
+
1. Web start point = the widget interaction as it works today for conversational chatbots.
|
|
18
|
+
2. Every start point is a dedicated box in design-studio; under the hood they all go through the webhook interface.
|
|
19
|
+
3. At most one start point per type for now; leave room to allow several per type later.
|
|
20
|
+
4. First spec: model + web + webhook. Scheduled and integration follow as separate specs.
|
|
21
|
+
5. The webhook stays unique per chatbot: it is coupled to the chatbot definition.
|
|
22
|
+
6. The start type is informative: it does not change how the flow runs. Type-specific input formats are
|
|
23
|
+
handled by a **mapping** attached to the start type.
|
|
24
|
+
7. The department is defined while modelling the chatbot, not per start type.
|
|
25
|
+
8. Start points are picked from a dedicated **palette section**; future external triggers go there too.
|
|
26
|
+
9. Backward compatibility is mandatory: no regression for existing chatbots.
|
|
27
|
+
|
|
28
|
+
## 3. Current state (verified 2026-09-30)
|
|
29
|
+
|
|
30
|
+
- The start block is identified only by the name `start` (design-studio `RESERVED_INTENT_NAMES`,
|
|
31
|
+
`cds-intent.component.ts:276`); `faq` has a unique index on `{id_project, id_faq_kb, intent_display_name}`.
|
|
32
|
+
- The widget and channels start a bot with the system message `/start` (`pubmodules/trigger/rulesTrigger.js`);
|
|
33
|
+
`event/botEvent.js` forwards `/start` / `\start` system messages; tybot resolves `/start` as the intent
|
|
34
|
+
named `start` and lets `attributes.action` override the text (`engine/TiledeskChatbot.js:135-147`);
|
|
35
|
+
`#<intent_id>` resolves by id (`engine/MongodbBotsDataSource.js:93-99`).
|
|
36
|
+
- `models/webhook.js`: one webhook per chatbot (unique `{id_project, chatbot_id}`); `mode: 'conversation'`
|
|
37
|
+
selects the conversation path (`services/webhookConversationService.js`), no mode = automation path
|
|
38
|
+
(`services/webhookService.run` → tybot `/block`, sync or async). Conversation mode ignores `block_id` and
|
|
39
|
+
always sends `/start`.
|
|
40
|
+
- `attributes.start_type` is written only as `'webhook'` and read nowhere.
|
|
41
|
+
- Publish/fork keeps `intent_id` (`routes/faq_kb.js:811, 925`), so a `block_id` stays valid on the published copy.
|
|
42
|
+
- `Change department` (tybot `department`) re-routes the conversation (`requestService.route` → `routeInternal`)
|
|
43
|
+
and replaces the participants: in a webhook flow it removes the bot.
|
|
44
|
+
|
|
45
|
+
## 4. Data model and API (tiledesk-server)
|
|
46
|
+
|
|
47
|
+
### 4.1 Webhook record
|
|
48
|
+
One per chatbot, unchanged identity (`webhook_id`, URL). Unchanged fields: `webhook_id`, `id_project`, `name`,
|
|
49
|
+
`chatbot_id`, `enabled` (master switch), `async`, `copilot`, `mode`, and `block_id` (automation path only).
|
|
50
|
+
|
|
51
|
+
New field (only when `mode === 'conversation'`):
|
|
52
|
+
```
|
|
53
|
+
start_points: [{
|
|
54
|
+
type: 'webhook', // enum; 'web' is implicit (see 4.3); later 'scheduled', 'integration'
|
|
55
|
+
block_id: <intent_id>, // the start box
|
|
56
|
+
enabled: Boolean (default true),
|
|
57
|
+
mapping: { source_name: String } // requester name; home of future type-specific mappings
|
|
58
|
+
}]
|
|
59
|
+
```
|
|
60
|
+
- At most one entry per `type`, enforced by the service (400), not by an index — allowing several per type later
|
|
61
|
+
is a rule change only.
|
|
62
|
+
- Removed from the conversation webhook: `department_id`, `source_name` (unreleased branch fields; no migration).
|
|
63
|
+
|
|
64
|
+
### 4.2 Admin API (project-scoped, admin role)
|
|
65
|
+
- `GET /:projectid/webhooks/:chatbot_id` — includes `start_points`.
|
|
66
|
+
- `PUT /:projectid/webhooks/:chatbot_id/start_points/:type` body `{block_id, enabled?, mapping?, confirm?}` —
|
|
67
|
+
create/update one start point; creates the webhook (`mode: 'conversation'`) if missing.
|
|
68
|
+
Errors: 400 unknown type / bot not subtype `chatbot`; 404 `block_id` not a block of this chatbot;
|
|
69
|
+
409 existing webhook without mode (automation) unless `confirm: true`, which converts it.
|
|
70
|
+
- `DELETE /:projectid/webhooks/:chatbot_id/start_points/:type` — removes it (404 if absent).
|
|
71
|
+
- Deleting a block that is a start point's `block_id` removes that start point (faq delete hook).
|
|
72
|
+
|
|
73
|
+
### 4.3 Web start point
|
|
74
|
+
The existing `start` block, implicit: not stored in `start_points`. The widget/channel path is unchanged — still
|
|
75
|
+
the `/start` system message — so no existing bot is touched. Design-studio shows it as the "Web start" box.
|
|
76
|
+
|
|
77
|
+
### 4.4 Start flow — `POST /webhook/:webhook_id` (and `/dev`), conversation mode
|
|
78
|
+
1. Load the webhook; 404 unknown; 422 if `enabled` false or the `webhook` start point is missing/disabled
|
|
79
|
+
("Webhook start is not enabled"); rate limit (429) as today.
|
|
80
|
+
2. Upsert lead `webhook_<webhook_id>` named `mapping.source_name` (default: the webhook name).
|
|
81
|
+
3. Create the request as today (`support-group-…`, bot participant, channel `webhook`), department = the
|
|
82
|
+
department whose bot is this chatbot, else the project default. Attributes
|
|
83
|
+
`{start_type: 'webhook', webhook_id, payload}`; dev keeps `sourcePage … &td_draft=true`.
|
|
84
|
+
4. Send system message `/start` with `attributes.action = '#<block_id>'` (forwarded by botEvent as today;
|
|
85
|
+
tybot runs the box block, the text stays `/start`).
|
|
86
|
+
5. 202 `{success: true, request_id}`.
|
|
87
|
+
|
|
88
|
+
Payload handling unchanged: non-object bodies stored as `payload.body`; tybot merges payload keys + `payload`.
|
|
89
|
+
|
|
90
|
+
### 4.5 Automation webhooks
|
|
91
|
+
Untouched: no `start_points`, `block_id` + `/block` sync/async as today.
|
|
92
|
+
|
|
93
|
+
## 5. tybot
|
|
94
|
+
|
|
95
|
+
- `start_type` becomes a flow variable (from request attributes, like `payload`); empty for web/channel starts.
|
|
96
|
+
- A `/start` whose `attributes.action` names a missing block falls back to the `start` block (warn log), so a
|
|
97
|
+
conversation never stalls.
|
|
98
|
+
- No other change.
|
|
99
|
+
|
|
100
|
+
## 6. design-studio
|
|
101
|
+
|
|
102
|
+
### 6.1 Palette section "Start points"
|
|
103
|
+
- New palette category next to Most used / AI / Flow / Integrations / Special, own icon, shown only for chatbot
|
|
104
|
+
subtype `chatbot`.
|
|
105
|
+
- Items: "Web" (shown as placed, not draggable; click focuses the `start` box) and "Webhook". Future: Scheduled,
|
|
106
|
+
integration triggers grouped by app.
|
|
107
|
+
- Dragging "Webhook" onto the canvas creates the start box at the drop point: a protected block (default name
|
|
108
|
+
"Webhook start", `attributes.start_point: 'webhook'`, one connector action like `start`), then
|
|
109
|
+
`PUT …/start_points/webhook {block_id, enabled: true}`; on 409 show the existing confirmation
|
|
110
|
+
("This chatbot already has a webhook that runs a block without a conversation…") and retry with `confirm: true`.
|
|
111
|
+
- A type already in the flow is disabled ("Already in this flow"); clicking it focuses the box.
|
|
112
|
+
|
|
113
|
+
### 6.2 Start boxes
|
|
114
|
+
- Identified by `attributes.start_point` + the start point's `block_id`, never by name (no new reserved names).
|
|
115
|
+
- Rendered like `start` (pill + type icon); no actions besides the outgoing connector; no rename as a normal
|
|
116
|
+
block; no "Start test from here". The `start` box is labelled "Web start".
|
|
117
|
+
- Webhook box panel: enabled switch, production + development URLs (copy, regenerate), Source name (mapping),
|
|
118
|
+
Test webhook start, hint that flows can read `start_type` and the payload variables. The "Start from webhook"
|
|
119
|
+
section of the `start` panel is removed.
|
|
120
|
+
- Delete (webhook box only): confirmation → `DELETE …/start_points/webhook` → delete the block.
|
|
121
|
+
- Test webhook start: from the webhook box; highlights that box.
|
|
122
|
+
- Help text of Change department: note that in conversations started by a webhook it re-routes and removes
|
|
123
|
+
the bot.
|
|
124
|
+
|
|
125
|
+
## 7. chat21-ionic
|
|
126
|
+
No change: conversations still carry `request_channel: 'webhook'`; the requester shown is the mapping's source name.
|
|
127
|
+
|
|
128
|
+
## 8. Backward compatibility
|
|
129
|
+
|
|
130
|
+
| Existing case | Guarantee |
|
|
131
|
+
|---|---|
|
|
132
|
+
| Conversational bots started by widget/channels | Path unchanged (`/start`); no data written to existing bots |
|
|
133
|
+
| Automation webhooks (webhook/copilot subtype or no mode) | Record, `block_id`, `/block` sync/async unchanged; converting needs explicit confirmation |
|
|
134
|
+
| Block names | Start boxes use an attribute marker; `start`, `webhook`, `defaultFallback` keep their meaning |
|
|
135
|
+
| Publish / fork / import / templates | `intent_id` preserved → `block_id` valid on published copies; webhooks not exported (as today) |
|
|
136
|
+
| Deleting bot | Deletes its webhook as today |
|
|
137
|
+
| Unreleased webhook-conversation fields | `department_id`, `source_name`, top-level conversation `block_id` dropped without migration |
|
|
138
|
+
|
|
139
|
+
## 9. Testing
|
|
140
|
+
|
|
141
|
+
- tiledesk-server (mocha): start_points CRUD + validation (400/404/409, one per type), webhook call runs the box
|
|
142
|
+
(`attributes.action`) with `start_type`, 422 cases, start point removed on block delete, department rule.
|
|
143
|
+
Regression: automation webhook sync + async, widget `/start` launch, published bot, imported bot.
|
|
144
|
+
- tybot: `/start` + `action: '#box'` runs the box with `start_type` variable; missing block → `start`;
|
|
145
|
+
existing conversation tests unchanged.
|
|
146
|
+
- design-studio: palette category (one-per-type disabling, drop creates box + start point, 409 confirmation),
|
|
147
|
+
webhook box panel, delete with confirmation; `ng build` with no new warnings.
|
|
148
|
+
- End-to-end (local stack, see memory `hitl-local-e2e-stack`): drag Webhook start → curl runs the webhook box with
|
|
149
|
+
invite/buttons/Leave; widget still runs `start`; disable box → 422; delete box → 422; publish → works on the
|
|
150
|
+
published copy.
|
|
151
|
+
|
|
152
|
+
## 10. Out of scope / later
|
|
153
|
+
- Scheduled start points (needs a cluster-safe scheduler; `node-schedule` is in-process only).
|
|
154
|
+
- Integration start points (Tiledesk-side subscription to connector `POST /triggers`, per-trigger mapping).
|
|
155
|
+
- Several start points of the same type.
|
|
156
|
+
- Moving the widget/channel start onto the start service.
|
|
@@ -0,0 +1,303 @@
|
|
|
1
|
+
# Scheduled start point — Design
|
|
2
|
+
|
|
3
|
+
Date: 2026-10-01 · Status: approved in conversation, pending written review
|
|
4
|
+
Repos: tiledesk-server, design-studio, tiledesk-agent-runtime. tiledesk-chatbot (tybot) and chat21-ionic unchanged.
|
|
5
|
+
Branch: `feature/scheduled-start-point`, created from `feature/webhook-chatbot-hitl` in each repo (start points
|
|
6
|
+
are not on master yet). Builds on `2026-09-30-flow-start-points-design.md` (web + webhook start points).
|
|
7
|
+
|
|
8
|
+
## 1. Goal
|
|
9
|
+
|
|
10
|
+
A conversational chatbot (subtype `chatbot`) can start its flow on a schedule. A **Scheduled** start box, picked
|
|
11
|
+
from the Start points palette like Web and Webhook, opens a new conversation at the chosen times and runs the
|
|
12
|
+
flow from that box. The schedule runs on the scheduler already used for KB auto re-index (Apache
|
|
13
|
+
DolphinScheduler), and it is live only for the **published** version of the chatbot.
|
|
14
|
+
|
|
15
|
+
## 2. Decisions (user, 2026-10-01)
|
|
16
|
+
|
|
17
|
+
1. Scheduler: DolphinScheduler, the same mechanism as KB auto re-index (not the scheduled-jobs service, which is
|
|
18
|
+
only on master-COLLAUDO).
|
|
19
|
+
2. Schedule expressiveness: presets plus time plus timezone — every N minutes/hours, daily at a time, weekly on
|
|
20
|
+
chosen weekdays at a time, monthly on a day at a time, in an IANA timezone. No raw cron.
|
|
21
|
+
3. Overlap: every run starts a new conversation, even while the previous scheduled one is open.
|
|
22
|
+
4. Run data: a Source name and a static payload, stored as JSON but edited through a visual builder (rows of
|
|
23
|
+
name / type / value), never a raw JSON editor.
|
|
24
|
+
5. Integration approach A: tiledesk-server talks to DolphinScheduler directly and synchronously; the workflow's
|
|
25
|
+
HTTP task calls a token-protected server endpoint.
|
|
26
|
+
6. Lifecycle: a schedule becomes triggerable only when the chatbot is published. Every change to the Scheduled
|
|
27
|
+
start point — timing, payload, Source name, the enabled switch, deleting the box — takes effect in production
|
|
28
|
+
at the next publish. Deleting the chatbot removes its schedule immediately.
|
|
29
|
+
7. At most one Scheduled start point per chatbot (same rule as the other types).
|
|
30
|
+
|
|
31
|
+
## 3. Current state (verified 2026-10-01)
|
|
32
|
+
|
|
33
|
+
- KB auto re-index: tiledesk-training-jobworker (`tiledesk/ReindexScheduler.js`, `utils/shedulerUtils.js`) creates a
|
|
34
|
+
DolphinScheduler process definition `auto-reindex-<content_id>` with one HTTP task
|
|
35
|
+
`POST {API_ENDPOINT}/webhook/kb/reindex` (body `content_id`, header `x-auth-token` = `KB_WEBHOOK_TOKEN`, both as
|
|
36
|
+
workflow global params), releases it, creates a schedule (Quartz cron from `daily|weekly|monthly` anchored on the
|
|
37
|
+
start time, `startTime` now+20s, `endTime` 2099-12-31) and sets it online. tiledesk-server only deletes them
|
|
38
|
+
(`services/aiReindexService.js`: schedule offline → definition offline → delete), from `/webhook/kb/reindex`.
|
|
39
|
+
Env: `SCHEDULER_BASEURL`, `SCHEDULER_TOKEN`, `SCHEDULER_PROJECT` (all set on stage).
|
|
40
|
+
- Stage runs `apache/dolphinscheduler-standalone-server:3.2.2` (tiledesk-argo `helm/values.yaml`). In 3.2 a schedule
|
|
41
|
+
carries `timezoneId`, and an online schedule must be taken offline to be updated.
|
|
42
|
+
- Publish (`PUT /:projectid/faq_kb/:id/publish`) forks a frozen copy (`root_id` = the chatbot, `trashed: true`),
|
|
43
|
+
points the chatbot's `url` at it and emits `faqbot.publish`. The fork keeps every `intent_id`. Production
|
|
44
|
+
conversations run the published copy; requests whose `sourcePage` contains `td_draft=true` run the draft.
|
|
45
|
+
- Start points: one webhook record per chatbot (`models/webhook.js`) with `start_points[]`; `startPointService`
|
|
46
|
+
(upsert/remove/find, one per type), routes `PUT|DELETE /:projectid/webhooks/:chatbot_id/start_points/:type`,
|
|
47
|
+
`startPointCleanupListener` on `faq.delete`, `webhookConversationService.start` (lead, department, request with
|
|
48
|
+
`start_type`, `/start` with `attributes.action = '#<block_id>'`).
|
|
49
|
+
- Invite/Remove human accept a request whose `channel.name === 'webhook'`
|
|
50
|
+
(`participantInviteService.js:45`, `participantRemoveService.js:27`).
|
|
51
|
+
- tybot already exposes `start_type` and the payload keys as flow variables and falls back to `start` when the
|
|
52
|
+
action's box is missing.
|
|
53
|
+
|
|
54
|
+
## 4. Data model (tiledesk-server)
|
|
55
|
+
|
|
56
|
+
`start_points[]` gains the type `'scheduled'`:
|
|
57
|
+
|
|
58
|
+
```
|
|
59
|
+
{
|
|
60
|
+
type: 'scheduled',
|
|
61
|
+
block_id: <intent_id of the Scheduled box>,
|
|
62
|
+
enabled: Boolean (default true),
|
|
63
|
+
mapping: {
|
|
64
|
+
source_name: String, // requester; empty → chatbot name (as webhook)
|
|
65
|
+
payload: { <key>: String|Number|Boolean } // static run data
|
|
66
|
+
},
|
|
67
|
+
schedule: {
|
|
68
|
+
frequency: 'interval' | 'daily' | 'weekly' | 'monthly',
|
|
69
|
+
every: Number, unit: 'minutes' | 'hours', // interval
|
|
70
|
+
time: 'HH:MM', // daily, weekly, monthly
|
|
71
|
+
weekdays: ['mon'..'sun'], // weekly, at least one
|
|
72
|
+
day_of_month: 1..28 | 'last', // monthly
|
|
73
|
+
timezone: String // IANA zone
|
|
74
|
+
},
|
|
75
|
+
live: { // server-owned: what is active in DolphinScheduler
|
|
76
|
+
enabled, block_id, mapping, schedule, // snapshot taken at publish
|
|
77
|
+
published_bot_id, published_at,
|
|
78
|
+
workflow_code, schedule_id,
|
|
79
|
+
error // last sync error message, absent when fine
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Everything except `live` is the **draft**, edited through the API. `live` is written only by the publish sync and
|
|
85
|
+
the sync/retry endpoint, never accepted from a client.
|
|
86
|
+
|
|
87
|
+
Validation (400, message naming the field):
|
|
88
|
+
- `interval`: `unit: 'minutes'` with `every` ∈ {5, 10, 15, 20, 30}; `unit: 'hours'` with `every` ∈ 1..12.
|
|
89
|
+
- `daily|weekly|monthly`: `time` matches `^([01]\d|2[0-3]):[0-5]\d$`; weekly needs ≥ 1 weekday; monthly
|
|
90
|
+
`day_of_month` ∈ 1..28 or `'last'`.
|
|
91
|
+
- `timezone`: a valid IANA zone (`Intl.DateTimeFormat(undefined, { timeZone })` does not throw).
|
|
92
|
+
- `payload`: a plain object, flat; keys match `^[A-Za-z_][A-Za-z0-9_]*$`; values string, finite number or boolean;
|
|
93
|
+
at most 50 keys and 8 KB serialized.
|
|
94
|
+
|
|
95
|
+
## 5. API (tiledesk-server)
|
|
96
|
+
|
|
97
|
+
Existing routes, new type value:
|
|
98
|
+
- `PUT /:projectid/webhooks/:chatbot_id/start_points/scheduled` body `{ block_id, enabled?, mapping?, schedule }`
|
|
99
|
+
(`schedule` required on create) — saves the draft only, no scheduler call. Same rules as webhook start points:
|
|
100
|
+
one per type, `block_id` of this chatbot (404), subtype `chatbot` (400), automation webhook → 409 unless
|
|
101
|
+
`confirm`. Response: the webhook, plus `schedule_summary` (draft) and `live_summary` (live, when present),
|
|
102
|
+
e.g. "Every weekday at 08:30 (Europe/Rome)".
|
|
103
|
+
- `DELETE …/start_points/scheduled` — removes the draft entry; `live` and the DolphinScheduler objects stay until
|
|
104
|
+
the next publish, which removes them (§7).
|
|
105
|
+
- `GET /:projectid/webhooks/:chatbot_id` — adds `scheduled_available: Boolean` and, for a live scheduled start point,
|
|
106
|
+
`next_runs` (next 3 fire times from DolphinScheduler's schedule preview; omitted on error).
|
|
107
|
+
|
|
108
|
+
New routes:
|
|
109
|
+
- `POST /:projectid/webhooks/:chatbot_id/start_points/scheduled/test` (admin) — runs the **draft** once now: starts a
|
|
110
|
+
conversation as in §8 but with the draft mapping and `sourcePage … dev=true&td_draft=true`, so the draft bot
|
|
111
|
+
runs. Answers 202 `{ request_id }`. Works whether or not the chatbot is published.
|
|
112
|
+
- `POST /:projectid/webhooks/:chatbot_id/start_points/scheduled/sync` (admin) — re-runs the publish sync of §7
|
|
113
|
+
against the latest published copy (the retry behind "Couldn't update the live schedule").
|
|
114
|
+
- `POST /webhook/scheduled-start` (public, called by DolphinScheduler) — §8.
|
|
115
|
+
|
|
116
|
+
Errors: `{ success: false, error }`. Scheduler not configured (`SCHEDULER_BASEURL`, `SCHEDULER_TOKEN`,
|
|
117
|
+
`SCHEDULER_PROJECT` or `SCHEDULED_START_TOKEN` missing): the `scheduled` routes answer 503
|
|
118
|
+
"Scheduled starts are not available on this installation" and `scheduled_available` is false.
|
|
119
|
+
|
|
120
|
+
## 6. Scheduler integration (tiledesk-server)
|
|
121
|
+
|
|
122
|
+
`services/dolphinSchedulerClient.js` — thin REST client (axios, header `token`), same endpoints as KB re-index:
|
|
123
|
+
create/release/delete process definition, create/update/online/offline schedule, find definition by name,
|
|
124
|
+
schedule preview. `AiReindexService` is not changed.
|
|
125
|
+
|
|
126
|
+
`services/scheduledStartService.js` — owns the DolphinScheduler objects of scheduled starts:
|
|
127
|
+
- Workflow per chatbot, name `auto-start-<chatbot_id>`: one HTTP task `POST {API_URL}/webhook/scheduled-start`,
|
|
128
|
+
body params `id_project`, `chatbot_id`, header `x-auth-token: ${token}`, values as workflow global params
|
|
129
|
+
(`SCHEDULED_START_TOKEN`), `connectTimeout`/`socketTimeout` 60000, no retries.
|
|
130
|
+
- Schedule: `crontab = toQuartzCron(schedule)`, `timezoneId = schedule.timezone`, `startTime` = now,
|
|
131
|
+
`endTime` = 2099-12-31 23:59:59, `failureStrategy: CONTINUE`, `warningType: NONE`.
|
|
132
|
+
- `toQuartzCron(schedule)` (pure):
|
|
133
|
+
- interval minutes N → `0 0/N * * * ? *`; hours N → `0 0 0/N * * ? *`
|
|
134
|
+
- daily HH:MM → `0 MM HH * * ? *`
|
|
135
|
+
- weekly → `0 MM HH ? * MON,WED,FRI *`
|
|
136
|
+
- monthly day D → `0 MM HH D * ? *`; `last` → `0 MM HH L * ? *`
|
|
137
|
+
- Operations: `activate(live)` (create definition → release online → create schedule → schedule online; on a
|
|
138
|
+
failing step, undo the steps done), `update(live)` (schedule offline → update → online), `deactivate(live)`
|
|
139
|
+
(schedule offline), `destroy(live)` (schedule offline → definition offline → delete). A stored
|
|
140
|
+
`workflow_code`/`schedule_id` that DolphinScheduler no longer knows is recreated instead of failing.
|
|
141
|
+
|
|
142
|
+
Running is cluster-safe: DolphinScheduler fires once per due time; whichever server pod receives the HTTP call
|
|
143
|
+
handles it. No in-process timer.
|
|
144
|
+
|
|
145
|
+
## 7. Publish lifecycle
|
|
146
|
+
|
|
147
|
+
A listener on `faqbot.publish` (registered in `app.js` next to the start point cleanup listener) syncs the live
|
|
148
|
+
schedule with the draft, using the published copy:
|
|
149
|
+
|
|
150
|
+
| Draft scheduled start point | Published copy has `block_id` | Result |
|
|
151
|
+
|---|---|---|
|
|
152
|
+
| present, enabled | yes | `activate` (no live yet) or `update` (live exists); `live` = snapshot of the draft + published ids |
|
|
153
|
+
| present, disabled | — | `deactivate`; `live.enabled = false` |
|
|
154
|
+
| present | no | `destroy`; `live` cleared |
|
|
155
|
+
| absent (box deleted) | — | `destroy`; `live` cleared |
|
|
156
|
+
|
|
157
|
+
- Publishing never fails because of the scheduler: a sync error is stored in `live.error`, logged, and shown by the
|
|
158
|
+
panel with a Retry (`…/scheduled/sync`).
|
|
159
|
+
- Deleting the chatbot (the existing faq_kb delete path) calls `destroy` immediately and deletes the webhook as
|
|
160
|
+
today.
|
|
161
|
+
- Restoring a release (`publish` with `restore_from`) goes through the same listener.
|
|
162
|
+
|
|
163
|
+
## 8. A scheduled run
|
|
164
|
+
|
|
165
|
+
`POST /webhook/scheduled-start`, body `{ id_project, chatbot_id }`, header `x-auth-token`:
|
|
166
|
+
1. Missing/wrong token → 401. Token not configured → 503.
|
|
167
|
+
2. Chatbot, webhook record, scheduled start point or `live` missing, or `live.enabled` false → 410 and the
|
|
168
|
+
workflow is taken offline (a dangling workflow removes itself, as `/webhook/kb/reindex` does for deleted
|
|
169
|
+
content). Rate limit as webhook calls (429).
|
|
170
|
+
3. Start the conversation through `webhookConversationService.start` with the scheduled start point:
|
|
171
|
+
- lead `scheduled_<chatbot_id>`, fullname `live.mapping.source_name` || chatbot name;
|
|
172
|
+
- department: same rule as webhook starts (`resolveChatbotDepartment`);
|
|
173
|
+
- channel `webhook` (chat21-ionic shows it as a webhook conversation; Invite/Remove human accept it);
|
|
174
|
+
- request attributes `{ start_type: 'scheduled', payload: live.mapping.payload, scheduled_at: <ISO time> }`;
|
|
175
|
+
- system `/start` with `attributes.action = '#' + live.block_id`;
|
|
176
|
+
- not a draft request, so the published copy runs.
|
|
177
|
+
4. 202 `{ success: true, request_id }`.
|
|
178
|
+
|
|
179
|
+
Every run is a new conversation (decision 3). A run missed while DolphinScheduler is down is not recovered.
|
|
180
|
+
|
|
181
|
+
tybot needs no change: `{{start_type}}` is `scheduled`, each payload key is a variable, `{{payload}}` the object,
|
|
182
|
+
and the missing-box fallback to `start` still applies.
|
|
183
|
+
|
|
184
|
+
The 422 messages of Invite/Remove human become "… requires a conversation started by a webhook or a schedule".
|
|
185
|
+
|
|
186
|
+
## 9. design-studio
|
|
187
|
+
|
|
188
|
+
- Palette Start points: item **Scheduled** (clock icon) next to Web and Webhook; one per type ("Already in this
|
|
189
|
+
flow"); subtype `chatbot` only; hidden when `scheduled_available` is false.
|
|
190
|
+
- Dropping it creates the protected block (`attributes.start_point: 'scheduled'`, name "Scheduled start"), then
|
|
191
|
+
`PUT …/start_points/scheduled` with defaults: daily 09:00, browser timezone, enabled, Source name = chatbot
|
|
192
|
+
name, empty payload. Same failure handling as the webhook box (the block is removed on error).
|
|
193
|
+
- The box behaves like the other start boxes (toolbar "⋮" only, one outgoing connector, delete from its panel,
|
|
194
|
+
no copy/paste) and shows the draft summary and a badge: **Live**, **Changes not live**, **Not live**.
|
|
195
|
+
- Panel:
|
|
196
|
+
1. Status: "Live — every weekday at 08:30 (Europe/Rome) · next run Tue 08:30" / "Not live — publish to activate"
|
|
197
|
+
/ "Changes not live — publish to apply" / "Couldn't update the live schedule — Retry".
|
|
198
|
+
2. Enabled switch, hint "Takes effect when you publish".
|
|
199
|
+
3. Schedule: Repeat (Every N minutes/hours, Daily, Weekly, Monthly) with the matching fields (interval
|
|
200
|
+
selector, time picker, weekday chips, day of month 1–28 or Last day), Timezone (searchable, browser default),
|
|
201
|
+
live summary sentence.
|
|
202
|
+
4. Source name (placeholder: chatbot name).
|
|
203
|
+
5. Data for each run — the visual payload builder: rows Name · Type (Text / Number / True-false) · Value ·
|
|
204
|
+
remove, "+ Add field"; names validated as typed (pattern, no duplicates); converted to and from the stored
|
|
205
|
+
JSON; hint "Each field is a variable in the flow, e.g. {{check}} — {{start_type}} is scheduled".
|
|
206
|
+
6. Test scheduled start (draft, now; opens the conversation and the logs, like Test webhook start).
|
|
207
|
+
7. Delete (confirmation; help text: a live schedule stops at the next publish).
|
|
208
|
+
- Saving: each change saves the draft with `PUT` (debounced), error toast on failure; invalid input is flagged and
|
|
209
|
+
not sent.
|
|
210
|
+
- Help text: stopping a live schedule takes a publish (switch off, then publish).
|
|
211
|
+
- master-pre: read-only mode disables the palette item and the panel (pre-only adaptation, as for the webhook box).
|
|
212
|
+
|
|
213
|
+
## 10. Vibe coder (tiledesk-agent-runtime)
|
|
214
|
+
|
|
215
|
+
- Prompt §1c: the **Scheduled start box** (`attributes.start_point: "scheduled"`) — an entry point wired like
|
|
216
|
+
`start`, never deleted or renamed; the agent cannot create it nor set its timing or data (the user does, from the
|
|
217
|
+
palette and the panel); a run reads `{{start_type}}` = `scheduled` and the payload fields; it is live only after
|
|
218
|
+
publish; nobody is on the other side until the flow invites someone; Invite/Remove human work.
|
|
219
|
+
- Back-office pattern: a scheduled example (a periodic check that invites a person only when something is wrong).
|
|
220
|
+
- Catalogue overlay of `invite_human`/`remove_human`: "started by a webhook or a schedule".
|
|
221
|
+
|
|
222
|
+
## 11. Backward compatibility
|
|
223
|
+
|
|
224
|
+
| Existing case | Guarantee |
|
|
225
|
+
|---|---|
|
|
226
|
+
| Web and webhook start points | Unchanged; their code paths do not read `schedule`/`live` |
|
|
227
|
+
| KB auto re-index | Untouched: `AiReindexService`, `/webhook/kb/reindex`, jobworker unchanged |
|
|
228
|
+
| Installations without DolphinScheduler / token | Feature off: palette item hidden, routes 503, publish unaffected |
|
|
229
|
+
| Publish | Never fails because of the scheduler (§7) |
|
|
230
|
+
| Chatbots without scheduled start | No listener work beyond a lookup; no scheduler calls |
|
|
231
|
+
|
|
232
|
+
## 12. Testing
|
|
233
|
+
|
|
234
|
+
- Server (mocha): `toQuartzCron` for every frequency and edge (5-minute floor, `last`, weekday lists); validation
|
|
235
|
+
400s; DolphinScheduler client against a stub HTTP server, including rollback on a failing step and recreate on
|
|
236
|
+
unknown ids; publish listener (activate, update, deactivate, destroy when box missing or deleted, `live.error` on
|
|
237
|
+
failure, restore_from); `/webhook/scheduled-start` (401, 503, 410 + offline, 429, conversation with published
|
|
238
|
+
copy, payload, `start_type`, `scheduled_at`); test-run on the draft; sync endpoint; chatbot delete destroys;
|
|
239
|
+
regression on webhook start points and `/webhook/kb/reindex`.
|
|
240
|
+
- design-studio (specs): schedule summary and validation; payload builder rows ↔ JSON; palette rules and
|
|
241
|
+
`scheduled_available`; panel states.
|
|
242
|
+
- runtime: prompt/catalogue assertions in `tests/test_examples.py`.
|
|
243
|
+
- End-to-end: local stack plus `apache/dolphinscheduler-standalone-server:3.2.2` in docker — add the box, test-run
|
|
244
|
+
the draft, publish, see a run fire, change the time without publishing (old schedule keeps firing), switch off +
|
|
245
|
+
publish (runs stop), delete the chatbot (workflow gone). Then the same on stage with test agents, checking the
|
|
246
|
+
workflow in the DolphinScheduler UI.
|
|
247
|
+
|
|
248
|
+
## 13. Delivery
|
|
249
|
+
|
|
250
|
+
Branch `feature/scheduled-start-point` from `feature/webhook-chatbot-hitl` in tiledesk-server, design-studio and
|
|
251
|
+
tiledesk-agent-runtime; then the usual path (tests, push, merge into the pre branches, tags, argo stage). Stage
|
|
252
|
+
needs `SCHEDULED_START_TOKEN` on the server and worker deployments (argo values or secret), checked before deploy.
|
|
253
|
+
|
|
254
|
+
## 14. Out of scope / later
|
|
255
|
+
|
|
256
|
+
- Several scheduled start points per chatbot.
|
|
257
|
+
- One-off runs at a date ("once"), as another frequency.
|
|
258
|
+
- Catching up runs missed while DolphinScheduler is down.
|
|
259
|
+
- Nested payload objects in the builder.
|
|
260
|
+
- Moving KB auto re-index onto `dolphinSchedulerClient`.
|
|
261
|
+
|
|
262
|
+
## 15. Running on tiledesk-scheduler-service (added 2026-10-13)
|
|
263
|
+
|
|
264
|
+
Scheduled starts can run on tiledesk-scheduler-service ≥ 1.1.0 instead of DolphinScheduler. `SCHEDULED_START_BACKEND` selects the backend per call:
|
|
265
|
+
|
|
266
|
+
| `SCHEDULED_START_BACKEND` | Required env on server + worker | Code |
|
|
267
|
+
|---|---|---|
|
|
268
|
+
| `dolphin` (default, also used for unknown values) | `SCHEDULER_BASEURL`, `SCHEDULER_TOKEN`, `SCHEDULER_PROJECT`, `SCHEDULED_START_TOKEN` | `services/scheduledStart/dolphinBackend.js` |
|
|
269
|
+
| `scheduler` | `SCHEDULER_SERVICE_URL`, `SCHEDULER_API_TOKEN`, `SCHEDULED_START_TOKEN` | `services/scheduledStart/schedulerBackend.js` |
|
|
270
|
+
|
|
271
|
+
`services/scheduledStartService.js` is a façade with the same API as before (`activate`, `update`, `deactivate`, `destroy`, `nextRuns`, `offlineFor`, `destroyFor`) plus `hasLive` and `liveIds`.
|
|
272
|
+
|
|
273
|
+
**How the scheduler backend works**
|
|
274
|
+
- One cron slot per chatbot: `projectId = id_project`, `flowId = auto-start-<chatbotId>`, `target = scheduledStart`, payload `{ id_project, chatbot_id }`.
|
|
275
|
+
- The cron is a standard 5-field expression from `toCron5`, with the schedule's IANA `timezone`.
|
|
276
|
+
- The scheduler adds `x-auth-token: $SCHEDULED_START_TOKEN` when the job runs and calls `$TILEDESK_API_URL/webhook/scheduled-start`. The token is never sent to the scheduler, and rotating it needs no re-sync. **Rotate it on the server and on the scheduler worker at the same time**: a mismatch gets a 401 on every run, the target never retries, and the bots silently stop.
|
|
277
|
+
- The `scheduledStart` target never retries, so there's at most one conversation per tick. A 410 from the run endpoint pauses the slot.
|
|
278
|
+
- `scheduled_live` stores `{ backend: 'scheduler', job_id, id_project, chatbot_id }` instead of `{ workflow_code, schedule_id }`.
|
|
279
|
+
- `next_runs` comes from `POST /schedules/preview`, formatted as `YYYY-MM-DD HH:mm:ss` wall-clock in the schedule's timezone, as with DolphinScheduler.
|
|
280
|
+
|
|
281
|
+
**Handover between backends**
|
|
282
|
+
- When `scheduled_live` holds ids of the backend that is *not* selected, the next `update` / `deactivate` / `destroy` / `offlineFor` / `destroyFor` works on the selected backend first, then removes the other backend's schedule.
|
|
283
|
+
- If that removal fails, the other backend's ids **stay** in `scheduled_live` next to the new ones, and every later call retries. Dropping them would leave that schedule firing with nothing left to remove it, and the bot would start twice per tick.
|
|
284
|
+
- The ids are dropped only when the removal succeeds, or when the other backend's env is no longer set.
|
|
285
|
+
- With no ids stored (e.g. a trashed chatbot whose webhook is already gone), `offlineFor` / `destroyFor` look the schedule up on both backends.
|
|
286
|
+
- All of this works in both directions, so a rollback to `dolphin` also removes the scheduler slot.
|
|
287
|
+
- The new schedule is created before the old one is removed: a tick that falls in that window of a few hundred milliseconds fires on both. Migrate away from tick times.
|
|
288
|
+
|
|
289
|
+
**Cutover (stage, then prod)**
|
|
290
|
+
1. Deploy tiledesk-scheduler-service ≥ 1.1.0 (API + worker).
|
|
291
|
+
- Scheduler env: `SCHEDULER_API_TOKEN`, `TILEDESK_API_URL` (server base URL reachable from the worker), `SCHEDULED_START_TOKEN` (same value as the server).
|
|
292
|
+
- Check `GET /health/ready` and `GET /targets` (lists `scheduledStart`).
|
|
293
|
+
2. Server + server-worker env (argo values / `config-scheduler`):
|
|
294
|
+
- `SCHEDULED_START_BACKEND=scheduler`
|
|
295
|
+
- `SCHEDULER_SERVICE_URL=http://<release>-scheduler-api`
|
|
296
|
+
- `SCHEDULER_API_TOKEN` (same as the scheduler).
|
|
297
|
+
- Keep the DolphinScheduler variables until step 4, so the handover can remove the old workflows.
|
|
298
|
+
3. Migrate every live bot with `POST /:projectid/webhooks/:chatbot_id/start_points/scheduled/migrate`, for each webhook with a `scheduled_live`.
|
|
299
|
+
- Migrate recreates the schedule from the **stored live snapshot**, so unpublished draft edits are not published. Do **not** use `/sync` here: it reads the draft, so it would publish draft edits, and it destroys the live schedule if the draft was deleted.
|
|
300
|
+
- It creates the slot (or pauses it when `enabled` is false) and removes the `auto-start-<id>` DolphinScheduler workflow.
|
|
301
|
+
- Check: the scheduler's `GET /jobs?projectId=` shows the slot; `scheduled_live` has no `workflow_code` left (if it still does, the DolphinScheduler cleanup failed: run migrate again); DolphinScheduler has no `auto-start-*` left; the first run creates a conversation with `start_type=scheduled`.
|
|
302
|
+
- Bots that are not migrated keep firing from DolphinScheduler until their next publish, which moves them.
|
|
303
|
+
4. Once no `auto-start-*` workflow is left, the DolphinScheduler variables can be removed from the server. DolphinScheduler itself stays while KB re-index still uses it.
|