@tiledesk/tiledesk-server 3.0.1 → 3.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (140) hide show
  1. package/.circleci/config.yml +1 -1
  2. package/CHANGELOG.md +47 -0
  3. package/Dockerfile +3 -2
  4. package/Dockerfile-en +3 -2
  5. package/Dockerfile-jobs +3 -2
  6. package/Dockerfile-profiler +3 -2
  7. package/README.md +15 -0
  8. package/app.js +12 -1
  9. package/bin/quota-reset +68 -0
  10. package/bin/quota-restore +86 -0
  11. package/channels/chat21/chat21Handler.js +50 -34
  12. package/channels/chat21/chat21JoinTracker.js +43 -0
  13. package/config/database.js +5 -5
  14. package/config/native_mcp/native_mcp_servers.json +1 -0
  15. package/controllers/chatbot.controller.js +80 -27
  16. package/controllers/intents.controller.js +42 -0
  17. package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-chat21-ionic.md +468 -0
  18. package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-design-studio.md +1349 -0
  19. package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-server.md +1694 -0
  20. package/docs/superpowers/plans/2026-09-15-webhook-chatbot-hitl-tybot.md +789 -0
  21. package/docs/superpowers/plans/2026-09-29-hitl-known-limits-server.md +83 -0
  22. package/docs/superpowers/plans/2026-09-29-hitl-known-limits-tybot.md +76 -0
  23. package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-chat21-ionic.md +97 -0
  24. package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-design-studio.md +72 -0
  25. package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-server.md +223 -0
  26. package/docs/superpowers/plans/2026-09-29-hitl-participant-removal-tybot.md +121 -0
  27. package/docs/superpowers/plans/2026-09-29-hitl-pre-branch-ports.md +65 -0
  28. package/docs/superpowers/plans/2026-09-30-flow-start-points-design-studio.md +88 -0
  29. package/docs/superpowers/plans/2026-09-30-flow-start-points-server.md +103 -0
  30. package/docs/superpowers/plans/2026-09-30-flow-start-points-tybot.md +43 -0
  31. package/docs/superpowers/plans/2026-10-01-scheduled-start-point-design-studio.md +80 -0
  32. package/docs/superpowers/plans/2026-10-01-scheduled-start-point-runtime.md +39 -0
  33. package/docs/superpowers/plans/2026-10-01-scheduled-start-point-server.md +106 -0
  34. package/docs/superpowers/specs/2026-09-15-webhook-chatbot-hitl-design.md +715 -0
  35. package/docs/superpowers/specs/2026-09-30-flow-start-points-design.md +156 -0
  36. package/docs/superpowers/specs/2026-10-01-scheduled-start-point-design.md +303 -0
  37. package/docs/superpowers/todo/2026-09-14-webhook-chatbot-shared-code-fixes.md +191 -0
  38. package/event/botEvent.js +12 -1
  39. package/event/emailEvent.js +3 -0
  40. package/event/messageEvent.js +44 -0
  41. package/jobs.js +10 -0
  42. package/jobsManager.js +19 -0
  43. package/middleware/errorHandler.js +6 -2
  44. package/models/analyticResult.js +1 -1
  45. package/models/channelConstants.js +3 -2
  46. package/models/dataTable.js +16 -0
  47. package/models/location.js +6 -12
  48. package/models/quotaUsage.js +71 -0
  49. package/models/request.js +71 -61
  50. package/models/requestSnapshot.js +12 -0
  51. package/models/routerLogger.js +1 -1
  52. package/models/webhook.js +44 -0
  53. package/package.json +9 -7
  54. package/pubmodules/emailNotification/requestNotification.js +14 -22
  55. package/pubmodules/events/test/eventRoute.js +7 -6
  56. package/pubmodules/rules/conciergeBot.js +9 -4
  57. package/routes/auth.js +1 -12
  58. package/routes/campaigns.js +0 -13
  59. package/routes/dataTable.js +23 -3
  60. package/routes/faq_kb.js +12 -1
  61. package/routes/messagesRoot.js +0 -1
  62. package/routes/quotes.js +38 -1
  63. package/routes/request.js +81 -5
  64. package/routes/users.js +0 -1
  65. package/routes/webhook.js +160 -8
  66. package/routes/webhooks.js +276 -6
  67. package/routes/widgetLoader.js +6 -3
  68. package/services/QuoteManager.js +77 -16
  69. package/services/aiService.js +1 -1
  70. package/services/chatbotService.js +55 -1
  71. package/services/dataTableService.js +436 -42
  72. package/services/departmentService.js +60 -43
  73. package/services/dolphinSchedulerClient.js +116 -0
  74. package/services/emailService.js +6 -1
  75. package/services/hitlParticipantListener.js +260 -0
  76. package/services/participantInviteService.js +302 -0
  77. package/services/participantRemoveService.js +79 -0
  78. package/services/quotaSnapshotService.js +1167 -0
  79. package/services/requestService.js +102 -144
  80. package/services/scheduledStart/dolphinBackend.js +246 -0
  81. package/services/scheduledStart/schedulerBackend.js +101 -0
  82. package/services/scheduledStart/schedulerServiceClient.js +41 -0
  83. package/services/scheduledStartPublishListener.js +186 -0
  84. package/services/scheduledStartRules.js +173 -0
  85. package/services/scheduledStartService.js +130 -0
  86. package/services/startPointCleanupListener.js +30 -0
  87. package/services/startPointService.js +158 -0
  88. package/services/testWsService.js +1 -4
  89. package/services/updateRequestSnapshotQueued.js +8 -9
  90. package/services/webhookConversationService.js +185 -0
  91. package/services/webhookSettingsValidator.js +61 -0
  92. package/template/email/checkpointReachedEmail.html +5 -0
  93. package/test/agentPromptsRoute.test.js +3 -2
  94. package/test/agentTokensQuota.js +237 -0
  95. package/test/botEventHitlFallback.js +19 -0
  96. package/test/chat21HandlerJoinWiring.js +59 -0
  97. package/test/dataTablesRoute.test.js +649 -2
  98. package/test/dateUtils.test.js +0 -3
  99. package/test/departmentService.js +8 -12
  100. package/test/{fileRoute.js → deprecated/_deprecated_fileRoute} +2 -2
  101. package/test/{kbsettingsRoute.js → deprecated/_deprecated_kbsettingsRoute} +3 -3
  102. package/test/faqRoute.js +6 -1
  103. package/test/faqkbRoute.js +14 -10
  104. package/test/faqkbV3Route.js +1 -1
  105. package/test/filepRoute.js +4 -50
  106. package/test/hitlParticipantListener.js +446 -0
  107. package/test/hitlQueueWiring.js +191 -0
  108. package/test/hitlWaitingReset.js +124 -0
  109. package/test/inviteParticipantsRoute.js +492 -0
  110. package/test/kbRoute.js +1 -1
  111. package/test/messageRoute.js +33 -21
  112. package/test/nativemcp.test.js +7 -1
  113. package/test/openaiRoute.js +2 -8
  114. package/test/publishFailLoud.test.js +414 -0
  115. package/test/quotaSnapshots.js +1044 -0
  116. package/test/removeParticipantsRoute.js +186 -0
  117. package/test/requestService.js +24 -16
  118. package/test/requestServiceRemoveParticipant.js +144 -0
  119. package/test/requestVisibility.test.js +186 -0
  120. package/test/scheduledStartFacade.js +184 -0
  121. package/test/scheduledStartHandover.js +95 -0
  122. package/test/scheduledStartPublish.js +320 -0
  123. package/test/scheduledStartRoute.js +163 -0
  124. package/test/scheduledStartRules.js +177 -0
  125. package/test/scheduledStartRun.js +299 -0
  126. package/test/scheduledStartSchedulerBackend.js +116 -0
  127. package/test/scheduledStartService.js +290 -0
  128. package/test/startPointsCleanup.js +69 -0
  129. package/test/startPointsRegression.js +169 -0
  130. package/test/startPointsRoute.js +158 -0
  131. package/test/webhookConversationRoute.js +347 -0
  132. package/test/webhookConversationService.js +261 -0
  133. package/test/webhookConversationTestUtils.js +55 -0
  134. package/utils/aiUtils.js +95 -1
  135. package/utils/dataTableUtils.js +99 -9
  136. package/utils/intentFieldNames.js +192 -0
  137. package/utils/quotaCliUtil.js +43 -0
  138. package/utils/requestVisibility.js +355 -0
  139. package/websocket/webSocketServer.js +36 -31
  140. /package/test/{imageRoute.js → deprecated/_deprecated_imageRoute} +0 -0
@@ -0,0 +1,156 @@
1
+ # Flow start points (web + webhook) — Design
2
+
3
+ Date: 2026-09-30 · Status: approved in conversation, pending written review
4
+ Repos: tiledesk-server, tiledesk-chatbot (tybot), design-studio. chat21-ionic unchanged.
5
+ Branch: continue on `feature/webhook-chatbot-hitl` (this changes the unreleased webhook-conversation feature,
6
+ spec `2026-09-15-webhook-chatbot-hitl-design.md`); `-PRE` ports afterwards as for the rest of that feature.
7
+
8
+ ## 1. Goal
9
+
10
+ A conversational chatbot flow can have several **start points**, each a dedicated box on the canvas with its
11
+ own arrow into the flow. First delivery: **web** (the widget interaction, as today) and **webhook**.
12
+ Later sub-projects, each with its own spec: **scheduled** and **integration triggers** (e.g. a new Gmail email
13
+ via the connector triggers).
14
+
15
+ ## 2. Decisions (user, 2026-09-30)
16
+
17
+ 1. Web start point = the widget interaction as it works today for conversational chatbots.
18
+ 2. Every start point is a dedicated box in design-studio; under the hood they all go through the webhook interface.
19
+ 3. At most one start point per type for now; leave room to allow several per type later.
20
+ 4. First spec: model + web + webhook. Scheduled and integration follow as separate specs.
21
+ 5. The webhook stays unique per chatbot: it is coupled to the chatbot definition.
22
+ 6. The start type is informative: it does not change how the flow runs. Type-specific input formats are
23
+ handled by a **mapping** attached to the start type.
24
+ 7. The department is defined while modelling the chatbot, not per start type.
25
+ 8. Start points are picked from a dedicated **palette section**; future external triggers go there too.
26
+ 9. Backward compatibility is mandatory: no regression for existing chatbots.
27
+
28
+ ## 3. Current state (verified 2026-09-30)
29
+
30
+ - The start block is identified only by the name `start` (design-studio `RESERVED_INTENT_NAMES`,
31
+ `cds-intent.component.ts:276`); `faq` has a unique index on `{id_project, id_faq_kb, intent_display_name}`.
32
+ - The widget and channels start a bot with the system message `/start` (`pubmodules/trigger/rulesTrigger.js`);
33
+ `event/botEvent.js` forwards `/start` / `\start` system messages; tybot resolves `/start` as the intent
34
+ named `start` and lets `attributes.action` override the text (`engine/TiledeskChatbot.js:135-147`);
35
+ `#<intent_id>` resolves by id (`engine/MongodbBotsDataSource.js:93-99`).
36
+ - `models/webhook.js`: one webhook per chatbot (unique `{id_project, chatbot_id}`); `mode: 'conversation'`
37
+ selects the conversation path (`services/webhookConversationService.js`), no mode = automation path
38
+ (`services/webhookService.run` → tybot `/block`, sync or async). Conversation mode ignores `block_id` and
39
+ always sends `/start`.
40
+ - `attributes.start_type` is written only as `'webhook'` and read nowhere.
41
+ - Publish/fork keeps `intent_id` (`routes/faq_kb.js:811, 925`), so a `block_id` stays valid on the published copy.
42
+ - `Change department` (tybot `department`) re-routes the conversation (`requestService.route` → `routeInternal`)
43
+ and replaces the participants: in a webhook flow it removes the bot.
44
+
45
+ ## 4. Data model and API (tiledesk-server)
46
+
47
+ ### 4.1 Webhook record
48
+ One per chatbot, unchanged identity (`webhook_id`, URL). Unchanged fields: `webhook_id`, `id_project`, `name`,
49
+ `chatbot_id`, `enabled` (master switch), `async`, `copilot`, `mode`, and `block_id` (automation path only).
50
+
51
+ New field (only when `mode === 'conversation'`):
52
+ ```
53
+ start_points: [{
54
+ type: 'webhook', // enum; 'web' is implicit (see 4.3); later 'scheduled', 'integration'
55
+ block_id: <intent_id>, // the start box
56
+ enabled: Boolean (default true),
57
+ mapping: { source_name: String } // requester name; home of future type-specific mappings
58
+ }]
59
+ ```
60
+ - At most one entry per `type`, enforced by the service (400), not by an index — allowing several per type later
61
+ is a rule change only.
62
+ - Removed from the conversation webhook: `department_id`, `source_name` (unreleased branch fields; no migration).
63
+
64
+ ### 4.2 Admin API (project-scoped, admin role)
65
+ - `GET /:projectid/webhooks/:chatbot_id` — includes `start_points`.
66
+ - `PUT /:projectid/webhooks/:chatbot_id/start_points/:type` body `{block_id, enabled?, mapping?, confirm?}` —
67
+ create/update one start point; creates the webhook (`mode: 'conversation'`) if missing.
68
+ Errors: 400 unknown type / bot not subtype `chatbot`; 404 `block_id` not a block of this chatbot;
69
+ 409 existing webhook without mode (automation) unless `confirm: true`, which converts it.
70
+ - `DELETE /:projectid/webhooks/:chatbot_id/start_points/:type` — removes it (404 if absent).
71
+ - Deleting a block that is a start point's `block_id` removes that start point (faq delete hook).
72
+
73
+ ### 4.3 Web start point
74
+ The existing `start` block, implicit: not stored in `start_points`. The widget/channel path is unchanged — still
75
+ the `/start` system message — so no existing bot is touched. Design-studio shows it as the "Web start" box.
76
+
77
+ ### 4.4 Start flow — `POST /webhook/:webhook_id` (and `/dev`), conversation mode
78
+ 1. Load the webhook; 404 unknown; 422 if `enabled` false or the `webhook` start point is missing/disabled
79
+ ("Webhook start is not enabled"); rate limit (429) as today.
80
+ 2. Upsert lead `webhook_<webhook_id>` named `mapping.source_name` (default: the webhook name).
81
+ 3. Create the request as today (`support-group-…`, bot participant, channel `webhook`), department = the
82
+ department whose bot is this chatbot, else the project default. Attributes
83
+ `{start_type: 'webhook', webhook_id, payload}`; dev keeps `sourcePage … &td_draft=true`.
84
+ 4. Send system message `/start` with `attributes.action = '#<block_id>'` (forwarded by botEvent as today;
85
+ tybot runs the box block, the text stays `/start`).
86
+ 5. 202 `{success: true, request_id}`.
87
+
88
+ Payload handling unchanged: non-object bodies stored as `payload.body`; tybot merges payload keys + `payload`.
89
+
90
+ ### 4.5 Automation webhooks
91
+ Untouched: no `start_points`, `block_id` + `/block` sync/async as today.
92
+
93
+ ## 5. tybot
94
+
95
+ - `start_type` becomes a flow variable (from request attributes, like `payload`); empty for web/channel starts.
96
+ - A `/start` whose `attributes.action` names a missing block falls back to the `start` block (warn log), so a
97
+ conversation never stalls.
98
+ - No other change.
99
+
100
+ ## 6. design-studio
101
+
102
+ ### 6.1 Palette section "Start points"
103
+ - New palette category next to Most used / AI / Flow / Integrations / Special, own icon, shown only for chatbot
104
+ subtype `chatbot`.
105
+ - Items: "Web" (shown as placed, not draggable; click focuses the `start` box) and "Webhook". Future: Scheduled,
106
+ integration triggers grouped by app.
107
+ - Dragging "Webhook" onto the canvas creates the start box at the drop point: a protected block (default name
108
+ "Webhook start", `attributes.start_point: 'webhook'`, one connector action like `start`), then
109
+ `PUT …/start_points/webhook {block_id, enabled: true}`; on 409 show the existing confirmation
110
+ ("This chatbot already has a webhook that runs a block without a conversation…") and retry with `confirm: true`.
111
+ - A type already in the flow is disabled ("Already in this flow"); clicking it focuses the box.
112
+
113
+ ### 6.2 Start boxes
114
+ - Identified by `attributes.start_point` + the start point's `block_id`, never by name (no new reserved names).
115
+ - Rendered like `start` (pill + type icon); no actions besides the outgoing connector; no rename as a normal
116
+ block; no "Start test from here". The `start` box is labelled "Web start".
117
+ - Webhook box panel: enabled switch, production + development URLs (copy, regenerate), Source name (mapping),
118
+ Test webhook start, hint that flows can read `start_type` and the payload variables. The "Start from webhook"
119
+ section of the `start` panel is removed.
120
+ - Delete (webhook box only): confirmation → `DELETE …/start_points/webhook` → delete the block.
121
+ - Test webhook start: from the webhook box; highlights that box.
122
+ - Help text of Change department: note that in conversations started by a webhook it re-routes and removes
123
+ the bot.
124
+
125
+ ## 7. chat21-ionic
126
+ No change: conversations still carry `request_channel: 'webhook'`; the requester shown is the mapping's source name.
127
+
128
+ ## 8. Backward compatibility
129
+
130
+ | Existing case | Guarantee |
131
+ |---|---|
132
+ | Conversational bots started by widget/channels | Path unchanged (`/start`); no data written to existing bots |
133
+ | Automation webhooks (webhook/copilot subtype or no mode) | Record, `block_id`, `/block` sync/async unchanged; converting needs explicit confirmation |
134
+ | Block names | Start boxes use an attribute marker; `start`, `webhook`, `defaultFallback` keep their meaning |
135
+ | Publish / fork / import / templates | `intent_id` preserved → `block_id` valid on published copies; webhooks not exported (as today) |
136
+ | Deleting bot | Deletes its webhook as today |
137
+ | Unreleased webhook-conversation fields | `department_id`, `source_name`, top-level conversation `block_id` dropped without migration |
138
+
139
+ ## 9. Testing
140
+
141
+ - tiledesk-server (mocha): start_points CRUD + validation (400/404/409, one per type), webhook call runs the box
142
+ (`attributes.action`) with `start_type`, 422 cases, start point removed on block delete, department rule.
143
+ Regression: automation webhook sync + async, widget `/start` launch, published bot, imported bot.
144
+ - tybot: `/start` + `action: '#box'` runs the box with `start_type` variable; missing block → `start`;
145
+ existing conversation tests unchanged.
146
+ - design-studio: palette category (one-per-type disabling, drop creates box + start point, 409 confirmation),
147
+ webhook box panel, delete with confirmation; `ng build` with no new warnings.
148
+ - End-to-end (local stack, see memory `hitl-local-e2e-stack`): drag Webhook start → curl runs the webhook box with
149
+ invite/buttons/Leave; widget still runs `start`; disable box → 422; delete box → 422; publish → works on the
150
+ published copy.
151
+
152
+ ## 10. Out of scope / later
153
+ - Scheduled start points (needs a cluster-safe scheduler; `node-schedule` is in-process only).
154
+ - Integration start points (Tiledesk-side subscription to connector `POST /triggers`, per-trigger mapping).
155
+ - Several start points of the same type.
156
+ - Moving the widget/channel start onto the start service.
@@ -0,0 +1,303 @@
1
+ # Scheduled start point — Design
2
+
3
+ Date: 2026-10-01 · Status: approved in conversation, pending written review
4
+ Repos: tiledesk-server, design-studio, tiledesk-agent-runtime. tiledesk-chatbot (tybot) and chat21-ionic unchanged.
5
+ Branch: `feature/scheduled-start-point`, created from `feature/webhook-chatbot-hitl` in each repo (start points
6
+ are not on master yet). Builds on `2026-09-30-flow-start-points-design.md` (web + webhook start points).
7
+
8
+ ## 1. Goal
9
+
10
+ A conversational chatbot (subtype `chatbot`) can start its flow on a schedule. A **Scheduled** start box, picked
11
+ from the Start points palette like Web and Webhook, opens a new conversation at the chosen times and runs the
12
+ flow from that box. The schedule runs on the scheduler already used for KB auto re-index (Apache
13
+ DolphinScheduler), and it is live only for the **published** version of the chatbot.
14
+
15
+ ## 2. Decisions (user, 2026-10-01)
16
+
17
+ 1. Scheduler: DolphinScheduler, the same mechanism as KB auto re-index (not the scheduled-jobs service, which is
18
+ only on master-COLLAUDO).
19
+ 2. Schedule expressiveness: presets plus time plus timezone — every N minutes/hours, daily at a time, weekly on
20
+ chosen weekdays at a time, monthly on a day at a time, in an IANA timezone. No raw cron.
21
+ 3. Overlap: every run starts a new conversation, even while the previous scheduled one is open.
22
+ 4. Run data: a Source name and a static payload, stored as JSON but edited through a visual builder (rows of
23
+ name / type / value), never a raw JSON editor.
24
+ 5. Integration approach A: tiledesk-server talks to DolphinScheduler directly and synchronously; the workflow's
25
+ HTTP task calls a token-protected server endpoint.
26
+ 6. Lifecycle: a schedule becomes triggerable only when the chatbot is published. Every change to the Scheduled
27
+ start point — timing, payload, Source name, the enabled switch, deleting the box — takes effect in production
28
+ at the next publish. Deleting the chatbot removes its schedule immediately.
29
+ 7. At most one Scheduled start point per chatbot (same rule as the other types).
30
+
31
+ ## 3. Current state (verified 2026-10-01)
32
+
33
+ - KB auto re-index: tiledesk-training-jobworker (`tiledesk/ReindexScheduler.js`, `utils/shedulerUtils.js`) creates a
34
+ DolphinScheduler process definition `auto-reindex-<content_id>` with one HTTP task
35
+ `POST {API_ENDPOINT}/webhook/kb/reindex` (body `content_id`, header `x-auth-token` = `KB_WEBHOOK_TOKEN`, both as
36
+ workflow global params), releases it, creates a schedule (Quartz cron from `daily|weekly|monthly` anchored on the
37
+ start time, `startTime` now+20s, `endTime` 2099-12-31) and sets it online. tiledesk-server only deletes them
38
+ (`services/aiReindexService.js`: schedule offline → definition offline → delete), from `/webhook/kb/reindex`.
39
+ Env: `SCHEDULER_BASEURL`, `SCHEDULER_TOKEN`, `SCHEDULER_PROJECT` (all set on stage).
40
+ - Stage runs `apache/dolphinscheduler-standalone-server:3.2.2` (tiledesk-argo `helm/values.yaml`). In 3.2 a schedule
41
+ carries `timezoneId`, and an online schedule must be taken offline to be updated.
42
+ - Publish (`PUT /:projectid/faq_kb/:id/publish`) forks a frozen copy (`root_id` = the chatbot, `trashed: true`),
43
+ points the chatbot's `url` at it and emits `faqbot.publish`. The fork keeps every `intent_id`. Production
44
+ conversations run the published copy; requests whose `sourcePage` contains `td_draft=true` run the draft.
45
+ - Start points: one webhook record per chatbot (`models/webhook.js`) with `start_points[]`; `startPointService`
46
+ (upsert/remove/find, one per type), routes `PUT|DELETE /:projectid/webhooks/:chatbot_id/start_points/:type`,
47
+ `startPointCleanupListener` on `faq.delete`, `webhookConversationService.start` (lead, department, request with
48
+ `start_type`, `/start` with `attributes.action = '#<block_id>'`).
49
+ - Invite/Remove human accept a request whose `channel.name === 'webhook'`
50
+ (`participantInviteService.js:45`, `participantRemoveService.js:27`).
51
+ - tybot already exposes `start_type` and the payload keys as flow variables and falls back to `start` when the
52
+ action's box is missing.
53
+
54
+ ## 4. Data model (tiledesk-server)
55
+
56
+ `start_points[]` gains the type `'scheduled'`:
57
+
58
+ ```
59
+ {
60
+ type: 'scheduled',
61
+ block_id: <intent_id of the Scheduled box>,
62
+ enabled: Boolean (default true),
63
+ mapping: {
64
+ source_name: String, // requester; empty → chatbot name (as webhook)
65
+ payload: { <key>: String|Number|Boolean } // static run data
66
+ },
67
+ schedule: {
68
+ frequency: 'interval' | 'daily' | 'weekly' | 'monthly',
69
+ every: Number, unit: 'minutes' | 'hours', // interval
70
+ time: 'HH:MM', // daily, weekly, monthly
71
+ weekdays: ['mon'..'sun'], // weekly, at least one
72
+ day_of_month: 1..28 | 'last', // monthly
73
+ timezone: String // IANA zone
74
+ },
75
+ live: { // server-owned: what is active in DolphinScheduler
76
+ enabled, block_id, mapping, schedule, // snapshot taken at publish
77
+ published_bot_id, published_at,
78
+ workflow_code, schedule_id,
79
+ error // last sync error message, absent when fine
80
+ }
81
+ }
82
+ ```
83
+
84
+ Everything except `live` is the **draft**, edited through the API. `live` is written only by the publish sync and
85
+ the sync/retry endpoint, never accepted from a client.
86
+
87
+ Validation (400, message naming the field):
88
+ - `interval`: `unit: 'minutes'` with `every` ∈ {5, 10, 15, 20, 30}; `unit: 'hours'` with `every` ∈ 1..12.
89
+ - `daily|weekly|monthly`: `time` matches `^([01]\d|2[0-3]):[0-5]\d$`; weekly needs ≥ 1 weekday; monthly
90
+ `day_of_month` ∈ 1..28 or `'last'`.
91
+ - `timezone`: a valid IANA zone (`Intl.DateTimeFormat(undefined, { timeZone })` does not throw).
92
+ - `payload`: a plain object, flat; keys match `^[A-Za-z_][A-Za-z0-9_]*$`; values string, finite number or boolean;
93
+ at most 50 keys and 8 KB serialized.
94
+
95
+ ## 5. API (tiledesk-server)
96
+
97
+ Existing routes, new type value:
98
+ - `PUT /:projectid/webhooks/:chatbot_id/start_points/scheduled` body `{ block_id, enabled?, mapping?, schedule }`
99
+ (`schedule` required on create) — saves the draft only, no scheduler call. Same rules as webhook start points:
100
+ one per type, `block_id` of this chatbot (404), subtype `chatbot` (400), automation webhook → 409 unless
101
+ `confirm`. Response: the webhook, plus `schedule_summary` (draft) and `live_summary` (live, when present),
102
+ e.g. "Every weekday at 08:30 (Europe/Rome)".
103
+ - `DELETE …/start_points/scheduled` — removes the draft entry; `live` and the DolphinScheduler objects stay until
104
+ the next publish, which removes them (§7).
105
+ - `GET /:projectid/webhooks/:chatbot_id` — adds `scheduled_available: Boolean` and, for a live scheduled start point,
106
+ `next_runs` (next 3 fire times from DolphinScheduler's schedule preview; omitted on error).
107
+
108
+ New routes:
109
+ - `POST /:projectid/webhooks/:chatbot_id/start_points/scheduled/test` (admin) — runs the **draft** once now: starts a
110
+ conversation as in §8 but with the draft mapping and `sourcePage … dev=true&td_draft=true`, so the draft bot
111
+ runs. Answers 202 `{ request_id }`. Works whether or not the chatbot is published.
112
+ - `POST /:projectid/webhooks/:chatbot_id/start_points/scheduled/sync` (admin) — re-runs the publish sync of §7
113
+ against the latest published copy (the retry behind "Couldn't update the live schedule").
114
+ - `POST /webhook/scheduled-start` (public, called by DolphinScheduler) — §8.
115
+
116
+ Errors: `{ success: false, error }`. Scheduler not configured (`SCHEDULER_BASEURL`, `SCHEDULER_TOKEN`,
117
+ `SCHEDULER_PROJECT` or `SCHEDULED_START_TOKEN` missing): the `scheduled` routes answer 503
118
+ "Scheduled starts are not available on this installation" and `scheduled_available` is false.
119
+
120
+ ## 6. Scheduler integration (tiledesk-server)
121
+
122
+ `services/dolphinSchedulerClient.js` — thin REST client (axios, header `token`), same endpoints as KB re-index:
123
+ create/release/delete process definition, create/update/online/offline schedule, find definition by name,
124
+ schedule preview. `AiReindexService` is not changed.
125
+
126
+ `services/scheduledStartService.js` — owns the DolphinScheduler objects of scheduled starts:
127
+ - Workflow per chatbot, name `auto-start-<chatbot_id>`: one HTTP task `POST {API_URL}/webhook/scheduled-start`,
128
+ body params `id_project`, `chatbot_id`, header `x-auth-token: ${token}`, values as workflow global params
129
+ (`SCHEDULED_START_TOKEN`), `connectTimeout`/`socketTimeout` 60000, no retries.
130
+ - Schedule: `crontab = toQuartzCron(schedule)`, `timezoneId = schedule.timezone`, `startTime` = now,
131
+ `endTime` = 2099-12-31 23:59:59, `failureStrategy: CONTINUE`, `warningType: NONE`.
132
+ - `toQuartzCron(schedule)` (pure):
133
+ - interval minutes N → `0 0/N * * * ? *`; hours N → `0 0 0/N * * ? *`
134
+ - daily HH:MM → `0 MM HH * * ? *`
135
+ - weekly → `0 MM HH ? * MON,WED,FRI *`
136
+ - monthly day D → `0 MM HH D * ? *`; `last` → `0 MM HH L * ? *`
137
+ - Operations: `activate(live)` (create definition → release online → create schedule → schedule online; on a
138
+ failing step, undo the steps done), `update(live)` (schedule offline → update → online), `deactivate(live)`
139
+ (schedule offline), `destroy(live)` (schedule offline → definition offline → delete). A stored
140
+ `workflow_code`/`schedule_id` that DolphinScheduler no longer knows is recreated instead of failing.
141
+
142
+ Running is cluster-safe: DolphinScheduler fires once per due time; whichever server pod receives the HTTP call
143
+ handles it. No in-process timer.
144
+
145
+ ## 7. Publish lifecycle
146
+
147
+ A listener on `faqbot.publish` (registered in `app.js` next to the start point cleanup listener) syncs the live
148
+ schedule with the draft, using the published copy:
149
+
150
+ | Draft scheduled start point | Published copy has `block_id` | Result |
151
+ |---|---|---|
152
+ | present, enabled | yes | `activate` (no live yet) or `update` (live exists); `live` = snapshot of the draft + published ids |
153
+ | present, disabled | — | `deactivate`; `live.enabled = false` |
154
+ | present | no | `destroy`; `live` cleared |
155
+ | absent (box deleted) | — | `destroy`; `live` cleared |
156
+
157
+ - Publishing never fails because of the scheduler: a sync error is stored in `live.error`, logged, and shown by the
158
+ panel with a Retry (`…/scheduled/sync`).
159
+ - Deleting the chatbot (the existing faq_kb delete path) calls `destroy` immediately and deletes the webhook as
160
+ today.
161
+ - Restoring a release (`publish` with `restore_from`) goes through the same listener.
162
+
163
+ ## 8. A scheduled run
164
+
165
+ `POST /webhook/scheduled-start`, body `{ id_project, chatbot_id }`, header `x-auth-token`:
166
+ 1. Missing/wrong token → 401. Token not configured → 503.
167
+ 2. Chatbot, webhook record, scheduled start point or `live` missing, or `live.enabled` false → 410 and the
168
+ workflow is taken offline (a dangling workflow removes itself, as `/webhook/kb/reindex` does for deleted
169
+ content). Rate limit as webhook calls (429).
170
+ 3. Start the conversation through `webhookConversationService.start` with the scheduled start point:
171
+ - lead `scheduled_<chatbot_id>`, fullname `live.mapping.source_name` || chatbot name;
172
+ - department: same rule as webhook starts (`resolveChatbotDepartment`);
173
+ - channel `webhook` (chat21-ionic shows it as a webhook conversation; Invite/Remove human accept it);
174
+ - request attributes `{ start_type: 'scheduled', payload: live.mapping.payload, scheduled_at: <ISO time> }`;
175
+ - system `/start` with `attributes.action = '#' + live.block_id`;
176
+ - not a draft request, so the published copy runs.
177
+ 4. 202 `{ success: true, request_id }`.
178
+
179
+ Every run is a new conversation (decision 3). A run missed while DolphinScheduler is down is not recovered.
180
+
181
+ tybot needs no change: `{{start_type}}` is `scheduled`, each payload key is a variable, `{{payload}}` the object,
182
+ and the missing-box fallback to `start` still applies.
183
+
184
+ The 422 messages of Invite/Remove human become "… requires a conversation started by a webhook or a schedule".
185
+
186
+ ## 9. design-studio
187
+
188
+ - Palette Start points: item **Scheduled** (clock icon) next to Web and Webhook; one per type ("Already in this
189
+ flow"); subtype `chatbot` only; hidden when `scheduled_available` is false.
190
+ - Dropping it creates the protected block (`attributes.start_point: 'scheduled'`, name "Scheduled start"), then
191
+ `PUT …/start_points/scheduled` with defaults: daily 09:00, browser timezone, enabled, Source name = chatbot
192
+ name, empty payload. Same failure handling as the webhook box (the block is removed on error).
193
+ - The box behaves like the other start boxes (toolbar "⋮" only, one outgoing connector, delete from its panel,
194
+ no copy/paste) and shows the draft summary and a badge: **Live**, **Changes not live**, **Not live**.
195
+ - Panel:
196
+ 1. Status: "Live — every weekday at 08:30 (Europe/Rome) · next run Tue 08:30" / "Not live — publish to activate"
197
+ / "Changes not live — publish to apply" / "Couldn't update the live schedule — Retry".
198
+ 2. Enabled switch, hint "Takes effect when you publish".
199
+ 3. Schedule: Repeat (Every N minutes/hours, Daily, Weekly, Monthly) with the matching fields (interval
200
+ selector, time picker, weekday chips, day of month 1–28 or Last day), Timezone (searchable, browser default),
201
+ live summary sentence.
202
+ 4. Source name (placeholder: chatbot name).
203
+ 5. Data for each run — the visual payload builder: rows Name · Type (Text / Number / True-false) · Value ·
204
+ remove, "+ Add field"; names validated as typed (pattern, no duplicates); converted to and from the stored
205
+ JSON; hint "Each field is a variable in the flow, e.g. {{check}} — {{start_type}} is scheduled".
206
+ 6. Test scheduled start (draft, now; opens the conversation and the logs, like Test webhook start).
207
+ 7. Delete (confirmation; help text: a live schedule stops at the next publish).
208
+ - Saving: each change saves the draft with `PUT` (debounced), error toast on failure; invalid input is flagged and
209
+ not sent.
210
+ - Help text: stopping a live schedule takes a publish (switch off, then publish).
211
+ - master-pre: read-only mode disables the palette item and the panel (pre-only adaptation, as for the webhook box).
212
+
213
+ ## 10. Vibe coder (tiledesk-agent-runtime)
214
+
215
+ - Prompt §1c: the **Scheduled start box** (`attributes.start_point: "scheduled"`) — an entry point wired like
216
+ `start`, never deleted or renamed; the agent cannot create it nor set its timing or data (the user does, from the
217
+ palette and the panel); a run reads `{{start_type}}` = `scheduled` and the payload fields; it is live only after
218
+ publish; nobody is on the other side until the flow invites someone; Invite/Remove human work.
219
+ - Back-office pattern: a scheduled example (a periodic check that invites a person only when something is wrong).
220
+ - Catalogue overlay of `invite_human`/`remove_human`: "started by a webhook or a schedule".
221
+
222
+ ## 11. Backward compatibility
223
+
224
+ | Existing case | Guarantee |
225
+ |---|---|
226
+ | Web and webhook start points | Unchanged; their code paths do not read `schedule`/`live` |
227
+ | KB auto re-index | Untouched: `AiReindexService`, `/webhook/kb/reindex`, jobworker unchanged |
228
+ | Installations without DolphinScheduler / token | Feature off: palette item hidden, routes 503, publish unaffected |
229
+ | Publish | Never fails because of the scheduler (§7) |
230
+ | Chatbots without scheduled start | No listener work beyond a lookup; no scheduler calls |
231
+
232
+ ## 12. Testing
233
+
234
+ - Server (mocha): `toQuartzCron` for every frequency and edge (5-minute floor, `last`, weekday lists); validation
235
+ 400s; DolphinScheduler client against a stub HTTP server, including rollback on a failing step and recreate on
236
+ unknown ids; publish listener (activate, update, deactivate, destroy when box missing or deleted, `live.error` on
237
+ failure, restore_from); `/webhook/scheduled-start` (401, 503, 410 + offline, 429, conversation with published
238
+ copy, payload, `start_type`, `scheduled_at`); test-run on the draft; sync endpoint; chatbot delete destroys;
239
+ regression on webhook start points and `/webhook/kb/reindex`.
240
+ - design-studio (specs): schedule summary and validation; payload builder rows ↔ JSON; palette rules and
241
+ `scheduled_available`; panel states.
242
+ - runtime: prompt/catalogue assertions in `tests/test_examples.py`.
243
+ - End-to-end: local stack plus `apache/dolphinscheduler-standalone-server:3.2.2` in docker — add the box, test-run
244
+ the draft, publish, see a run fire, change the time without publishing (old schedule keeps firing), switch off +
245
+ publish (runs stop), delete the chatbot (workflow gone). Then the same on stage with test agents, checking the
246
+ workflow in the DolphinScheduler UI.
247
+
248
+ ## 13. Delivery
249
+
250
+ Branch `feature/scheduled-start-point` from `feature/webhook-chatbot-hitl` in tiledesk-server, design-studio and
251
+ tiledesk-agent-runtime; then the usual path (tests, push, merge into the pre branches, tags, argo stage). Stage
252
+ needs `SCHEDULED_START_TOKEN` on the server and worker deployments (argo values or secret), checked before deploy.
253
+
254
+ ## 14. Out of scope / later
255
+
256
+ - Several scheduled start points per chatbot.
257
+ - One-off runs at a date ("once"), as another frequency.
258
+ - Catching up runs missed while DolphinScheduler is down.
259
+ - Nested payload objects in the builder.
260
+ - Moving KB auto re-index onto `dolphinSchedulerClient`.
261
+
262
+ ## 15. Running on tiledesk-scheduler-service (added 2026-10-13)
263
+
264
+ Scheduled starts can run on tiledesk-scheduler-service ≥ 1.1.0 instead of DolphinScheduler. `SCHEDULED_START_BACKEND` selects the backend per call:
265
+
266
+ | `SCHEDULED_START_BACKEND` | Required env on server + worker | Code |
267
+ |---|---|---|
268
+ | `dolphin` (default, also used for unknown values) | `SCHEDULER_BASEURL`, `SCHEDULER_TOKEN`, `SCHEDULER_PROJECT`, `SCHEDULED_START_TOKEN` | `services/scheduledStart/dolphinBackend.js` |
269
+ | `scheduler` | `SCHEDULER_SERVICE_URL`, `SCHEDULER_API_TOKEN`, `SCHEDULED_START_TOKEN` | `services/scheduledStart/schedulerBackend.js` |
270
+
271
+ `services/scheduledStartService.js` is a façade with the same API as before (`activate`, `update`, `deactivate`, `destroy`, `nextRuns`, `offlineFor`, `destroyFor`) plus `hasLive` and `liveIds`.
272
+
273
+ **How the scheduler backend works**
274
+ - One cron slot per chatbot: `projectId = id_project`, `flowId = auto-start-<chatbotId>`, `target = scheduledStart`, payload `{ id_project, chatbot_id }`.
275
+ - The cron is a standard 5-field expression from `toCron5`, with the schedule's IANA `timezone`.
276
+ - The scheduler adds `x-auth-token: $SCHEDULED_START_TOKEN` when the job runs and calls `$TILEDESK_API_URL/webhook/scheduled-start`. The token is never sent to the scheduler, and rotating it needs no re-sync. **Rotate it on the server and on the scheduler worker at the same time**: a mismatch gets a 401 on every run, the target never retries, and the bots silently stop.
277
+ - The `scheduledStart` target never retries, so there's at most one conversation per tick. A 410 from the run endpoint pauses the slot.
278
+ - `scheduled_live` stores `{ backend: 'scheduler', job_id, id_project, chatbot_id }` instead of `{ workflow_code, schedule_id }`.
279
+ - `next_runs` comes from `POST /schedules/preview`, formatted as `YYYY-MM-DD HH:mm:ss` wall-clock in the schedule's timezone, as with DolphinScheduler.
280
+
281
+ **Handover between backends**
282
+ - When `scheduled_live` holds ids of the backend that is *not* selected, the next `update` / `deactivate` / `destroy` / `offlineFor` / `destroyFor` works on the selected backend first, then removes the other backend's schedule.
283
+ - If that removal fails, the other backend's ids **stay** in `scheduled_live` next to the new ones, and every later call retries. Dropping them would leave that schedule firing with nothing left to remove it, and the bot would start twice per tick.
284
+ - The ids are dropped only when the removal succeeds, or when the other backend's env is no longer set.
285
+ - With no ids stored (e.g. a trashed chatbot whose webhook is already gone), `offlineFor` / `destroyFor` look the schedule up on both backends.
286
+ - All of this works in both directions, so a rollback to `dolphin` also removes the scheduler slot.
287
+ - The new schedule is created before the old one is removed: a tick that falls in that window of a few hundred milliseconds fires on both. Migrate away from tick times.
288
+
289
+ **Cutover (stage, then prod)**
290
+ 1. Deploy tiledesk-scheduler-service ≥ 1.1.0 (API + worker).
291
+ - Scheduler env: `SCHEDULER_API_TOKEN`, `TILEDESK_API_URL` (server base URL reachable from the worker), `SCHEDULED_START_TOKEN` (same value as the server).
292
+ - Check `GET /health/ready` and `GET /targets` (lists `scheduledStart`).
293
+ 2. Server + server-worker env (argo values / `config-scheduler`):
294
+ - `SCHEDULED_START_BACKEND=scheduler`
295
+ - `SCHEDULER_SERVICE_URL=http://<release>-scheduler-api`
296
+ - `SCHEDULER_API_TOKEN` (same as the scheduler).
297
+ - Keep the DolphinScheduler variables until step 4, so the handover can remove the old workflows.
298
+ 3. Migrate every live bot with `POST /:projectid/webhooks/:chatbot_id/start_points/scheduled/migrate`, for each webhook with a `scheduled_live`.
299
+ - Migrate recreates the schedule from the **stored live snapshot**, so unpublished draft edits are not published. Do **not** use `/sync` here: it reads the draft, so it would publish draft edits, and it destroys the live schedule if the draft was deleted.
300
+ - It creates the slot (or pauses it when `enabled` is false) and removes the `auto-start-<id>` DolphinScheduler workflow.
301
+ - Check: the scheduler's `GET /jobs?projectId=` shows the slot; `scheduled_live` has no `workflow_code` left (if it still does, the DolphinScheduler cleanup failed: run migrate again); DolphinScheduler has no `auto-start-*` left; the first run creates a conversation with `start_type=scheduled`.
302
+ - Bots that are not migrated keep firing from DolphinScheduler until their next publish, which moves them.
303
+ 4. Once no `auto-start-*` workflow is left, the DolphinScheduler variables can be removed from the server. DolphinScheduler itself stays while KB re-index still uses it.