@iwonz/ywtc 3.5.0 → 3.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/README.md +6 -4
- package/dist/content-model.js +315 -24
- package/dist/content-model.js.map +1 -1
- package/dist/conversion.js +46 -10
- package/dist/conversion.js.map +1 -1
- package/dist/domain.d.ts +7 -1
- package/dist/domain.js.map +1 -1
- package/dist/filesystem-export.js +1 -1
- package/dist/filesystem-export.js.map +1 -1
- package/dist/layout.d.ts +18 -0
- package/dist/layout.js +134 -0
- package/dist/layout.js.map +1 -0
- package/dist/migrator.js +2 -1
- package/dist/migrator.js.map +1 -1
- package/dist/report-ui/index.html +1 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,20 @@
|
|
|
2
2
|
|
|
3
3
|
Notable changes to `ywtc` are documented in this file. Versions follow [Semantic Versioning](https://semver.org/).
|
|
4
4
|
|
|
5
|
+
## [3.6.0](https://github.com/iwonz/ywtc/compare/v3.5.0...v3.6.0) (2026-10-16)
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
### Features
|
|
9
|
+
|
|
10
|
+
* **migration:** preserve empty pages and native layouts ([e11185e](https://github.com/iwonz/ywtc/commit/e11185ee0fa81b26e39ceebbd49f6aee7ae22cd8))
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### Maintenance
|
|
14
|
+
|
|
15
|
+
* **homebrew:** update formula to v3.5.0 ([6d98178](https://github.com/iwonz/ywtc/commit/6d9817809008283cbd797640825a1fe676d1ddae))
|
|
16
|
+
* **release:** continue semantic layout publication ([4e9119a](https://github.com/iwonz/ywtc/commit/4e9119a5078b42c463a0d990b66508495083a012))
|
|
17
|
+
* **release:** publish semantic layout support ([84c34b8](https://github.com/iwonz/ywtc/commit/84c34b84216454f653ce00e26b756db3c1c3f311))
|
|
18
|
+
|
|
5
19
|
## [3.5.0](https://github.com/iwonz/ywtc/compare/v3.4.1...v3.5.0) (2026-10-16)
|
|
6
20
|
|
|
7
21
|
|
package/README.md
CHANGED
|
@@ -14,13 +14,13 @@
|
|
|
14
14
|
|
|
15
15
|
| Исходный материал | Как переносится |
|
|
16
16
|
| --- | --- |
|
|
17
|
-
| Страницы и структура | Одна страница, полное структурное поддерево или граф связанных страниц. Иерархия восстанавливается, циклы обрабатываются без повторного переноса. |
|
|
18
|
-
| Текст и оформление | Заголовки, абзацы,
|
|
17
|
+
| Страницы и структура | Одна страница, полное структурное поддерево или граф связанных страниц. Иерархия восстанавливается, циклы обрабатываются без повторного переноса. Пустая страница-раздел переносится как корректная пустая страница, а не как fallback. |
|
|
18
|
+
| Текст и оформление | Заголовки, абзацы, жирное/курсивное/зачёркнутое и цветное начертание, списки и задачи, цитаты, разделители, inline/preformatted code с языком и обычные таблицы преобразуются в нативную или семантически эквивалентную разметку destination. |
|
|
19
|
+
| Секции и колонки | YFM `layout/block` и Confluence page layouts сохраняются нативными секциями и колонками в Yandex Wiki и Confluence. HTML использует адаптивную сетку; Markdown сохраняет все ячейки по порядку и явно отмечает линеаризацию как degradation. |
|
|
19
20
|
| Ссылки и якоря | Ссылки между выбранными страницами переписываются на destination, якоря сохраняются. Внешние ссылки и ссылки за пределы выбранного набора остаются исходными. |
|
|
20
21
|
| Вложения | Переносятся все доступные через API для скачивания файлы, включая PNG/JPEG/GIF/WebP/SVG, PDF, ZIP, DOC/DOCX, XLS/XLSX и PPT/PPTX. Файлы передаются потоково, совпадения имён разрешаются стабильно, размер и SHA-256 проверяются. |
|
|
21
22
|
| Изображения | Прикреплённые изображения становятся attachments destination или локальными файлами экспорта. Внешние изображения остаются внешними; подписи и размеры сохраняются, если destination это поддерживает. |
|
|
22
|
-
| Специальная разметка
|
|
23
|
-
| Специальная разметка Confluence | Безопасный Storage Format сохраняется при переносе между Confluence. В других направлениях code становится блоком кода, info/tip/warning/note — callout, expand — раскрываемым или статическим блоком, anchor — якорем; остальные macros получают fallback. |
|
|
23
|
+
| Специальная разметка | В обе стороны сопоставляются notes ↔ `info`/`tip`/`warning`/`note`, cuts ↔ `expand`, tree ↔ `children`, задачи, code, anchors, page/attachment links и изображения. Tabs становятся статическими секциями, Grid — статической таблицей. Безопасный Storage Format и YFM сохраняются нативно при переносе внутри одного провайдера. |
|
|
24
24
|
| Комментарии | По умолчанию все доступные комментарии страницы собираются в один архивный комментарий: отдельные раскрываемые ветки сохраняют участников, авторов, даты, статусы, inline-контекст, удалённые записи и вложенность ответов. Вложения, на которые есть ссылки в тексте комментариев, складываются в один проверяемый ZIP с hash-prefixed именами и manifest. Для Markdown/HTML создаются локальные секции и ZIP. |
|
|
25
25
|
| Конструкции без точного эквивалента | Создаётся читаемый fallback и сохраняется raw snapshot. Проверка destination отражает упрощения как degradation или `partial`, поэтому потеря не маскируется успешным статусом. |
|
|
26
26
|
|
|
@@ -33,6 +33,8 @@
|
|
|
33
33
|
| Комментарии непосредственно к attachments, blog posts и custom content | Не входят в архив страницы; читаются только footer/inline page comments. |
|
|
34
34
|
| Пользовательские сущности, watchers, права и ограничения доступа | Не создаются; исходное авторство комментариев сохраняется в архивном тексте, а destination-объекты создаются пользователем текущего токена и наследуют права destination. |
|
|
35
35
|
| Удалённые, недоступные или не возвращённые source API страницы | Не восстанавливаются; недоступность и пропуски фиксируются в state и отчёте. |
|
|
36
|
+
| Точная пиксельная геометрия, произвольный CSS и активные скрипты | Не исполняются и не обещают визуальную идентичность. Содержимое и поддерживаемая структура сохраняются нативно или статически; неподдерживаемая часть получает читаемый fallback и raw snapshot. |
|
|
37
|
+
| Сторонние и неизвестные macros/виджеты | Не подменяются приблизительным macro без доказанного эквивалента: сохраняются безопасным статическим представлением и raw snapshot с явной degradation. |
|
|
36
38
|
| Внешние файлы и изображения, которых нет среди доступных для скачивания ресурсов | Не копируются; сохраняется внешняя или исходная ссылка. |
|
|
37
39
|
| Активные скрипты, iframe, формы и интерактивное поведение embed-объектов | Не исполняются в destination; вместо них сохраняется безопасное статическое представление или raw snapshot. |
|
|
38
40
|
| Пиксельная идентичность и поведение vendor-specific элементов без эквивалента | Используется ближайшее семантическое представление; различия явно отмечаются в отчёте. |
|
package/dist/content-model.js
CHANGED
|
@@ -12,6 +12,7 @@ import remarkStringify from 'remark-stringify';
|
|
|
12
12
|
import { unified } from 'unified';
|
|
13
13
|
import { convertYfmToStorage } from './conversion.js';
|
|
14
14
|
import { normalizeWikiSlug, resolveRelativeWikiSlug } from './input.js';
|
|
15
|
+
import { hasYfmLayouts, parseYfmDocumentSegments } from './layout.js';
|
|
15
16
|
const linkifier = new LinkifyIt();
|
|
16
17
|
function sha256(value) {
|
|
17
18
|
return createHash('sha256').update(value).digest('hex');
|
|
@@ -201,31 +202,76 @@ export function extractPageReferences(content, format) {
|
|
|
201
202
|
}
|
|
202
203
|
return uniqueReferences(references);
|
|
203
204
|
}
|
|
205
|
+
const confluenceLayoutCellCounts = new Map([
|
|
206
|
+
['single', 1],
|
|
207
|
+
['two_equal', 2],
|
|
208
|
+
['two_left_sidebar', 2],
|
|
209
|
+
['two_right_sidebar', 2],
|
|
210
|
+
['three_equal', 3],
|
|
211
|
+
['three_with_sidebars', 3],
|
|
212
|
+
]);
|
|
204
213
|
function storageToSemanticHtml(storage, warnings) {
|
|
205
214
|
const $ = load(`<main data-ywtc-root>${storage}</main>`, { xmlMode: false }, false);
|
|
206
215
|
$('script, iframe, object, embed, form').each((_index, element) => {
|
|
207
216
|
warnings.push(`Active or embedded element <${element.tagName}> was replaced by a safe fallback`);
|
|
208
217
|
$(element).replaceWith(`<pre data-ywtc-fallback>${escapeHtml($.html(element))}</pre>`);
|
|
209
218
|
});
|
|
219
|
+
$('ac\\:layout').toArray().reverse().forEach((element) => {
|
|
220
|
+
const layout = $(element);
|
|
221
|
+
if (layout.parents('ac\\:layout').length > 0)
|
|
222
|
+
warnings.push('Nested Confluence layout was preserved as a readable layout fallback');
|
|
223
|
+
const sections = layout.children('ac\\:layout-section').toArray().map((section) => {
|
|
224
|
+
const item = $(section);
|
|
225
|
+
const requestedType = item.attr('ac:type') ?? 'unsupported';
|
|
226
|
+
const expectedCells = confluenceLayoutCellCounts.get(requestedType);
|
|
227
|
+
const cells = item.children('ac\\:layout-cell').toArray();
|
|
228
|
+
const sectionType = expectedCells === cells.length ? requestedType : 'unsupported';
|
|
229
|
+
if (sectionType === 'unsupported') {
|
|
230
|
+
warnings.push(`Confluence layout section "${requestedType}" with ${cells.length} cells was preserved as an unsupported-width layout fallback`);
|
|
231
|
+
}
|
|
232
|
+
const body = cells.map((cell) => `<div class="ywtc-layout-cell">${$(cell).html() ?? ''}</div>`).join('');
|
|
233
|
+
return `<section class="ywtc-layout-section ywtc-layout-${escapeHtml(sectionType)} ywtc-layout-columns-${cells.length}" data-ywtc-layout-type="${escapeHtml(sectionType)}">${body}</section>`;
|
|
234
|
+
}).join('');
|
|
235
|
+
layout.replaceWith(`<section class="ywtc-layout" data-ywtc-layout-source="confluence">${sections}</section>`);
|
|
236
|
+
});
|
|
210
237
|
$('ac\\:structured-macro').toArray().reverse().forEach((element) => {
|
|
211
238
|
const macro = $(element);
|
|
212
239
|
const name = macro.attr('ac:name') ?? 'unknown';
|
|
213
240
|
const richBody = macro.children('ac\\:rich-text-body').html() ?? '';
|
|
214
241
|
const plainBody = macro.children('ac\\:plain-text-body').text();
|
|
215
|
-
const
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
macro.replaceWith(`<
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
242
|
+
const parameters = new Map(macro.children('ac\\:parameter').toArray().map((item) => [$(item).attr('ac:name') ?? '', $(item).text().trim()]));
|
|
243
|
+
const title = parameters.get('title') ?? '';
|
|
244
|
+
if (name === 'code') {
|
|
245
|
+
const language = parameters.get('language');
|
|
246
|
+
macro.replaceWith(`<pre><code${language ? ` class="language-${escapeHtml(language)}"` : ''}>${escapeHtml(plainBody)}</code></pre>`);
|
|
247
|
+
}
|
|
248
|
+
else if (name === 'expand') {
|
|
249
|
+
macro.replaceWith(`<details class="ywtc-expand" data-ywtc-title="${escapeHtml(title || 'Details')}" open><summary>${escapeHtml(title || 'Details')}</summary>${richBody}</details>`);
|
|
250
|
+
}
|
|
251
|
+
else if (['info', 'tip', 'warning', 'note'].includes(name)) {
|
|
252
|
+
macro.replaceWith(`<aside class="ywtc-note ywtc-${name}" data-ywtc-note-type="${name}" data-ywtc-title="${escapeHtml(title)}">${richBody}</aside>`);
|
|
253
|
+
}
|
|
254
|
+
else if (name === 'anchor') {
|
|
255
|
+
macro.replaceWith(`<span class="ywtc-anchor" id="${escapeHtml(macro.children('ac\\:parameter').first().text())}"></span>`);
|
|
256
|
+
}
|
|
257
|
+
else if (name === 'children' && !parameters.get('page')) {
|
|
258
|
+
const depth = Number(parameters.get('depth'));
|
|
259
|
+
macro.replaceWith(`<nav class="ywtc-children" data-ywtc-depth="${Number.isInteger(depth) && depth > 0 ? depth : ''}" data-ywtc-sort="${escapeHtml(parameters.get('sort') ?? '')}" data-ywtc-reverse="${parameters.get('reverse') === 'true' ? 'true' : 'false'}"></nav>`);
|
|
260
|
+
}
|
|
224
261
|
else {
|
|
225
262
|
warnings.push(`Confluence macro "${name}" was converted to a static fallback`);
|
|
226
263
|
macro.replaceWith(`<aside class="ywtc-fallback"><strong>${escapeHtml(name)}</strong>${richBody || `<pre>${escapeHtml(plainBody)}</pre>`}</aside>`);
|
|
227
264
|
}
|
|
228
265
|
});
|
|
266
|
+
$('ac\\:task-list').toArray().reverse().forEach((element) => {
|
|
267
|
+
const tasks = $(element).children('ac\\:task').toArray().map((task) => {
|
|
268
|
+
const item = $(task);
|
|
269
|
+
const status = item.children('ac\\:task-status').text().trim() === 'complete' ? 'complete' : 'incomplete';
|
|
270
|
+
const body = item.children('ac\\:task-body').html() ?? '';
|
|
271
|
+
return `<li class="ywtc-task" data-ywtc-task-status="${status}">${body}</li>`;
|
|
272
|
+
}).join('');
|
|
273
|
+
$(element).replaceWith(`<ul class="ywtc-task-list">${tasks}</ul>`);
|
|
274
|
+
});
|
|
229
275
|
$('ac\\:image').each((_index, element) => {
|
|
230
276
|
const image = $(element);
|
|
231
277
|
const attachment = image.children('ri\\:attachment').attr('ri:filename');
|
|
@@ -319,7 +365,7 @@ function extractTreeDirectives(content) {
|
|
|
319
365
|
function containsEmptyYfmHeading(content) {
|
|
320
366
|
return /^\s{0,3}#{1,6}\s*$/m.test(content);
|
|
321
367
|
}
|
|
322
|
-
function
|
|
368
|
+
function yfmFragmentToHtml(content, warnings, normalizations) {
|
|
323
369
|
try {
|
|
324
370
|
const result = transform(content || '', {
|
|
325
371
|
allowHTML: true,
|
|
@@ -361,10 +407,65 @@ function yfmToHtml(content, warnings, normalizations = []) {
|
|
|
361
407
|
return `<pre data-ywtc-fallback><code>${escapeHtml(content)}</code></pre>`;
|
|
362
408
|
}
|
|
363
409
|
}
|
|
410
|
+
function yfmToHtml(content, warnings, normalizations = []) {
|
|
411
|
+
const segments = parseYfmDocumentSegments(content);
|
|
412
|
+
if (!hasYfmLayouts(segments))
|
|
413
|
+
return yfmFragmentToHtml(content, warnings, normalizations);
|
|
414
|
+
return segments.map((segment) => {
|
|
415
|
+
if (segment.type === 'content') {
|
|
416
|
+
return segment.raw.trim() ? yfmFragmentToHtml(segment.raw, warnings, normalizations) : '';
|
|
417
|
+
}
|
|
418
|
+
const gap = segment.gap?.replace(/[^a-z0-9_-]/gi, '') || 'default';
|
|
419
|
+
const cells = segment.cells.map((cell) => `<div class="ywtc-layout-cell">${yfmFragmentToHtml(cell, warnings, normalizations)}</div>`).join('');
|
|
420
|
+
return `<section class="ywtc-layout" data-ywtc-layout-source="yandex-wiki"><section class="ywtc-layout-section ywtc-layout-columns-${segment.cells.length} ywtc-layout-gap-${gap}" data-ywtc-layout-type="${segment.cells.length === 1 ? 'single' : segment.cells.length === 2 ? 'two_equal' : segment.cells.length === 3 ? 'three_equal' : 'unsupported'}">${cells}</section></section>`;
|
|
421
|
+
}).join('');
|
|
422
|
+
}
|
|
364
423
|
function metrics(html) {
|
|
365
424
|
const $ = load(html, null, false);
|
|
366
425
|
return { links: $('a').length, images: $('img').length, tables: $('table').length, codeBlocks: $('pre').length };
|
|
367
426
|
}
|
|
427
|
+
function normalizedNodesFromHtml(html, provider) {
|
|
428
|
+
const $ = load(`<main data-ywtc-nodes>${html}</main>`, null, false);
|
|
429
|
+
const nodes = [];
|
|
430
|
+
let pending = '';
|
|
431
|
+
const flush = () => {
|
|
432
|
+
if (!pending)
|
|
433
|
+
return;
|
|
434
|
+
nodes.push({ type: 'html', value: pending });
|
|
435
|
+
pending = '';
|
|
436
|
+
};
|
|
437
|
+
for (const element of $('main[data-ywtc-nodes]').contents().toArray()) {
|
|
438
|
+
const item = $(element);
|
|
439
|
+
if (!item.is('section.ywtc-layout')) {
|
|
440
|
+
pending += $.html(element);
|
|
441
|
+
continue;
|
|
442
|
+
}
|
|
443
|
+
flush();
|
|
444
|
+
const sections = item.children('section.ywtc-layout-section').toArray().map((section) => {
|
|
445
|
+
const sectionItem = $(section);
|
|
446
|
+
const rawType = sectionItem.attr('data-ywtc-layout-type') ?? 'unsupported';
|
|
447
|
+
const sectionType = confluenceLayoutCellCounts.has(rawType) || rawType === 'unsupported'
|
|
448
|
+
? rawType
|
|
449
|
+
: 'unsupported';
|
|
450
|
+
const gap = String(sectionItem.attr('class') ?? '').split(/\s+/).find((value) => value.startsWith('ywtc-layout-gap-'))?.slice('ywtc-layout-gap-'.length);
|
|
451
|
+
const cells = sectionItem.children('.ywtc-layout-cell').toArray().map((cell) => ({
|
|
452
|
+
type: 'layout-cell',
|
|
453
|
+
children: [{ type: 'html', value: $(cell).html() ?? '' }],
|
|
454
|
+
}));
|
|
455
|
+
return { type: 'layout-section', layout: { source: provider, sectionType, ...(gap ? { gap } : {}) }, children: cells };
|
|
456
|
+
});
|
|
457
|
+
nodes.push({ type: 'layout', layout: { source: provider }, children: sections });
|
|
458
|
+
}
|
|
459
|
+
flush();
|
|
460
|
+
return nodes.length > 0 ? nodes : [{ type: 'html', value: '<p></p>' }];
|
|
461
|
+
}
|
|
462
|
+
function semanticallyEmptyHtml(html) {
|
|
463
|
+
const $ = load(html, null, false);
|
|
464
|
+
$('script,style').remove();
|
|
465
|
+
if (semanticText($.root().text()))
|
|
466
|
+
return false;
|
|
467
|
+
return $('a[href],a[name],[id],img,video,audio,table,pre,code,hr,blockquote,ul,ol,.ywtc-note,.ywtc-expand,.ywtc-children,.ywtc-task-list,.ywtc-fallback').length === 0;
|
|
468
|
+
}
|
|
368
469
|
export function normalizePageContent(page) {
|
|
369
470
|
const warnings = [];
|
|
370
471
|
const normalizations = [];
|
|
@@ -378,7 +479,7 @@ export function normalizePageContent(page) {
|
|
|
378
479
|
label: reference.pageTitle ?? reference.href ?? reference.raw,
|
|
379
480
|
...(reference.anchor ? { anchor: reference.anchor } : {}),
|
|
380
481
|
}));
|
|
381
|
-
const children =
|
|
482
|
+
const children = normalizedNodesFromHtml(html, page.provider);
|
|
382
483
|
children.push(...tree.directives.map((directive) => ({
|
|
383
484
|
type: 'tree-directive',
|
|
384
485
|
value: directive.raw,
|
|
@@ -398,6 +499,7 @@ export function normalizePageContent(page) {
|
|
|
398
499
|
warnings,
|
|
399
500
|
normalizations,
|
|
400
501
|
degraded: warnings.length > 0,
|
|
502
|
+
semanticallyEmpty: tree.directives.length === 0 && semanticallyEmptyHtml(html),
|
|
401
503
|
metrics: metrics(html),
|
|
402
504
|
};
|
|
403
505
|
}
|
|
@@ -429,10 +531,13 @@ export function appendGridSnapshots(document, resources) {
|
|
|
429
531
|
const htmlNode = document.root.children?.find((node) => node.type === 'html');
|
|
430
532
|
if (htmlNode)
|
|
431
533
|
htmlNode.value = `${htmlNode.value ?? ''}${html}`;
|
|
534
|
+
else
|
|
535
|
+
document.root.children?.push({ type: 'html', value: html });
|
|
432
536
|
return {
|
|
433
537
|
...document,
|
|
434
538
|
warnings: [...document.warnings, ...grids.map((resource) => `Dynamic table "${resource.name}" was converted to a static table`)],
|
|
435
539
|
degraded: true,
|
|
540
|
+
semanticallyEmpty: false,
|
|
436
541
|
metrics: { ...document.metrics, tables: document.metrics.tables + grids.length },
|
|
437
542
|
};
|
|
438
543
|
}
|
|
@@ -445,20 +550,138 @@ async function htmlToMarkdown(html) {
|
|
|
445
550
|
.process(html);
|
|
446
551
|
return `${String(result).trimEnd()}\n`;
|
|
447
552
|
}
|
|
553
|
+
function linearizeLayoutsForMarkdown(html) {
|
|
554
|
+
const $ = load(html, null, false);
|
|
555
|
+
let layouts = 0;
|
|
556
|
+
$('section.ywtc-layout').toArray().reverse().forEach((layout) => {
|
|
557
|
+
const sourceSections = $(layout).children('section.ywtc-layout-section').toArray();
|
|
558
|
+
if (sourceSections.some((section) => $(section).children('.ywtc-layout-cell').length > 1))
|
|
559
|
+
layouts += 1;
|
|
560
|
+
const sections = sourceSections.map((section) => {
|
|
561
|
+
const cells = $(section).children('.ywtc-layout-cell').toArray();
|
|
562
|
+
if (cells.length <= 1)
|
|
563
|
+
return cells.map((cell) => $(cell).html() ?? '').join('');
|
|
564
|
+
return cells.map((cell, index) => `<section class="ywtc-layout-linear-cell"><h3>Column ${index + 1}</h3>${$(cell).html() ?? ''}</section>`).join('');
|
|
565
|
+
}).join('');
|
|
566
|
+
$(layout).replaceWith(sections);
|
|
567
|
+
});
|
|
568
|
+
return { html: $.html(), layouts };
|
|
569
|
+
}
|
|
570
|
+
async function semanticHtmlToYfm(html) {
|
|
571
|
+
const warnings = new Set();
|
|
572
|
+
let tokenSequence = 0;
|
|
573
|
+
const convert = async (fragment) => {
|
|
574
|
+
const $ = load(fragment, null, false);
|
|
575
|
+
const tokens = new Map();
|
|
576
|
+
const selector = 'section.ywtc-layout,details.ywtc-expand,aside.ywtc-note,nav.ywtc-children,ul.ywtc-task-list,span.ywtc-anchor';
|
|
577
|
+
const elements = $(selector).filter((_index, element) => $(element).parents(selector).length === 0).toArray().reverse();
|
|
578
|
+
for (const element of elements) {
|
|
579
|
+
const item = $(element);
|
|
580
|
+
let rendered = '';
|
|
581
|
+
if (item.is('section.ywtc-layout')) {
|
|
582
|
+
const sections = [];
|
|
583
|
+
for (const section of item.children('section.ywtc-layout-section').toArray()) {
|
|
584
|
+
const sectionItem = $(section);
|
|
585
|
+
const sectionType = sectionItem.attr('data-ywtc-layout-type') ?? 'unsupported';
|
|
586
|
+
if (['two_left_sidebar', 'two_right_sidebar', 'three_with_sidebars', 'unsupported'].includes(sectionType)) {
|
|
587
|
+
warnings.add(`Confluence ${sectionType} layout widths were converted to equal YFM columns`);
|
|
588
|
+
}
|
|
589
|
+
const blocks = [];
|
|
590
|
+
for (const cell of sectionItem.children('.ywtc-layout-cell').toArray()) {
|
|
591
|
+
const body = (await convert($(cell).html() ?? '')).trim();
|
|
592
|
+
blocks.push(`{% block %}\n\n${body}\n\n{% endblock %}`);
|
|
593
|
+
}
|
|
594
|
+
sections.push(`{% layout %}\n\n${blocks.join('\n\n')}\n\n{% endlayout %}`);
|
|
595
|
+
}
|
|
596
|
+
rendered = sections.join('\n\n');
|
|
597
|
+
}
|
|
598
|
+
else if (item.is('details.ywtc-expand')) {
|
|
599
|
+
const title = item.attr('data-ywtc-title') ?? (item.children('summary').first().text().trim() || 'Details');
|
|
600
|
+
const clone = item.clone();
|
|
601
|
+
clone.children('summary').first().remove();
|
|
602
|
+
rendered = `{% cut "${title.replaceAll('"', '\\"')}" %}\n\n${(await convert(clone.html() ?? '')).trim()}\n\n{% endcut %}`;
|
|
603
|
+
}
|
|
604
|
+
else if (item.is('aside.ywtc-note')) {
|
|
605
|
+
const rawType = item.attr('data-ywtc-note-type') ?? 'info';
|
|
606
|
+
const type = rawType === 'note' ? 'info' : rawType;
|
|
607
|
+
const title = item.attr('data-ywtc-title')?.trim();
|
|
608
|
+
rendered = `{% note ${type}${title ? ` "${title.replaceAll('"', '\\"')}"` : ''} %}\n\n${(await convert(item.html() ?? '')).trim()}\n\n{% endnote %}`;
|
|
609
|
+
}
|
|
610
|
+
else if (item.is('nav.ywtc-children')) {
|
|
611
|
+
const depth = item.attr('data-ywtc-depth');
|
|
612
|
+
const sort = item.attr('data-ywtc-sort');
|
|
613
|
+
const reverse = item.attr('data-ywtc-reverse') === 'true';
|
|
614
|
+
const attributes = [
|
|
615
|
+
depth ? `depth="${depth}"` : '',
|
|
616
|
+
sort === 'title' ? 'sort_by="title"' : '',
|
|
617
|
+
sort === 'title' ? `sort="${reverse ? 'desc' : 'asc'}"` : '',
|
|
618
|
+
].filter(Boolean).join(' ');
|
|
619
|
+
rendered = `{% tree${attributes ? ` ${attributes}` : ''} %}`;
|
|
620
|
+
}
|
|
621
|
+
else if (item.is('ul.ywtc-task-list')) {
|
|
622
|
+
const tasks = [];
|
|
623
|
+
for (const task of item.children('li.ywtc-task').toArray()) {
|
|
624
|
+
const marker = $(task).attr('data-ywtc-task-status') === 'complete' ? 'x' : ' ';
|
|
625
|
+
const body = (await convert($(task).html() ?? '')).trim().replace(/\n/g, '\n ');
|
|
626
|
+
tasks.push(`- [${marker}] ${body}`);
|
|
627
|
+
}
|
|
628
|
+
rendered = tasks.join('\n');
|
|
629
|
+
}
|
|
630
|
+
else if (item.is('span.ywtc-anchor')) {
|
|
631
|
+
const id = item.attr('id');
|
|
632
|
+
rendered = id ? `<a id="${escapeHtml(id)}"></a>` : '';
|
|
633
|
+
}
|
|
634
|
+
const token = `YWTCBLOCKTOKEN${tokenSequence++}END`;
|
|
635
|
+
tokens.set(token, rendered);
|
|
636
|
+
item.replaceWith(`<p>${token}</p>`);
|
|
637
|
+
}
|
|
638
|
+
let markdown = await htmlToMarkdown($.html());
|
|
639
|
+
for (const [token, rendered] of tokens)
|
|
640
|
+
markdown = markdown.replaceAll(token, rendered);
|
|
641
|
+
return markdown.trim();
|
|
642
|
+
};
|
|
643
|
+
return { body: `${await convert(html)}\n`, warnings: [...warnings] };
|
|
644
|
+
}
|
|
448
645
|
async function sanitizeHtml(html) {
|
|
646
|
+
const parsed = load(html, null, false);
|
|
647
|
+
parsed('[class]').each((_index, element) => {
|
|
648
|
+
const safeClasses = String(parsed(element).attr('class') ?? '').split(/\s+/).filter((value) => /^ywtc-[\w-]+$/.test(value));
|
|
649
|
+
if (safeClasses.length > 0)
|
|
650
|
+
parsed(element).attr('class', safeClasses.join(' '));
|
|
651
|
+
else
|
|
652
|
+
parsed(element).removeAttr('class');
|
|
653
|
+
});
|
|
449
654
|
const schema = {
|
|
450
655
|
...defaultSchema,
|
|
451
656
|
tagNames: [...(defaultSchema.tagNames ?? []), 'aside', 'details', 'summary'],
|
|
452
657
|
attributes: {
|
|
453
658
|
...defaultSchema.attributes,
|
|
454
|
-
'*': [...(defaultSchema.attributes?.['*'] ?? []),
|
|
659
|
+
'*': [...(defaultSchema.attributes?.['*'] ?? []), 'className'],
|
|
660
|
+
section: [...(defaultSchema.attributes?.section ?? []).filter((attribute) => !(Array.isArray(attribute) && attribute[0] === 'className')), 'className'],
|
|
455
661
|
},
|
|
456
662
|
};
|
|
457
|
-
const result = await unified().use(rehypeParse, { fragment: true }).use(rehypeSanitize, schema).use(rehypeStringify).process(html);
|
|
663
|
+
const result = await unified().use(rehypeParse, { fragment: true }).use(rehypeSanitize, schema).use(rehypeStringify).process(parsed.html());
|
|
458
664
|
return String(result);
|
|
459
665
|
}
|
|
666
|
+
function normalizedNodeHtml(node) {
|
|
667
|
+
if (node.type === 'html')
|
|
668
|
+
return node.value ?? '';
|
|
669
|
+
if (node.type === 'layout-cell')
|
|
670
|
+
return `<div class="ywtc-layout-cell">${(node.children ?? []).map(normalizedNodeHtml).join('')}</div>`;
|
|
671
|
+
if (node.type === 'layout-section') {
|
|
672
|
+
const cells = node.children ?? [];
|
|
673
|
+
const type = node.layout?.sectionType ?? 'unsupported';
|
|
674
|
+
const gap = node.layout?.gap ? ` ywtc-layout-gap-${node.layout.gap.replace(/[^a-z0-9_-]/gi, '')}` : '';
|
|
675
|
+
return `<section class="ywtc-layout-section ywtc-layout-${type} ywtc-layout-columns-${cells.length}${gap}" data-ywtc-layout-type="${type}">${cells.map(normalizedNodeHtml).join('')}</section>`;
|
|
676
|
+
}
|
|
677
|
+
if (node.type === 'layout') {
|
|
678
|
+
return `<section class="ywtc-layout" data-ywtc-layout-source="${node.layout?.source ?? 'yandex-wiki'}">${(node.children ?? []).map(normalizedNodeHtml).join('')}</section>`;
|
|
679
|
+
}
|
|
680
|
+
return '';
|
|
681
|
+
}
|
|
460
682
|
function htmlFromDocument(document) {
|
|
461
|
-
|
|
683
|
+
const html = document.root.children?.map(normalizedNodeHtml).join('') ?? '';
|
|
684
|
+
return html || '<p></p>';
|
|
462
685
|
}
|
|
463
686
|
export function normalizedPreviewHtml(document) {
|
|
464
687
|
return htmlFromDocument(document);
|
|
@@ -526,7 +749,7 @@ function semanticContent(body, provider) {
|
|
|
526
749
|
$('script,style').remove();
|
|
527
750
|
$('h1,h2,h3,h4,h5,h6,p,div,span').each((_index, element) => {
|
|
528
751
|
const node = $(element);
|
|
529
|
-
if (!semanticText(node.text()) && node.find('img,table,pre,code,a[href]').length === 0)
|
|
752
|
+
if (!semanticText(node.text()) && node.find('img,table,pre,code,a[href],[id],a[name]').length === 0 && !node.is('[id],a[name]'))
|
|
530
753
|
node.remove();
|
|
531
754
|
});
|
|
532
755
|
const textValues = (selector) => $(selector).toArray().map((element) => semanticText($(element).text()));
|
|
@@ -540,6 +763,24 @@ function semanticContent(body, provider) {
|
|
|
540
763
|
macroNames.push(`${name}[${parameters.join(',')}]`);
|
|
541
764
|
});
|
|
542
765
|
}
|
|
766
|
+
const layouts = $('.ywtc-layout').toArray().flatMap((layout, layoutIndex) => $(layout).children('.ywtc-layout-section').toArray().flatMap((section, sectionIndex) => {
|
|
767
|
+
const item = $(section);
|
|
768
|
+
const cells = item.children('.ywtc-layout-cell').toArray().map((cell) => {
|
|
769
|
+
const value = $(cell);
|
|
770
|
+
const signature = [
|
|
771
|
+
semanticText(value.text()),
|
|
772
|
+
`links=${value.find('a[href]').length}`,
|
|
773
|
+
`images=${value.find('img').length}`,
|
|
774
|
+
`tables=${value.find('table').length}`,
|
|
775
|
+
`code=${value.find('pre,code').length}`,
|
|
776
|
+
`anchors=${value.find('[id],a[name]').length}`,
|
|
777
|
+
].join('|');
|
|
778
|
+
return signature === '|links=0|images=0|tables=0|code=0|anchors=0' ? '' : signature;
|
|
779
|
+
});
|
|
780
|
+
if (cells.every((cell) => !cell))
|
|
781
|
+
return [];
|
|
782
|
+
return [`${layoutIndex}:${sectionIndex}:${item.attr('data-ywtc-layout-type') ?? 'unsupported'}:${cells.join('::')}`];
|
|
783
|
+
}));
|
|
543
784
|
return {
|
|
544
785
|
text: semanticText($.root().text()),
|
|
545
786
|
headings: $('h1,h2,h3,h4,h5,h6').toArray().map((element) => `${element.tagName}:${semanticText($(element).text())}`),
|
|
@@ -558,21 +799,42 @@ function semanticContent(body, provider) {
|
|
|
558
799
|
codeBlocks: $('pre').length,
|
|
559
800
|
},
|
|
560
801
|
macros: macroNames.sort(),
|
|
802
|
+
layouts,
|
|
561
803
|
};
|
|
562
804
|
}
|
|
805
|
+
function semanticContentIsEmpty(content) {
|
|
806
|
+
return !content.text
|
|
807
|
+
&& content.headings.length === 0
|
|
808
|
+
&& content.listItems.length === 0
|
|
809
|
+
&& content.tableCells.length === 0
|
|
810
|
+
&& content.code.length === 0
|
|
811
|
+
&& content.links.length === 0
|
|
812
|
+
&& content.images.length === 0
|
|
813
|
+
&& content.anchors.length === 0
|
|
814
|
+
&& content.macros.length === 0
|
|
815
|
+
&& content.layouts.length === 0
|
|
816
|
+
&& Object.values(content.structure).every((count) => count === 0);
|
|
817
|
+
}
|
|
563
818
|
export function semanticContentFingerprint(body, provider) {
|
|
564
819
|
return sha256(JSON.stringify(semanticContent(body, provider)));
|
|
565
820
|
}
|
|
566
821
|
export function verifyDestinationContent(expectedBody, actualBody, provider, page) {
|
|
567
822
|
if (actualBody === undefined)
|
|
568
823
|
return { status: 'failed', differences: [`Verification could not read destination content for ${page.title}`] };
|
|
569
|
-
if (actualBody.trim().length === 0)
|
|
570
|
-
return { status: 'failed', differences: [`Verification found empty destination content for ${page.title}`] };
|
|
571
824
|
if (isLegacyTransferPlaceholder(actualBody, provider, page)) {
|
|
572
825
|
return { status: 'failed', differences: [`Verification found a transfer placeholder instead of source content for ${page.title}`] };
|
|
573
826
|
}
|
|
574
827
|
const expected = semanticContent(expectedBody, provider);
|
|
575
828
|
const actual = semanticContent(actualBody, provider);
|
|
829
|
+
const expectedEmpty = semanticContentIsEmpty(expected);
|
|
830
|
+
const actualEmpty = semanticContentIsEmpty(actual);
|
|
831
|
+
if (expectedEmpty && actualEmpty)
|
|
832
|
+
return { status: 'complete', differences: [] };
|
|
833
|
+
if (!expectedEmpty && actualEmpty)
|
|
834
|
+
return { status: 'failed', differences: [`Verification found empty destination content for ${page.title}`] };
|
|
835
|
+
if (expectedEmpty && !actualEmpty) {
|
|
836
|
+
return { status: 'failed', differences: [`Verification found unexpected destination content for semantically empty source page ${page.title}`] };
|
|
837
|
+
}
|
|
576
838
|
const differs = (key) => JSON.stringify(expected[key]) !== JSON.stringify(actual[key]);
|
|
577
839
|
const missingData = ['text', 'tableCells', 'code', 'links', 'images', 'anchors', 'macros'].filter(differs);
|
|
578
840
|
if (missingData.length > 0) {
|
|
@@ -581,7 +843,7 @@ export function verifyDestinationContent(expectedBody, actualBody, provider, pag
|
|
|
581
843
|
differences: [`Verification found destination content that differs from the rendered source for ${page.title}: ${missingData.join(', ')}`],
|
|
582
844
|
};
|
|
583
845
|
}
|
|
584
|
-
const simplifiedFormatting = ['headings', 'listItems', 'imageLabels', 'structure'].filter(differs);
|
|
846
|
+
const simplifiedFormatting = ['headings', 'listItems', 'imageLabels', 'structure', 'layouts'].filter(differs);
|
|
585
847
|
if (simplifiedFormatting.length > 0) {
|
|
586
848
|
return {
|
|
587
849
|
status: 'partial',
|
|
@@ -681,20 +943,40 @@ function renderTreeHtml(context, directive) {
|
|
|
681
943
|
return tree ? `<nav class="ywtc-tree" aria-label="Child pages">${tree}</nav>` : '';
|
|
682
944
|
}
|
|
683
945
|
export async function renderMarkdown(document, context) {
|
|
684
|
-
|
|
946
|
+
if (document.semanticallyEmpty) {
|
|
947
|
+
return { body: '', warnings: [...document.warnings], normalizations: [...document.normalizations], degraded: document.degraded, representedReferences: document.references.length };
|
|
948
|
+
}
|
|
949
|
+
const portable = rewritePortableHtml(htmlFromDocument(document), context);
|
|
950
|
+
const linearized = linearizeLayoutsForMarkdown(portable);
|
|
685
951
|
const tree = treeDirectiveNodes(document).map((node) => renderTreeMarkdown(context, node.tree ?? {})).filter(Boolean).join('\n\n');
|
|
686
|
-
const content = await htmlToMarkdown(html);
|
|
952
|
+
const content = await htmlToMarkdown(linearized.html);
|
|
687
953
|
const body = tree ? `${content.trimEnd()}\n\n${tree}\n` : content;
|
|
688
|
-
|
|
954
|
+
const layoutWarnings = linearized.layouts > 0 ? [`${linearized.layouts} column layout${linearized.layouts === 1 ? ' was' : 's were'} linearized for portable Markdown`] : [];
|
|
955
|
+
return { body, warnings: [...document.warnings, ...layoutWarnings], normalizations: [...(document.normalizations ?? [])], degraded: document.degraded || layoutWarnings.length > 0, representedReferences: document.references.length };
|
|
689
956
|
}
|
|
690
957
|
export async function renderStaticHtml(document, context) {
|
|
958
|
+
if (document.semanticallyEmpty) {
|
|
959
|
+
return { body: '', warnings: [...document.warnings], normalizations: [...document.normalizations], degraded: document.degraded, representedReferences: document.references.length };
|
|
960
|
+
}
|
|
691
961
|
const rewritten = rewritePortableHtml(htmlFromDocument(document), context);
|
|
692
962
|
const content = await sanitizeHtml(rewritten);
|
|
693
963
|
const tree = treeDirectiveNodes(document).map((node) => renderTreeHtml(context, node.tree ?? {})).filter(Boolean).join('');
|
|
694
964
|
const body = `${content}${tree}`;
|
|
695
|
-
|
|
965
|
+
const unsupportedLayouts = [];
|
|
966
|
+
const visit = (node) => {
|
|
967
|
+
if (node.type === 'layout-section' && node.layout?.sectionType === 'unsupported')
|
|
968
|
+
unsupportedLayouts.push(node.children?.length ?? 0);
|
|
969
|
+
for (const child of node.children ?? [])
|
|
970
|
+
visit(child);
|
|
971
|
+
};
|
|
972
|
+
visit(document.root);
|
|
973
|
+
const layoutWarnings = unsupportedLayouts.map((cells) => `Unsupported ${cells}-cell layout was stacked in static HTML without losing cell content`);
|
|
974
|
+
return { body, warnings: [...document.warnings, ...layoutWarnings], normalizations: [...(document.normalizations ?? [])], degraded: document.degraded || layoutWarnings.length > 0, representedReferences: document.references.length };
|
|
696
975
|
}
|
|
697
976
|
export async function renderYfm(document, context) {
|
|
977
|
+
if (document.semanticallyEmpty) {
|
|
978
|
+
return { body: '', warnings: [...document.warnings], normalizations: [...document.normalizations], degraded: document.degraded, representedReferences: document.references.length };
|
|
979
|
+
}
|
|
698
980
|
if (document.rawFormat === 'yfm') {
|
|
699
981
|
let body = document.raw;
|
|
700
982
|
const resourceWarnings = new Set();
|
|
@@ -739,10 +1021,19 @@ export async function renderYfm(document, context) {
|
|
|
739
1021
|
representedReferences: document.references.length,
|
|
740
1022
|
};
|
|
741
1023
|
}
|
|
742
|
-
const
|
|
743
|
-
return {
|
|
1024
|
+
const converted = await semanticHtmlToYfm(rewritePortableHtml(htmlFromDocument(document), context));
|
|
1025
|
+
return {
|
|
1026
|
+
body: converted.body,
|
|
1027
|
+
warnings: [...document.warnings, ...converted.warnings],
|
|
1028
|
+
normalizations: [...(document.normalizations ?? []), 'Converted supported Confluence Storage Format constructs to native YFM equivalents'],
|
|
1029
|
+
degraded: document.degraded || converted.warnings.length > 0,
|
|
1030
|
+
representedReferences: document.references.length,
|
|
1031
|
+
};
|
|
744
1032
|
}
|
|
745
1033
|
export async function renderConfluenceStorage(document, page, context, spaceKey) {
|
|
1034
|
+
if (document.semanticallyEmpty) {
|
|
1035
|
+
return { body: '<p></p>', warnings: [...document.warnings], normalizations: [...document.normalizations], degraded: document.degraded, representedReferences: document.references.length };
|
|
1036
|
+
}
|
|
746
1037
|
if (document.rawFormat === 'confluence-storage') {
|
|
747
1038
|
const $ = load(`<main data-root>${document.raw}</main>`, { xmlMode: true }, false);
|
|
748
1039
|
const warnings = [...document.warnings];
|