@sillsdev/docu-notion 0.14.0 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +39 -22
- package/dist/MakeImagePersistencePlan.d.ts +2 -1
- package/dist/MakeImagePersistencePlan.js +68 -23
- package/dist/{notion-styles.css → docu-notion-styles.css} +60 -58
- package/dist/images.d.ts +3 -4
- package/dist/images.js +17 -14
- package/dist/latex.spec.d.ts +1 -0
- package/dist/latex.spec.js +101 -0
- package/dist/makeImagePersistencePlan.spec.js +66 -37
- package/dist/plugins/pluginTestRun.d.ts +1 -1
- package/dist/plugins/pluginTestRun.js +8 -5
- package/dist/plugins/pluginTypes.d.ts +6 -2
- package/dist/pull.d.ts +4 -0
- package/dist/pull.js +27 -5
- package/dist/run.d.ts +1 -1
- package/dist/run.js +80 -23
- package/package.json +9 -8
package/README.md
CHANGED
|
@@ -6,10 +6,12 @@ Example Site: https://sillsdev.github.io/docu-notion-sample-site/
|
|
|
6
6
|
|
|
7
7
|
# Instructions
|
|
8
8
|
|
|
9
|
-
## 1. Set up your documentation site
|
|
9
|
+
## 1. Set up your documentation site
|
|
10
10
|
|
|
11
11
|
First, prepare your markdown-based static file system like [Docusaurus](https://docusaurus.io/). For a shortcut with github actions, search, and deployment to github pages, you can just copy [this template](https://github.com/sillsdev/docu-notion-sample-site).
|
|
12
12
|
|
|
13
|
+
If you do not use the above sample, you will need to manually tell your `docusaurus.config.js` about `docu-notion-styles.css`. See [Styling and Layout](https://docusaurus.io/docs/styling-layout). This stylesheet enables various Notion things to look right, for example multi-column layouts. By default, docu-notion will copy this file to the `css/` directory. There is an option to change that location if you want.
|
|
14
|
+
|
|
13
15
|
## 2. In Notion, duplicate the docu-notion template
|
|
14
16
|
|
|
15
17
|
Go to [this template page](https://hattonjohn.notion.site/Documentation-Template-Docusaurus-0e998b32da3c47edad0f62a25b49818c). Duplicate it into your own workspace.
|
|
@@ -17,25 +19,25 @@ You can name it anything you like, e.g. "Documentation Root".
|
|
|
17
19
|
|
|
18
20
|
## 3. Create a Notion Integration
|
|
19
21
|
|
|
20
|
-
In order for docu-notion to read your site via Notion's API, you need to create what Notion calls an "integration". Follow [these instructions](https://developers.notion.com/docs/getting-started) to make an integration and get your token.
|
|
22
|
+
In order for docu-notion to read your site via Notion's API, you need to create what Notion calls an "integration". Follow [these instructions](https://developers.notion.com/docs/getting-started) to make an integration and get your token. Remember to limit your integration to "READ" access.
|
|
21
23
|
|
|
22
|
-
## 4.
|
|
24
|
+
## 4. Connect your Integration
|
|
23
25
|
|
|
24
|
-
|
|
26
|
+
Go to the page that will be the root of your site. This page should have, as direct children, your "Outline" (required) and "Database" (optional) pages. Follow [these instructions](https://developers.notion.com/docs/create-a-notion-integration#give-your-integration-page-permissions).
|
|
25
27
|
|
|
26
|
-
|
|
28
|
+
<img width="318" alt="image" src="https://github.com/sillsdev/docu-notion/assets/8448/810c6dca-f9ab-4370-93b4-dc1479332af7">
|
|
27
29
|
|
|
28
|
-
## 5. Add your pages under your Outline page
|
|
30
|
+
## 5. Add your pages under your Outline page
|
|
29
31
|
|
|
30
32
|
Currently, docu-notion expects that each page has only one of the following: sub-pages, links to other pages, or normal content. Do not mix them. You can add content pages directly here, but then you won't be able to make use of the workflow features. If those matter to you, instead make new pages under the "Database" and then link to them in your outline pages.
|
|
31
33
|
|
|
32
34
|
## 6. Pull your pages
|
|
33
35
|
|
|
34
|
-
First, determine the
|
|
35
|
-
https://www.notion.so/hattonjohn/My-Docs-
|
|
36
|
-
means that the
|
|
36
|
+
First, determine the ID of your root page by clicking "Share" and looking at the url it gives you. E.g.
|
|
37
|
+
`https://www.notion.so/hattonjohn/My-Docs-0456aa5842946PRETEND4f37c97a0e5`
|
|
38
|
+
means that the ID is `0456aa5842946PRETEND4f37c97a0e5`.
|
|
37
39
|
|
|
38
|
-
|
|
40
|
+
Try it out:
|
|
39
41
|
|
|
40
42
|
```
|
|
41
43
|
npx @sillsdev/docu-notion -n secret_PRETEND123456789PRETEND123456789PRETEND6789 -r 0456aa5842946PRETEND4f37c97a0e5"
|
|
@@ -88,7 +90,9 @@ One of the big attractions of Notion for large documentation projects is that yo
|
|
|
88
90
|
|
|
89
91
|
## Slugs
|
|
90
92
|
|
|
91
|
-
By default, pages will be given a slug based on the Notion
|
|
93
|
+
By default, pages will be given a slug based on the Notion ID. For a human-readable URL, add a notion property named `Slug` to your database pages and enter a value in there that will work well in a URL. That is, no spaces, ?, #, /, etc.
|
|
94
|
+
|
|
95
|
+
See `Options` to require slugs in Notion.
|
|
92
96
|
|
|
93
97
|
## Known Limitations
|
|
94
98
|
|
|
@@ -110,25 +114,27 @@ NOTE: if you just localize an image, it will not get picked up. You also must lo
|
|
|
110
114
|
|
|
111
115
|
# Automated builds with Github Actions
|
|
112
116
|
|
|
113
|
-
Here is a working Github Action script to copy and customize
|
|
117
|
+
Here is a [working Github Action script to copy and customize](https://github.com/BloomBooks/bloom-docs/blob/master/.github/workflows/release.yml).
|
|
114
118
|
|
|
115
119
|
# Command line
|
|
116
120
|
|
|
117
|
-
Usage: docu-notion -n <token> -r <root> [options]
|
|
121
|
+
Usage: `docu-notion -n <token> -r <root> [options]`
|
|
118
122
|
|
|
119
123
|
Options:
|
|
120
124
|
|
|
121
125
|
| flag | required? | description |
|
|
122
126
|
| ------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
123
|
-
|
|
|
124
|
-
|
|
|
125
|
-
|
|
|
126
|
-
|
|
|
127
|
-
|
|
|
128
|
-
|
|
|
129
|
-
|
|
|
130
|
-
|
|
|
131
|
-
| -
|
|
127
|
+
| `-n, --notion-token <string>` | required | notion api token, which looks like `secret_3bc1b50XFYb15123RHF243x43450XFY33250XFYa343` |
|
|
128
|
+
| `-r, --root-page <string>` | required | The 31 character ID of the page which is the root of your docs page in notion. The code will look like `9120ec9960244ead80fa2ef4bc1bba25`. This page must have a child page named 'Outline' |
|
|
129
|
+
| `-m, --markdown-output-path <string>` | | Root of the hierarchy for md files. WARNING: node-pull-mdx will delete files from this directory. Note also that if it finds localized images, it will create an i18n/ directory as a sibling. (default: `./docs`) |
|
|
130
|
+
| `-t, --status-tag <string>` | | Database pages without a Notion page property 'status' matching this will be ignored. Use '\*' to ignore status altogether. (default: `Publish`) |
|
|
131
|
+
| `--locales <codes>` | | Comma-separated list of iso 639-2 codes, the same list as in docusaurus.config.js, minus the primary (i.e. 'en'). This is needed for image localization. (default: `[]`) |
|
|
132
|
+
| `-l, --log-level <level>` | | Log level (choices: `info`, `verbose`, `debug`) |
|
|
133
|
+
| `-i, --img-output-path <string>` | | Path to directory where images will be stored. If this is not included, images will be placed in the same directory as the document that uses them, which then allows for localization of screenshots. |
|
|
134
|
+
| `-p, --img-prefix-in-markdown <string>` | | When referencing an image from markdown, prefix with this path instead of the full img-output-path. Should be used only in conjunction with --img-output-path. |
|
|
135
|
+
| `--require-slugs` | | If set, docu-notion will fail if any pages it would otherwise publish are missing a slug in Notion. |
|
|
136
|
+
| `--image-file-name-format <format>` | | choices:<ul><li>`default`: {page slug (if any)}.{image block ID}</li><li>`content-hash`: Use a hash of the image content.</li><li>`legacy`: Use the legacy (before v0.16) method of determining file names. Set this to maintain backward compatibility.</li></ul>All formats will use the original file extension. |
|
|
137
|
+
| `-h, --help` | | display help for command |
|
|
132
138
|
|
|
133
139
|
# Plugins
|
|
134
140
|
|
|
@@ -146,3 +152,14 @@ To map Notion callouts to Docusaurus admonitions, ensure the icon is for the typ
|
|
|
146
152
|
- 🔥➜ danger
|
|
147
153
|
|
|
148
154
|
The default admonition type, if no matching icon is found, is "note".
|
|
155
|
+
|
|
156
|
+
# Known Workarounds
|
|
157
|
+
|
|
158
|
+
### Start a numbered list at a number other than 1
|
|
159
|
+
|
|
160
|
+
In Notion, make sure the block is "Text," not "Numbered List".
|
|
161
|
+
|
|
162
|
+
- But make sure the number does NOT have a space in front of it. This can/will cause issues with sub-list items.
|
|
163
|
+
- One way to get Notion to let you do this:
|
|
164
|
+
- Create a numbered list item where the text duplicates the number you want. Convert that numbered list item to "Text."
|
|
165
|
+
- i.e. Type "1. 1. Item one." Notion makes the first "1." into a number in a list. When you convert back to "Text," you're left with plain text "1. Item one."
|
|
@@ -1,3 +1,4 @@
|
|
|
1
1
|
import { ImageSet } from "./images";
|
|
2
|
-
|
|
2
|
+
import { DocuNotionOptions } from "./pull";
|
|
3
|
+
export declare function makeImagePersistencePlan(options: DocuNotionOptions, imageSet: ImageSet, imageBlockId: string, imageOutputRootPath: string, imagePrefix: string): void;
|
|
3
4
|
export declare function hashOfString(s: string): number;
|
|
@@ -22,40 +22,81 @@ var __importStar = (this && this.__importStar) || function (mod) {
|
|
|
22
22
|
__setModuleDefault(result, mod);
|
|
23
23
|
return result;
|
|
24
24
|
};
|
|
25
|
+
var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
26
|
+
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
27
|
+
};
|
|
25
28
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
26
29
|
exports.hashOfString = exports.makeImagePersistencePlan = void 0;
|
|
27
30
|
const Path = __importStar(require("path"));
|
|
28
31
|
const log_1 = require("./log");
|
|
29
32
|
const process_1 = require("process");
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
//
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
const thingToHash = (_b = findLastUuid(urlBeforeQuery)) !== null && _b !== void 0 ? _b : urlBeforeQuery;
|
|
41
|
-
const hash = hashOfString(thingToHash);
|
|
42
|
-
imageSet.outputFileName = `${hash}.${imageSet.fileType.ext}`;
|
|
43
|
-
imageSet.primaryFileOutputPath = Path.posix.join((imageOutputRootPath === null || imageOutputRootPath === void 0 ? void 0 : imageOutputRootPath.length) > 0
|
|
44
|
-
? imageOutputRootPath
|
|
45
|
-
: imageSet.pathToParentDocument, imageSet.outputFileName);
|
|
46
|
-
if (imageOutputRootPath && imageSet.localizedUrls.length) {
|
|
47
|
-
(0, log_1.error)("imageOutputPath was declared, but one or more localizedUrls were found too. If you are going to localize screenshots, then you can't declare an imageOutputPath.");
|
|
33
|
+
const crypto_1 = __importDefault(require("crypto"));
|
|
34
|
+
function makeImagePersistencePlan(options, imageSet, imageBlockId, imageOutputRootPath, imagePrefix) {
|
|
35
|
+
var _a, _b, _c;
|
|
36
|
+
const urlBeforeQuery = imageSet.primaryUrl.split("?")[0];
|
|
37
|
+
let imageFileExtension = (_a = imageSet.fileType) === null || _a === void 0 ? void 0 : _a.ext;
|
|
38
|
+
if (!imageFileExtension) {
|
|
39
|
+
// Try to get the extension from the url
|
|
40
|
+
imageFileExtension = urlBeforeQuery.split(".").pop();
|
|
41
|
+
if (!imageFileExtension) {
|
|
42
|
+
(0, log_1.error)(`Something wrong with the filetype extension on the blob we got from ${imageSet.primaryUrl}`);
|
|
48
43
|
(0, process_1.exit)(1);
|
|
49
44
|
}
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
45
|
+
}
|
|
46
|
+
if (options.imageFileNameFormat === "legacy") {
|
|
47
|
+
// Original behavior and comment:
|
|
48
|
+
// Since most images come from pasting screenshots, there isn't normally a filename. That's fine, we just make a hash of the url
|
|
49
|
+
// Images that are stored by notion come to us with a complex url that changes over time, so we pick out the UUID that doesn't change. Example:
|
|
50
|
+
// https://s3.us-west-2.amazonaws.com/secure.notion-static.com/d1058f46-4d2f-4292-8388-4ad393383439/Untitled.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIAT73L2G45EIPT3X45%2F20220516%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20220516T233630Z&X-Amz-Expires=3600&X-Amz-Signature=f215704094fcc884d37073b0b108cf6d1c9da9b7d57a898da38bc30c30b4c4b5&X-Amz-SignedHeaders=host&x-id=GetObject
|
|
51
|
+
// But around Sept 2023, they changed the url to be something like:
|
|
52
|
+
// https://prod-files-secure.s3.us-west-2.amazonaws.com/d9a2b712-cf69-4bd6-9d65-87a4ceeacca2/d1bcdc8c-b065-4e40-9a11-392aabeb220e/Untitled.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIAT73L2G45EIPT3X45%2F20230915%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20230915T161258Z&X-Amz-Expires=3600&X-Amz-Signature=28fca48e65fba86d539c3c4b7676fce1fa0857aa194f7b33dd4a468ecca6ab24&X-Amz-SignedHeaders=host&x-id=GetObject
|
|
53
|
+
// The thing we want is the last UUID before the ?
|
|
54
|
+
const thingToHash = (_b = findLastUuid(urlBeforeQuery)) !== null && _b !== void 0 ? _b : urlBeforeQuery;
|
|
55
|
+
const hash = hashOfString(thingToHash);
|
|
56
|
+
imageSet.outputFileName = `${hash}.${imageFileExtension}`;
|
|
57
|
+
}
|
|
58
|
+
else if (options.imageFileNameFormat === "content-hash") {
|
|
59
|
+
// This was requested by a user: https://github.com/sillsdev/docu-notion/issues/76.
|
|
60
|
+
// We chose not to include it in the default file name because we want to maintain
|
|
61
|
+
// as much stability in the file name as feasible for an image localization workflow.
|
|
62
|
+
// However, particularly in a workflow which is not concerned with localization,
|
|
63
|
+
// this could be a good option. One benefit is that the image only needs to exist once
|
|
64
|
+
// in the file system regardless of how many times it is used in the site.
|
|
65
|
+
const imageHash = hashOfBufferContent(imageSet.primaryBuffer);
|
|
66
|
+
imageSet.outputFileName = `${imageHash}.${imageFileExtension}`;
|
|
54
67
|
}
|
|
55
68
|
else {
|
|
56
|
-
|
|
69
|
+
// We decided not to do this for the default format because it means
|
|
70
|
+
// instability for the file name in Crowdin, which causes loss of localizations.
|
|
71
|
+
// If we decide to include it in the future, we should add a unit test.
|
|
72
|
+
// const imageFileName = Path.basename(urlBeforeQuery);
|
|
73
|
+
// const imageFileNameWithoutExtension = Path.parse(imageFileName).name;
|
|
74
|
+
// const originalFileNamePart = ["untitled", "unnamed"].includes(
|
|
75
|
+
// imageFileNameWithoutExtension.toLocaleLowerCase()
|
|
76
|
+
// )
|
|
77
|
+
// ? ""
|
|
78
|
+
// : `${imageFileNameWithoutExtension.substring(0, 50)}.`;
|
|
79
|
+
// Format is page slug (if there is one) followed by the image block ID from Notion.
|
|
80
|
+
// The image block ID will remain stable as long as any changes to the image are done
|
|
81
|
+
// using the Replace feature. Also, image blocks can be moved using the Move To feature.
|
|
82
|
+
// We decided to include the page slug for easier workflow during localization, particularly in Crowdin.
|
|
83
|
+
// The block ID is a unique GUID and thus provides a unique file name.
|
|
84
|
+
const pageSlugPart = ((_c = imageSet.pageInfo) === null || _c === void 0 ? void 0 : _c.slug)
|
|
85
|
+
? `${imageSet.pageInfo.slug.replace(/^\//, "")}.`
|
|
86
|
+
: "";
|
|
87
|
+
imageSet.outputFileName = `${pageSlugPart}${imageBlockId}.${imageFileExtension}`;
|
|
88
|
+
}
|
|
89
|
+
imageSet.primaryFileOutputPath = Path.posix.join((imageOutputRootPath === null || imageOutputRootPath === void 0 ? void 0 : imageOutputRootPath.length) > 0
|
|
90
|
+
? imageOutputRootPath
|
|
91
|
+
: imageSet.pageInfo.directoryContainingMarkdown, decodeURI(imageSet.outputFileName));
|
|
92
|
+
if (imageOutputRootPath && imageSet.localizedUrls.length) {
|
|
93
|
+
(0, log_1.error)("imageOutputPath was declared, but one or more localizedUrls were found too. If you are going to localize screenshots, then you can't declare an imageOutputPath.");
|
|
57
94
|
(0, process_1.exit)(1);
|
|
58
95
|
}
|
|
96
|
+
imageSet.filePathToUseInMarkdown =
|
|
97
|
+
((imagePrefix === null || imagePrefix === void 0 ? void 0 : imagePrefix.length) > 0 ? imagePrefix : ".") +
|
|
98
|
+
"/" +
|
|
99
|
+
imageSet.outputFileName;
|
|
59
100
|
}
|
|
60
101
|
exports.makeImagePersistencePlan = makeImagePersistencePlan;
|
|
61
102
|
function findLastUuid(url) {
|
|
@@ -73,3 +114,7 @@ function hashOfString(s) {
|
|
|
73
114
|
return Math.abs(hash);
|
|
74
115
|
}
|
|
75
116
|
exports.hashOfString = hashOfString;
|
|
117
|
+
function hashOfBufferContent(buffer) {
|
|
118
|
+
const hash = crypto_1.default.createHash("sha256").update(buffer).digest("hex");
|
|
119
|
+
return hash.slice(0, 20);
|
|
120
|
+
}
|
|
@@ -1,58 +1,60 @@
|
|
|
1
|
-
/*
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
https://github1s.com/NotionX/react-notion-x/blob/master/packages/react-notion-x/src/styles.css#
|
|
9
|
-
|
|
10
|
-
.notion-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
margin-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
margin-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
1
|
+
/* This should be added to the docusaurus.config.js in order to show some notion things correctly.
|
|
2
|
+
See the option: --css-output-directory
|
|
3
|
+
See the docusaurus docs: https://docusaurus.io/docs/styling-layout
|
|
4
|
+
See the use in the docu-notion-sample-site: https://github.com/sillsdev/docu-notion-sample-site/blob/main/docusaurus.config.js
|
|
5
|
+
*/
|
|
6
|
+
|
|
7
|
+
/* Copied from
|
|
8
|
+
https://github1s.com/NotionX/react-notion-x/blob/master/packages/react-notion-x/src/styles.css#L934
|
|
9
|
+
and
|
|
10
|
+
https://github1s.com/NotionX/react-notion-x/blob/master/packages/react-notion-x/src/styles.css#L1063
|
|
11
|
+
*/
|
|
12
|
+
.notion-column {
|
|
13
|
+
display: flex;
|
|
14
|
+
flex-direction: column;
|
|
15
|
+
padding-top: 12px;
|
|
16
|
+
padding-bottom: 12px;
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
.notion-column > *:first-child {
|
|
20
|
+
margin-top: 0;
|
|
21
|
+
margin-left: 0;
|
|
22
|
+
margin-right: 0;
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
.notion-column > *:last-child {
|
|
26
|
+
margin-left: 0;
|
|
27
|
+
margin-right: 0;
|
|
28
|
+
margin-bottom: 0;
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
.notion-row {
|
|
32
|
+
display: flex;
|
|
33
|
+
overflow: hidden;
|
|
34
|
+
width: 100%;
|
|
35
|
+
max-width: 100%;
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
@media (max-width: 640px) {
|
|
39
|
+
.notion-row {
|
|
40
|
+
flex-direction: column;
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
.notion-row .notion-column {
|
|
44
|
+
width: 100% !important;
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
.notion-row .notion-spacer {
|
|
48
|
+
display: none;
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
.notion-spacer {
|
|
53
|
+
/* This matches the value in ColumnTransformer.ts */
|
|
54
|
+
width: calc(min(32px, 4vw));
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
.notion-spacer:last-child {
|
|
58
|
+
display: none;
|
|
59
|
+
}
|
|
60
|
+
/* End copied from NotionX */
|
package/dist/images.d.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
/// <reference types="node" />
|
|
2
2
|
import { FileTypeResult } from "file-type";
|
|
3
3
|
import { ListBlockChildrenResponseResult } from "notion-to-md/build/types";
|
|
4
|
-
import { IPlugin } from "./plugins/pluginTypes";
|
|
4
|
+
import { IDocuNotionContext, IDocuNotionContextPageInfo, IPlugin } from "./plugins/pluginTypes";
|
|
5
5
|
export type ImageSet = {
|
|
6
6
|
primaryUrl: string;
|
|
7
7
|
caption?: string;
|
|
@@ -9,8 +9,7 @@ export type ImageSet = {
|
|
|
9
9
|
iso632Code: string;
|
|
10
10
|
url: string;
|
|
11
11
|
}>;
|
|
12
|
-
|
|
13
|
-
relativePathToParentDocument?: string;
|
|
12
|
+
pageInfo?: IDocuNotionContextPageInfo;
|
|
14
13
|
primaryBuffer?: Buffer;
|
|
15
14
|
fileType?: FileTypeResult;
|
|
16
15
|
primaryFileOutputPath?: string;
|
|
@@ -19,6 +18,6 @@ export type ImageSet = {
|
|
|
19
18
|
};
|
|
20
19
|
export declare function initImageHandling(prefix: string, outputPath: string, incomingLocales: string[]): Promise<void>;
|
|
21
20
|
export declare const standardImageTransformer: IPlugin;
|
|
22
|
-
export declare function markdownToMDImageTransformer(block: ListBlockChildrenResponseResult,
|
|
21
|
+
export declare function markdownToMDImageTransformer(block: ListBlockChildrenResponseResult, context: IDocuNotionContext): Promise<string>;
|
|
23
22
|
export declare function parseImageBlock(image: any): ImageSet;
|
|
24
23
|
export declare function cleanupOldImages(): Promise<void>;
|
package/dist/images.js
CHANGED
|
@@ -38,11 +38,11 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
38
38
|
exports.cleanupOldImages = exports.parseImageBlock = exports.markdownToMDImageTransformer = exports.standardImageTransformer = exports.initImageHandling = void 0;
|
|
39
39
|
const fs = __importStar(require("fs-extra"));
|
|
40
40
|
const file_type_1 = __importDefault(require("file-type"));
|
|
41
|
-
const
|
|
41
|
+
const axios_1 = __importDefault(require("axios"));
|
|
42
42
|
const Path = __importStar(require("path"));
|
|
43
43
|
const MakeImagePersistencePlan_1 = require("./MakeImagePersistencePlan");
|
|
44
44
|
const log_1 = require("./log");
|
|
45
|
-
// We several things here:
|
|
45
|
+
// We handle several things here:
|
|
46
46
|
// 1) copy images locally instead of leaving them in Notion
|
|
47
47
|
// 2) change the links to point here
|
|
48
48
|
// 3) read the caption and if there are localized images, get those too
|
|
@@ -75,16 +75,16 @@ exports.standardImageTransformer = {
|
|
|
75
75
|
type: "image",
|
|
76
76
|
// we have to set this one up for each page because we need to
|
|
77
77
|
// give it two extra parameters that are context for each page
|
|
78
|
-
getStringFromBlock: (context, block) => markdownToMDImageTransformer(block, context
|
|
78
|
+
getStringFromBlock: (context, block) => markdownToMDImageTransformer(block, context),
|
|
79
79
|
},
|
|
80
80
|
],
|
|
81
81
|
};
|
|
82
82
|
// This is a "custom transformer" function passed to notion-to-markdown
|
|
83
83
|
// eslint-disable-next-line @typescript-eslint/require-await
|
|
84
|
-
function markdownToMDImageTransformer(block,
|
|
84
|
+
function markdownToMDImageTransformer(block, context) {
|
|
85
85
|
return __awaiter(this, void 0, void 0, function* () {
|
|
86
86
|
const image = block.image;
|
|
87
|
-
yield processImageBlock(
|
|
87
|
+
yield processImageBlock(block, context);
|
|
88
88
|
// just concatenate the caption text parts together
|
|
89
89
|
const altText = image.caption
|
|
90
90
|
// eslint-disable-next-line @typescript-eslint/no-unsafe-return
|
|
@@ -95,17 +95,17 @@ function markdownToMDImageTransformer(block, fullPathToDirectoryContainingMarkdo
|
|
|
95
95
|
});
|
|
96
96
|
}
|
|
97
97
|
exports.markdownToMDImageTransformer = markdownToMDImageTransformer;
|
|
98
|
-
function processImageBlock(
|
|
98
|
+
function processImageBlock(block, context) {
|
|
99
99
|
return __awaiter(this, void 0, void 0, function* () {
|
|
100
|
+
const imageBlock = block.image;
|
|
100
101
|
(0, log_1.logDebug)("processImageBlock", JSON.stringify(imageBlock));
|
|
101
102
|
const imageSet = parseImageBlock(imageBlock);
|
|
102
|
-
imageSet.
|
|
103
|
-
imageSet.relativePathToParentDocument = relativePathToThisPage;
|
|
103
|
+
imageSet.pageInfo = context.pageInfo;
|
|
104
104
|
// enhance: it would much better if we could split the changes to markdown separately from actual reading/writing,
|
|
105
105
|
// so that this wasn't part of the markdown-creation loop. It's already almost there; we just need to
|
|
106
106
|
// save the imageSets somewhere and then do the actual reading/writing later.
|
|
107
107
|
yield readPrimaryImage(imageSet);
|
|
108
|
-
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(imageSet, imageOutputPath, imagePrefix);
|
|
108
|
+
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(context.options, imageSet, block.id, imageOutputPath, imagePrefix);
|
|
109
109
|
yield saveImage(imageSet);
|
|
110
110
|
// change the src to point to our copy of the image
|
|
111
111
|
if ("file" in imageBlock) {
|
|
@@ -131,9 +131,12 @@ function processImageBlock(imageBlock, pathToParentDocument, relativePathToThisP
|
|
|
131
131
|
}
|
|
132
132
|
function readPrimaryImage(imageSet) {
|
|
133
133
|
return __awaiter(this, void 0, void 0, function* () {
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
134
|
+
// In Mar 2024, we started having a problem getting a particular gif from imgur using
|
|
135
|
+
// node-fetch. Switching to axios resolved it. I don't know why.
|
|
136
|
+
const response = yield axios_1.default.get(imageSet.primaryUrl, {
|
|
137
|
+
responseType: "arraybuffer",
|
|
138
|
+
});
|
|
139
|
+
imageSet.primaryBuffer = Buffer.from(response.data, "utf-8");
|
|
137
140
|
imageSet.fileType = yield file_type_1.default.fromBuffer(imageSet.primaryBuffer);
|
|
138
141
|
});
|
|
139
142
|
}
|
|
@@ -145,7 +148,7 @@ function saveImage(imageSet) {
|
|
|
145
148
|
// if we have a urls for the localized screenshot, download it
|
|
146
149
|
if ((localizedImage === null || localizedImage === void 0 ? void 0 : localizedImage.url.length) > 0) {
|
|
147
150
|
(0, log_1.verbose)(`Retrieving ${localizedImage.iso632Code} version...`);
|
|
148
|
-
const response = yield (
|
|
151
|
+
const response = yield fetch(localizedImage.url);
|
|
149
152
|
const arrayBuffer = yield response.arrayBuffer();
|
|
150
153
|
buffer = Buffer.from(arrayBuffer);
|
|
151
154
|
}
|
|
@@ -153,7 +156,7 @@ function saveImage(imageSet) {
|
|
|
153
156
|
(0, log_1.verbose)(`No localized image specified for ${localizedImage.iso632Code}, will use primary image.`);
|
|
154
157
|
// otherwise, we're going to fall back to outputting the primary image here
|
|
155
158
|
}
|
|
156
|
-
const directory = `./i18n/${localizedImage.iso632Code}/docusaurus-plugin-content-docs/current/${imageSet.
|
|
159
|
+
const directory = `./i18n/${localizedImage.iso632Code}/docusaurus-plugin-content-docs/current/${imageSet.pageInfo.relativeFilePathToFolderContainingPage}`;
|
|
157
160
|
writeImageIfNew((directory + "/" + imageSet.outputFileName).replaceAll("//", "/"), buffer);
|
|
158
161
|
}
|
|
159
162
|
});
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
export {};
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
|
|
3
|
+
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
|
|
4
|
+
return new (P || (P = Promise))(function (resolve, reject) {
|
|
5
|
+
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
|
|
6
|
+
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
|
|
7
|
+
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
|
|
8
|
+
step((generator = generator.apply(thisArg, _arguments || [])).next());
|
|
9
|
+
});
|
|
10
|
+
};
|
|
11
|
+
var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
12
|
+
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
13
|
+
};
|
|
14
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
15
|
+
const notion_to_md_1 = require("notion-to-md");
|
|
16
|
+
const HierarchicalNamedLayoutStrategy_1 = require("./HierarchicalNamedLayoutStrategy");
|
|
17
|
+
const transform_1 = require("./transform");
|
|
18
|
+
const internalLinks_1 = require("./plugins/internalLinks");
|
|
19
|
+
const pull_1 = require("./pull");
|
|
20
|
+
const default_docunotion_config_1 = __importDefault(require("./config/default.docunotion.config"));
|
|
21
|
+
test("Latex Rendering", () => __awaiter(void 0, void 0, void 0, function* () {
|
|
22
|
+
const pages = new Array();
|
|
23
|
+
const counts = {
|
|
24
|
+
output_normally: 0,
|
|
25
|
+
skipped_because_empty: 0,
|
|
26
|
+
skipped_because_status: 0,
|
|
27
|
+
skipped_because_level_cannot_have_content: 0,
|
|
28
|
+
};
|
|
29
|
+
const notionClient = (0, pull_1.initNotionClient)("");
|
|
30
|
+
const layoutStrategy = new HierarchicalNamedLayoutStrategy_1.HierarchicalNamedLayoutStrategy();
|
|
31
|
+
const config = default_docunotion_config_1.default;
|
|
32
|
+
const context = {
|
|
33
|
+
getBlockChildren: (id) => {
|
|
34
|
+
return new Promise(resolve => resolve(new Array()));
|
|
35
|
+
},
|
|
36
|
+
// this changes with each page
|
|
37
|
+
pageInfo: {
|
|
38
|
+
directoryContainingMarkdown: "",
|
|
39
|
+
relativeFilePathToFolderContainingPage: "",
|
|
40
|
+
slug: "",
|
|
41
|
+
},
|
|
42
|
+
layoutStrategy: layoutStrategy,
|
|
43
|
+
notionToMarkdown: new notion_to_md_1.NotionToMarkdown({ notionClient }),
|
|
44
|
+
options: {
|
|
45
|
+
notionToken: "",
|
|
46
|
+
rootPage: "",
|
|
47
|
+
locales: [""],
|
|
48
|
+
markdownOutputPath: "",
|
|
49
|
+
imgOutputPath: "",
|
|
50
|
+
imgPrefixInMarkdown: "",
|
|
51
|
+
statusTag: "",
|
|
52
|
+
},
|
|
53
|
+
pages: pages,
|
|
54
|
+
counts: counts,
|
|
55
|
+
imports: [],
|
|
56
|
+
convertNotionLinkToLocalDocusaurusLink: (url) => (0, internalLinks_1.convertInternalUrl)(context, url),
|
|
57
|
+
};
|
|
58
|
+
const blocks = [
|
|
59
|
+
{
|
|
60
|
+
object: "block",
|
|
61
|
+
id: "169e1c47-6706-4518-adca-73086b2738ac",
|
|
62
|
+
parent: {
|
|
63
|
+
type: "page_id",
|
|
64
|
+
page_id: "2acc11a4-82a9-4759-b429-fa011c164888",
|
|
65
|
+
},
|
|
66
|
+
created_time: "2023-08-18T15:51:00.000Z",
|
|
67
|
+
last_edited_time: "2023-08-18T15:51:00.000Z",
|
|
68
|
+
created_by: {
|
|
69
|
+
object: "user",
|
|
70
|
+
id: "af5c163e-82b1-49d1-9f1c-539907bb9fb9",
|
|
71
|
+
},
|
|
72
|
+
last_edited_by: {
|
|
73
|
+
object: "user",
|
|
74
|
+
id: "af5c163e-82b1-49d1-9f1c-539907bb9fb9",
|
|
75
|
+
},
|
|
76
|
+
has_children: false,
|
|
77
|
+
archived: false,
|
|
78
|
+
type: "paragraph",
|
|
79
|
+
paragraph: {
|
|
80
|
+
rich_text: [
|
|
81
|
+
{
|
|
82
|
+
type: "equation",
|
|
83
|
+
equation: { expression: "x" },
|
|
84
|
+
annotations: {
|
|
85
|
+
bold: false,
|
|
86
|
+
italic: false,
|
|
87
|
+
strikethrough: false,
|
|
88
|
+
underline: false,
|
|
89
|
+
code: false,
|
|
90
|
+
color: "default",
|
|
91
|
+
},
|
|
92
|
+
plain_text: "x",
|
|
93
|
+
href: null,
|
|
94
|
+
},
|
|
95
|
+
],
|
|
96
|
+
color: "default",
|
|
97
|
+
},
|
|
98
|
+
},
|
|
99
|
+
];
|
|
100
|
+
expect(yield (0, transform_1.getMarkdownFromNotionBlocks)(context, config, blocks)).toContain("$x$");
|
|
101
|
+
}));
|
|
@@ -1,50 +1,79 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
3
|
const MakeImagePersistencePlan_1 = require("./MakeImagePersistencePlan");
|
|
4
|
+
const optionsUsingDefaultNaming = {
|
|
5
|
+
notionToken: "",
|
|
6
|
+
rootPage: "",
|
|
7
|
+
locales: [],
|
|
8
|
+
markdownOutputPath: "",
|
|
9
|
+
imgOutputPath: "",
|
|
10
|
+
imgPrefixInMarkdown: "",
|
|
11
|
+
statusTag: "",
|
|
12
|
+
};
|
|
13
|
+
const testImageSet = {
|
|
14
|
+
primaryUrl: "https://s3.us-west-2.amazonaws.com/primaryImage?Blah=foo",
|
|
15
|
+
localizedUrls: [],
|
|
16
|
+
pageInfo: {
|
|
17
|
+
directoryContainingMarkdown: "/pathToParentSomewhere/",
|
|
18
|
+
relativeFilePathToFolderContainingPage: "",
|
|
19
|
+
slug: "my-page",
|
|
20
|
+
},
|
|
21
|
+
fileType: { ext: "png", mime: "image/png" },
|
|
22
|
+
primaryBuffer: Buffer.from("some fake image content"),
|
|
23
|
+
};
|
|
4
24
|
test("primary file with explicit file output path and prefix", () => {
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
};
|
|
11
|
-
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(imageSet, "./static/notion_imgs", "/notion_imgs");
|
|
12
|
-
const expectedHash = (0, MakeImagePersistencePlan_1.hashOfString)("https://s3.us-west-2.amazonaws.com/primaryImage");
|
|
13
|
-
expect(imageSet.outputFileName).toBe(`${expectedHash}.png`);
|
|
14
|
-
expect(imageSet.primaryFileOutputPath).toBe(`static/notion_imgs/${expectedHash}.png`);
|
|
15
|
-
expect(imageSet.filePathToUseInMarkdown).toBe(`/notion_imgs/${expectedHash}.png`);
|
|
25
|
+
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(optionsUsingDefaultNaming, testImageSet, "ABC-123", "./static/notion_imgs", "/notion_imgs");
|
|
26
|
+
const expectedFileName = "my-page.ABC-123.png";
|
|
27
|
+
expect(testImageSet.outputFileName).toBe(`${expectedFileName}`);
|
|
28
|
+
expect(testImageSet.primaryFileOutputPath).toBe(`static/notion_imgs/${expectedFileName}`);
|
|
29
|
+
expect(testImageSet.filePathToUseInMarkdown).toBe(`/notion_imgs/${expectedFileName}`);
|
|
16
30
|
});
|
|
17
31
|
test("primary file with defaults for image output path and prefix", () => {
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
pathToParentDocument: "/pathToParentSomewhere/",
|
|
22
|
-
fileType: { ext: "png", mime: "image/png" },
|
|
23
|
-
};
|
|
24
|
-
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(imageSet, "", "");
|
|
25
|
-
const expectedHash = (0, MakeImagePersistencePlan_1.hashOfString)("https://s3.us-west-2.amazonaws.com/primaryImage");
|
|
26
|
-
expect(imageSet.outputFileName).toBe(`${expectedHash}.png`);
|
|
32
|
+
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(optionsUsingDefaultNaming, testImageSet, "ABC-123", "", "");
|
|
33
|
+
const expectedFileName = "my-page.ABC-123.png";
|
|
34
|
+
expect(testImageSet.outputFileName).toBe(`${expectedFileName}`);
|
|
27
35
|
// the default behavior is to put the image next to the markdown file
|
|
28
|
-
expect(
|
|
29
|
-
expect(
|
|
30
|
-
});
|
|
31
|
-
test("
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
36
|
+
expect(testImageSet.primaryFileOutputPath).toBe(`/pathToParentSomewhere/${expectedFileName}`);
|
|
37
|
+
expect(testImageSet.filePathToUseInMarkdown).toBe(`./${expectedFileName}`);
|
|
38
|
+
});
|
|
39
|
+
test("falls back to getting file extension from url if not in fileType", () => {
|
|
40
|
+
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(optionsUsingDefaultNaming, testImageSet, "ABC-123", "", "");
|
|
41
|
+
expect(testImageSet.outputFileName).toBe("my-page.ABC-123.png");
|
|
42
|
+
});
|
|
43
|
+
// I'm not sure it is even possible to have encoded characters in the slug, but this proves
|
|
44
|
+
// we are properly encoding them in the markdown file but not in the file system.
|
|
45
|
+
// (This test originally was for including original file names, but we decided not to do that.)
|
|
46
|
+
test("handles encoded characters", () => {
|
|
47
|
+
const imageSet = Object.assign(Object.assign({}, testImageSet), { pageInfo: {
|
|
48
|
+
directoryContainingMarkdown: "/pathToParentSomewhere/",
|
|
49
|
+
relativeFilePathToFolderContainingPage: "",
|
|
50
|
+
slug: "my-page%281%29",
|
|
51
|
+
} });
|
|
52
|
+
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(optionsUsingDefaultNaming, imageSet, "ABC-123", "", "");
|
|
53
|
+
expect(imageSet.primaryFileOutputPath).toBe(`/pathToParentSomewhere/my-page(1).ABC-123.png`);
|
|
54
|
+
expect(imageSet.filePathToUseInMarkdown).toBe(`./my-page%281%29.ABC-123.png`);
|
|
55
|
+
});
|
|
56
|
+
const optionsUsingHashNaming = Object.assign(Object.assign({}, optionsUsingDefaultNaming), { imageFileNameFormat: "content-hash" });
|
|
57
|
+
test("hash naming", () => {
|
|
58
|
+
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(optionsUsingHashNaming, testImageSet, "ABC-123", "", "");
|
|
59
|
+
const expectedFileName = "fe3f26fd515b3cf299ac.png";
|
|
60
|
+
expect(testImageSet.outputFileName).toBe(`${expectedFileName}`);
|
|
61
|
+
});
|
|
62
|
+
const optionsUsingLegacyNaming = Object.assign(Object.assign({}, optionsUsingDefaultNaming), { imageFileNameFormat: "legacy" });
|
|
63
|
+
test("Legacy naming", () => {
|
|
64
|
+
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(optionsUsingLegacyNaming, testImageSet, "ABC-123", "./static/notion_imgs", "/notion_imgs");
|
|
65
|
+
const expectedHash = (0, MakeImagePersistencePlan_1.hashOfString)("https://s3.us-west-2.amazonaws.com/primaryImage");
|
|
66
|
+
expect(testImageSet.outputFileName).toBe(`${expectedHash}.png`);
|
|
67
|
+
});
|
|
68
|
+
test("Legacy naming - properly extract UUID from old-style notion image url", () => {
|
|
69
|
+
const imageSet = Object.assign(Object.assign({}, testImageSet), { primaryUrl: "https://s3.us-west-2.amazonaws.com/secure.notion-static.com/e1058f46-4d2f-4292-8388-4ad393383439/Untitled.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIAT73L2G45EIPT3X45%2F20220516%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20220516T233630Z&X-Amz-Expires=3600&X-Amz-Signature=f215704094fcc884d37073b0b108cf6d1c9da9b7d57a898da38bc30c30b4c4b5&X-Amz-SignedHeaders=host&x-id=GetObject" });
|
|
70
|
+
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(optionsUsingLegacyNaming, imageSet, "ABC-123", "./static/notion_imgs", "/notion_imgs");
|
|
38
71
|
const expectedHash = (0, MakeImagePersistencePlan_1.hashOfString)("e1058f46-4d2f-4292-8388-4ad393383439");
|
|
39
72
|
expect(imageSet.outputFileName).toBe(`${expectedHash}.png`);
|
|
40
73
|
});
|
|
41
|
-
test("properly extract UUID from new-style (Sept 2023) notion image url", () => {
|
|
42
|
-
const imageSet = {
|
|
43
|
-
|
|
44
|
-
localizedUrls: [],
|
|
45
|
-
fileType: { ext: "png", mime: "image/png" },
|
|
46
|
-
};
|
|
47
|
-
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(imageSet, "./static/notion_imgs", "/notion_imgs");
|
|
74
|
+
test("Legacy naming - properly extract UUID from new-style (Sept 2023) notion image url", () => {
|
|
75
|
+
const imageSet = Object.assign(Object.assign({}, testImageSet), { primaryUrl: "https://prod-files-secure.s3.us-west-2.amazonaws.com/d9a2b712-cf69-4bd6-9d65-87a4ceeacca2/d1bcdc8c-b065-4e40-9a11-392aabeb220e/Untitled.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIAT73L2G45EIPT3X45%2F20230915%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20230915T161258Z&X-Amz-Expires=3600&X-Amz-Signature=28fca48e65fba86d539c3c4b7676fce1fa0857aa194f7b33dd4a468ecca6ab24&X-Amz-SignedHeaders=host&x-id=GetObject" });
|
|
76
|
+
(0, MakeImagePersistencePlan_1.makeImagePersistencePlan)(optionsUsingLegacyNaming, imageSet, "ABC-123", "./static/notion_imgs", "/notion_imgs");
|
|
48
77
|
const expectedHash = (0, MakeImagePersistencePlan_1.hashOfString)("d1bcdc8c-b065-4e40-9a11-392aabeb220e");
|
|
49
78
|
expect(imageSet.outputFileName).toBe(`${expectedHash}.png`);
|
|
50
79
|
});
|
|
@@ -7,4 +7,4 @@ export declare function makeSamplePageObject(options: {
|
|
|
7
7
|
name?: string;
|
|
8
8
|
id?: string;
|
|
9
9
|
}): NotionPage;
|
|
10
|
-
export declare function oneBlockToMarkdown(config: IDocuNotionConfig, block:
|
|
10
|
+
export declare function oneBlockToMarkdown(config: IDocuNotionConfig, block: Record<string, unknown>, targetPage?: NotionPage): Promise<string>;
|
|
@@ -49,8 +49,11 @@ children, validApiKey) {
|
|
|
49
49
|
},
|
|
50
50
|
imports: [],
|
|
51
51
|
//TODO might be needed for some tests, e.g. the image transformer...
|
|
52
|
-
|
|
53
|
-
|
|
52
|
+
pageInfo: {
|
|
53
|
+
directoryContainingMarkdown: "not yet",
|
|
54
|
+
relativeFilePathToFolderContainingPage: "not yet",
|
|
55
|
+
slug: "not yet",
|
|
56
|
+
},
|
|
54
57
|
layoutStrategy: new HierarchicalNamedLayoutStrategy_1.HierarchicalNamedLayoutStrategy(),
|
|
55
58
|
options: {
|
|
56
59
|
notionToken: "",
|
|
@@ -90,8 +93,8 @@ children, validApiKey) {
|
|
|
90
93
|
// },
|
|
91
94
|
};
|
|
92
95
|
if (pages && pages.length) {
|
|
93
|
-
console.log(pages[0].matchesLinkId);
|
|
94
|
-
console.log(docunotionContext.pages[0].matchesLinkId);
|
|
96
|
+
// console.log(pages[0].matchesLinkId);
|
|
97
|
+
// console.log(docunotionContext.pages[0].matchesLinkId);
|
|
95
98
|
}
|
|
96
99
|
const r = yield (0, transform_1.getMarkdownFromNotionBlocks)(docunotionContext, config, blocks);
|
|
97
100
|
//console.log("blocksToMarkdown", r);
|
|
@@ -219,7 +222,7 @@ function makeSamplePageObject(options) {
|
|
|
219
222
|
metadata: m,
|
|
220
223
|
foundDirectlyInOutline: false,
|
|
221
224
|
});
|
|
222
|
-
console.log(p.matchesLinkId);
|
|
225
|
+
// console.log(p.matchesLinkId);
|
|
223
226
|
return p;
|
|
224
227
|
}
|
|
225
228
|
exports.makeSamplePageObject = makeSamplePageObject;
|
|
@@ -35,11 +35,15 @@ export type IDocuNotionContext = {
|
|
|
35
35
|
options: DocuNotionOptions;
|
|
36
36
|
getBlockChildren: IGetBlockChildrenFn;
|
|
37
37
|
notionToMarkdown: NotionToMarkdown;
|
|
38
|
-
|
|
39
|
-
relativeFilePathToFolderContainingPage: string;
|
|
38
|
+
pageInfo: IDocuNotionContextPageInfo;
|
|
40
39
|
convertNotionLinkToLocalDocusaurusLink: (url: string) => string | undefined;
|
|
41
40
|
pages: NotionPage[];
|
|
42
41
|
counts: ICounts;
|
|
43
42
|
imports: string[];
|
|
44
43
|
};
|
|
44
|
+
export type IDocuNotionContextPageInfo = {
|
|
45
|
+
directoryContainingMarkdown: string;
|
|
46
|
+
relativeFilePathToFolderContainingPage: string;
|
|
47
|
+
slug: string;
|
|
48
|
+
};
|
|
45
49
|
export {};
|
package/dist/pull.d.ts
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { Client } from "@notionhq/client";
|
|
2
2
|
import { ListBlockChildrenResponseResults } from "notion-to-md/build/types";
|
|
3
|
+
type ImageFileNameFormat = "default" | "content-hash" | "legacy";
|
|
3
4
|
export type DocuNotionOptions = {
|
|
4
5
|
notionToken: string;
|
|
5
6
|
rootPage: string;
|
|
@@ -8,8 +9,11 @@ export type DocuNotionOptions = {
|
|
|
8
9
|
imgOutputPath: string;
|
|
9
10
|
imgPrefixInMarkdown: string;
|
|
10
11
|
statusTag: string;
|
|
12
|
+
requireSlugs?: boolean;
|
|
13
|
+
imageFileNameFormat?: ImageFileNameFormat;
|
|
11
14
|
};
|
|
12
15
|
export declare function notionPull(options: DocuNotionOptions): Promise<void>;
|
|
13
16
|
export declare function executeWithRateLimitAndRetries<T>(label: string, asyncFunction: () => Promise<T>): Promise<T>;
|
|
14
17
|
export declare function initNotionClient(notionToken: string): Client;
|
|
15
18
|
export declare function numberChildrenIfNumberedList(blocks: ListBlockChildrenResponseResults): void;
|
|
19
|
+
export {};
|
package/dist/pull.js
CHANGED
|
@@ -54,6 +54,7 @@ const counts = {
|
|
|
54
54
|
skipped_because_empty: 0,
|
|
55
55
|
skipped_because_status: 0,
|
|
56
56
|
skipped_because_level_cannot_have_content: 0,
|
|
57
|
+
error_because_no_slug: 0,
|
|
57
58
|
};
|
|
58
59
|
function notionPull(options) {
|
|
59
60
|
return __awaiter(this, void 0, void 0, function* () {
|
|
@@ -62,7 +63,7 @@ function notionPull(options) {
|
|
|
62
63
|
const optionsForLogging = Object.assign({}, options);
|
|
63
64
|
// Just show the first few letters of the notion token, which start with "secret" anyhow.
|
|
64
65
|
optionsForLogging.notionToken =
|
|
65
|
-
optionsForLogging.notionToken.substring(0,
|
|
66
|
+
optionsForLogging.notionToken.substring(0, 10) + "...";
|
|
66
67
|
const config = yield (0, configuration_1.loadConfigAsync)();
|
|
67
68
|
(0, log_1.verbose)(`Options:${JSON.stringify(optionsForLogging, null, 2)}`);
|
|
68
69
|
yield (0, images_1.initImageHandling)(options.imgPrefixInMarkdown || options.imgOutputPath || "", options.imgOutputPath || "", options.locales);
|
|
@@ -73,6 +74,16 @@ function notionPull(options) {
|
|
|
73
74
|
layoutStrategy.setRootDirectoryForMarkdown(options.markdownOutputPath.replace(/\/+$/, "") // trim any trailing slash
|
|
74
75
|
);
|
|
75
76
|
(0, log_1.info)("Connecting to Notion...");
|
|
77
|
+
// Do a quick test to see if we can connect to the root so that we can give a better error than just a generic "could not find page" one.
|
|
78
|
+
try {
|
|
79
|
+
yield executeWithRateLimitAndRetries("retrieving root page", () => __awaiter(this, void 0, void 0, function* () {
|
|
80
|
+
yield notionClient.pages.retrieve({ page_id: options.rootPage });
|
|
81
|
+
}));
|
|
82
|
+
}
|
|
83
|
+
catch (e) {
|
|
84
|
+
(0, log_1.error)(`docu-notion could not retrieve the root page from Notion. \r\na) Check that the root page id really is "${options.rootPage}".\r\nb) Check that your Notion API token (the "Integration Secret") is correct. It starts with "${optionsForLogging.notionToken}".\r\nc) Check that your root page includes your "integration" in its "connections".\r\nThis internal error message may help:\r\n ${e.message}`);
|
|
85
|
+
(0, process_1.exit)(1);
|
|
86
|
+
}
|
|
76
87
|
(0, log_1.group)("Stage 1: walk children of the page named 'Outline', looking for pages...");
|
|
77
88
|
yield getPagesRecursively(options, "", options.rootPage, 0, true);
|
|
78
89
|
(0, log_1.logDebug)("getPagesRecursively", JSON.stringify(pages, null, 2));
|
|
@@ -92,8 +103,12 @@ function outputPages(options, config, pages) {
|
|
|
92
103
|
return __awaiter(this, void 0, void 0, function* () {
|
|
93
104
|
const context = {
|
|
94
105
|
getBlockChildren: getBlockChildren,
|
|
95
|
-
|
|
96
|
-
|
|
106
|
+
// this changes with each page
|
|
107
|
+
pageInfo: {
|
|
108
|
+
directoryContainingMarkdown: "",
|
|
109
|
+
relativeFilePathToFolderContainingPage: "",
|
|
110
|
+
slug: "",
|
|
111
|
+
},
|
|
97
112
|
layoutStrategy: layoutStrategy,
|
|
98
113
|
notionToMarkdown: notionToMarkdown,
|
|
99
114
|
options: options,
|
|
@@ -106,10 +121,11 @@ function outputPages(options, config, pages) {
|
|
|
106
121
|
layoutStrategy.pageWasSeen(page);
|
|
107
122
|
const mdPath = layoutStrategy.getPathForPage(page, ".md");
|
|
108
123
|
// most plugins should not write to disk, but those handling image files need these paths
|
|
109
|
-
context.directoryContainingMarkdown = Path.dirname(mdPath);
|
|
124
|
+
context.pageInfo.directoryContainingMarkdown = Path.dirname(mdPath);
|
|
110
125
|
// TODO: This needs clarifying: getLinkPathForPage() is about urls, but
|
|
111
126
|
// downstream images.ts is using it as a file system path
|
|
112
|
-
context.relativeFilePathToFolderContainingPage = Path.dirname(layoutStrategy.getLinkPathForPage(page));
|
|
127
|
+
context.pageInfo.relativeFilePathToFolderContainingPage = Path.dirname(layoutStrategy.getLinkPathForPage(page));
|
|
128
|
+
context.pageInfo.slug = page.slug;
|
|
113
129
|
if (page.type === NotionPage_1.PageType.DatabasePage &&
|
|
114
130
|
context.options.statusTag != "*" &&
|
|
115
131
|
page.status !== context.options.statusTag) {
|
|
@@ -117,10 +133,16 @@ function outputPages(options, config, pages) {
|
|
|
117
133
|
++context.counts.skipped_because_status;
|
|
118
134
|
}
|
|
119
135
|
else {
|
|
136
|
+
if (options.requireSlugs && !page.hasExplicitSlug) {
|
|
137
|
+
(0, log_1.error)(`Page "${page.nameOrTitle}" is missing a required slug. (--require-slugs is set.)`);
|
|
138
|
+
++counts.error_because_no_slug;
|
|
139
|
+
}
|
|
120
140
|
const markdown = yield (0, transform_1.getMarkdownForPage)(config, context, page);
|
|
121
141
|
writePage(page, markdown);
|
|
122
142
|
}
|
|
123
143
|
}
|
|
144
|
+
if (counts.error_because_no_slug > 0)
|
|
145
|
+
(0, process_1.exit)(1);
|
|
124
146
|
(0, log_1.info)(`Finished processing ${pages.length} pages`);
|
|
125
147
|
(0, log_1.info)(JSON.stringify(counts));
|
|
126
148
|
});
|
package/dist/run.d.ts
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
export declare function run(): void
|
|
1
|
+
export declare function run(): Promise<void>;
|
package/dist/run.js
CHANGED
|
@@ -1,33 +1,90 @@
|
|
|
1
1
|
"use strict";
|
|
2
|
+
var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
|
|
3
|
+
if (k2 === undefined) k2 = k;
|
|
4
|
+
var desc = Object.getOwnPropertyDescriptor(m, k);
|
|
5
|
+
if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
|
|
6
|
+
desc = { enumerable: true, get: function() { return m[k]; } };
|
|
7
|
+
}
|
|
8
|
+
Object.defineProperty(o, k2, desc);
|
|
9
|
+
}) : (function(o, m, k, k2) {
|
|
10
|
+
if (k2 === undefined) k2 = k;
|
|
11
|
+
o[k2] = m[k];
|
|
12
|
+
}));
|
|
13
|
+
var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
|
|
14
|
+
Object.defineProperty(o, "default", { enumerable: true, value: v });
|
|
15
|
+
}) : function(o, v) {
|
|
16
|
+
o["default"] = v;
|
|
17
|
+
});
|
|
18
|
+
var __importStar = (this && this.__importStar) || function (mod) {
|
|
19
|
+
if (mod && mod.__esModule) return mod;
|
|
20
|
+
var result = {};
|
|
21
|
+
if (mod != null) for (var k in mod) if (k !== "default" && Object.prototype.hasOwnProperty.call(mod, k)) __createBinding(result, mod, k);
|
|
22
|
+
__setModuleDefault(result, mod);
|
|
23
|
+
return result;
|
|
24
|
+
};
|
|
25
|
+
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
|
|
26
|
+
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
|
|
27
|
+
return new (P || (P = Promise))(function (resolve, reject) {
|
|
28
|
+
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
|
|
29
|
+
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
|
|
30
|
+
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
|
|
31
|
+
step((generator = generator.apply(thisArg, _arguments || [])).next());
|
|
32
|
+
});
|
|
33
|
+
};
|
|
34
|
+
var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
35
|
+
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
36
|
+
};
|
|
2
37
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
38
|
exports.run = void 0;
|
|
39
|
+
const fs = __importStar(require("fs-extra"));
|
|
4
40
|
const commander_1 = require("commander");
|
|
5
41
|
const log_1 = require("./log");
|
|
6
42
|
const pull_1 = require("./pull");
|
|
43
|
+
const path_1 = __importDefault(require("path"));
|
|
7
44
|
function run() {
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
.
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
45
|
+
return __awaiter(this, void 0, void 0, function* () {
|
|
46
|
+
const pkg = require("../package.json");
|
|
47
|
+
// eslint-disable-next-line @typescript-eslint/restrict-template-expressions
|
|
48
|
+
console.log(`docu-notion version ${pkg.version}`);
|
|
49
|
+
commander_1.program.name("docu-notion").description("");
|
|
50
|
+
commander_1.program.usage("-n <token> -r <root> [options]");
|
|
51
|
+
commander_1.program
|
|
52
|
+
.requiredOption("-n, --notion-token <string>", "notion api token, which looks like secret_3bc1b50XFYb15123RHF243x43450XFY33250XFYa343")
|
|
53
|
+
.requiredOption("-r, --root-page <string>", "The 31 character ID of the page which is the root of your docs page in notion. The code will look like 9120ec9960244ead80fa2ef4bc1bba25. This page must have a child page named 'Outline'")
|
|
54
|
+
.option("-m, --markdown-output-path <string>", "Root of the hierarchy for md files. WARNING: docu-notion will delete files from this directory. Note also that if it finds localized images, it will create an i18n/ directory as a sibling.", "./docs")
|
|
55
|
+
.option("--css-output-directory <string>", "docu-notion has a docu-notion-styles.css file that you will need to use to get things like notion columns to look right. This option specifies where that file should be copied to.", "./css")
|
|
56
|
+
.option("-t, --status-tag <string>", "Database pages without a Notion page property 'status' matching this will be ignored. Use '*' to ignore status altogether.", "Publish")
|
|
57
|
+
.option("--locales <codes>", "Comma-separated list of iso 639-2 codes, the same list as in docusaurus.config.js, minus the primary (i.e. 'en'). This is needed for image localization.", parseLocales, [])
|
|
58
|
+
.addOption(new commander_1.Option("-l, --log-level <level>", "Log level").choices([
|
|
59
|
+
"info",
|
|
60
|
+
"verbose",
|
|
61
|
+
"debug",
|
|
62
|
+
]))
|
|
63
|
+
.option("-i, --img-output-path <string>", "Path to directory where images will be stored. If this is not included, images will be placed in the same directory as the document that uses them, which then allows for localization of screenshots.")
|
|
64
|
+
.option("-p, --img-prefix-in-markdown <string>", "When referencing an image from markdown, prefix with this path instead of the full img-output-path. Should be used only in conjunction with --img-output-path.")
|
|
65
|
+
.option("--require-slugs", "If set, docu-notion will fail if any pages it would otherwise publish are missing a slug in Notion.", false)
|
|
66
|
+
.addOption(new commander_1.Option("--image-file-name-format <format>", "format:\n- default: {page slug (if any)}.{image block ID}\n- content-hash: Use a hash of the image content.\n- legacy: Use the legacy (before v0.16) method of determining file names. Set this to maintain backward compatibility.\nAll formats will use the original file extension.")
|
|
67
|
+
.choices(["default", "content-hash", "legacy"])
|
|
68
|
+
.default("default"));
|
|
69
|
+
commander_1.program.showHelpAfterError();
|
|
70
|
+
commander_1.program.parse();
|
|
71
|
+
(0, log_1.setLogLevel)(commander_1.program.opts().logLevel);
|
|
72
|
+
console.log(JSON.stringify(commander_1.program.opts()));
|
|
73
|
+
// copy in the this version of the css needed to make columns (and maybe other things?) work
|
|
74
|
+
let pathToCss = "";
|
|
75
|
+
try {
|
|
76
|
+
pathToCss = require.resolve("@sillsdev/docu-notion/dist/docu-notion-styles.css");
|
|
77
|
+
}
|
|
78
|
+
catch (e) {
|
|
79
|
+
// when testing from the docu-notion project itself:
|
|
80
|
+
pathToCss = "./src/css/docu-notion-styles.css";
|
|
81
|
+
}
|
|
82
|
+
// make any missing parts of the path exist
|
|
83
|
+
fs.ensureDirSync(commander_1.program.opts().cssOutputDirectory);
|
|
84
|
+
fs.copyFileSync(pathToCss, path_1.default.join(commander_1.program.opts().cssOutputDirectory, "docu-notion-styles.css"));
|
|
85
|
+
// pull and convert
|
|
86
|
+
yield (0, pull_1.notionPull)(commander_1.program.opts()).then(() => console.log("docu-notion Finished."));
|
|
87
|
+
});
|
|
31
88
|
}
|
|
32
89
|
exports.run = run;
|
|
33
90
|
function parseLocales(value) {
|
package/package.json
CHANGED
|
@@ -12,33 +12,34 @@
|
|
|
12
12
|
"tsc": "tsc",
|
|
13
13
|
"// test out with a private sample notion db": "",
|
|
14
14
|
"large-site-test": "npm run ts -- -n $SIL_BLOOM_DOCS_NOTION_TOKEN -r $SIL_BLOOM_DOCS_NOTION_ROOT_PAGE --locales en,fr --log-level debug",
|
|
15
|
-
"pull-test-tagged": "npm run ts -- -n $DOCU_NOTION_INTEGRATION_TOKEN -r $DOCU_NOTION_TEST_ROOT_PAGE_ID --log-level
|
|
15
|
+
"pull-test-tagged": "npm run ts -- -n $DOCU_NOTION_INTEGRATION_TOKEN -r $DOCU_NOTION_TEST_ROOT_PAGE_ID --log-level info --status-tag test",
|
|
16
|
+
"pull-test-css": "npm run ts -- --css-output-directory ./test/css -n $DOCU_NOTION_INTEGRATION_TOKEN -r $DOCU_NOTION_TEST_ROOT_PAGE_ID --log-level debug --status-tag test",
|
|
16
17
|
"pull-sample-site": "npm run ts -- -n $DOCU_NOTION_INTEGRATION_TOKEN -r $DOCU_NOTION_SAMPLE_ROOT_PAGE --log-level debug",
|
|
17
18
|
"// test with a semi-stable/public site:": "",
|
|
18
19
|
"pull-sample": "npm run ts -- -n $DOCU_NOTION_INTEGRATION_TOKEN -r $DOCU_NOTION_SAMPLE_ROOT_PAGE -m ./sample --locales en,es,fr,de --log-level verbose",
|
|
19
|
-
"pull-sample-with-paths": "npm run ts -- -n $DOCU_NOTION_INTEGRATION_TOKEN -r $DOCU_NOTION_SAMPLE_ROOT_PAGE -m ./sample --img-output-path ./sample_img"
|
|
20
|
+
"pull-sample-with-paths": "npm run ts -- -n $DOCU_NOTION_INTEGRATION_TOKEN -r $DOCU_NOTION_SAMPLE_ROOT_PAGE -m ./sample --img-output-path ./sample_img",
|
|
21
|
+
"lint": "eslint . --ext .ts"
|
|
20
22
|
},
|
|
21
23
|
"//file-type": "have to use this version before they switched to ESM, which gives a compile error related to require()",
|
|
22
|
-
"//node-fetch@2.6.6file-type": "have to use this version before they switched to ESM, which gives a compile error related to require()",
|
|
23
24
|
"//chalk@4": "also ESM related problem",
|
|
24
25
|
"//notion-client@4": "also ESM related problem",
|
|
25
26
|
"//note: ts-node": "really is a runtime dependency",
|
|
26
27
|
"dependencies": {
|
|
27
28
|
"@notionhq/client": "2.2.3",
|
|
29
|
+
"axios": "^1.6.8",
|
|
28
30
|
"chalk": "^4.1.2",
|
|
29
31
|
"commander": "^9.2.0",
|
|
30
32
|
"cosmiconfig": "^8.0.0",
|
|
31
33
|
"cosmiconfig-typescript-loader": "^4.3.0",
|
|
32
|
-
"file-type": "16.5.
|
|
34
|
+
"file-type": "16.5.3",
|
|
33
35
|
"fs-extra": "^10.1.0",
|
|
34
36
|
"limiter": "^2.1.0",
|
|
35
37
|
"markdown-table": "^2.0.0",
|
|
36
|
-
"node-fetch": "2.6.6",
|
|
37
38
|
"notion-client": "^4",
|
|
38
39
|
"notion-to-md": "3.1.1",
|
|
39
40
|
"path": "^0.12.7",
|
|
40
|
-
"
|
|
41
|
-
"
|
|
41
|
+
"sanitize-filename": "^1.6.3",
|
|
42
|
+
"ts-node": "^10.2.1"
|
|
42
43
|
},
|
|
43
44
|
"devDependencies": {
|
|
44
45
|
"@types/fs-extra": "^9.0.13",
|
|
@@ -90,5 +91,5 @@
|
|
|
90
91
|
"volta": {
|
|
91
92
|
"node": "18.16.0"
|
|
92
93
|
},
|
|
93
|
-
"version": "0.
|
|
94
|
+
"version": "0.16.0"
|
|
94
95
|
}
|