@promptbook/website-crawler 0.114.0-1 → 0.114.0-3

package/README.md

@@ -504,26 +504,27 @@ ptbk coder verify
 | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | ------ | ---- | ----- | ------------------------------------------------------------------------ |
 | `ptbk coder init`                     | Creates `prompts/`, `prompts/done/`, the project-generic template files materialized in `prompts/templates/` (currently `common.md`), and a starter `AGENTS.md`; ensures `.env` contains `CODING_AGENT_GIT_NAME`, `CODING_AGENT_GIT_EMAIL`, and `CODING_AGENT_GIT_SIGNING_KEY`; adds helper coder scripts to `package.json`; ensures `.gitignore` contains `/.promptbook`; and configures `.vscode/settings.json` to save pasted prompt images into `prompts/screenshots/`. |
 | `ptbk coder generate-boilerplates`    | Creates new prompt markdown files with fresh emoji tags so you can quickly fill in coding tasks; `--template` accepts either a built-in alias or a markdown file path relative to the project root.                                                                                                                                                                                                                                                                         |
-| `ptbk coder run`                      | Picks the next ready prompt, appends optional context, runs it through the selected coding agent, can optionally verify each attempt with a shell test command and feed failing output back for retries, then marks success or failure, commits the result, and pushes only when `--auto-push` is enabled.                                                                                                                                                                  |
+| `ptbk coder run`                      | Picks the next ready prompt, can run tests before coding to stop or repair pre-existing failures, appends optional context, runs it through the selected coding agent, can optionally verify each attempt with a shell test command and feed failing output back for retries, then marks success or failure, commits the result, and pushes only when `--auto-push` is enabled.                                                                                             |
 | `ptbk coder find-refactor-candidates` | Scans the repository for oversized or overpacked files and writes prompt files for likely refactors; `--level <xlow                                                                                                                                                                                                                                                                                                                                                         | low | medium | high | xhigh | extreme>` ranges from a very benevolent scan to a very aggressive sweep. |
 | `ptbk coder verify`                   | Walks through completed prompts, archives truly finished work, and adds follow-up repair prompts for unfinished results.                                                                                                                                                                                                                                                                                                                                                    |
 
 #### Most useful `ptbk coder run` flags
 
-| Flag                       | Purpose                                                                                            |
-| -------------------------- | -------------------------------------------------------------------------------------------------- |
-| `--harness <name>`         | Selects the coding harness.                                                                        |
-| `--model <model>`          | Chooses the runner model; required for `openai-codex` and `gemini`, optional for `github-copilot`. |
-| `--context <text-or-file>` | Appends extra instructions inline or from a file like `AGENTS.md`.                                 |
-| `--test <command>`         | Runs a verification command after each prompt attempt and feeds failing output back for retries.   |
-| `--thinking-level <level>` | Sets reasoning effort for supported runners.                                                       |
-| `--no-auto`                | Waits for user confirmation before each prompt instead of running automatically through the queue. |
-| `--ignore-git-changes`     | Disables the clean-working-tree guard.                                                             |
-| `--priority <n>`           | Runs only prompts at or above the given priority.                                                  |
-| `--dry-run`                | Prints which prompts are ready instead of executing them.                                          |
-| `--allow-credits`          | Lets OpenAI Codex spend credits when required.                                                     |
-| `--auto-push`              | Pushes each successful coding-agent commit to the configured remote.                               |
-| `--auto-migrate`           | Runs testing-server database migrations after each successful prompt.                              |
+| Flag                       | Purpose                                                                                                                                                                                |
+| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--harness <name>`         | Selects the coding harness.                                                                                                                                                            |
+| `--model <model>`          | Chooses the runner model; required for `openai-codex` and `gemini`, optional for `github-copilot`.                                                                                     |
+| `--context <text-or-file>` | Appends extra instructions inline or from a file like `AGENTS.md`.                                                                                                                     |
+| `--test <command>`         | Runs a verification command after each prompt attempt and feeds failing output back for retries.                                                                                       |
+| `--test-before <mode>`     | Runs tests before coding: `no` (default), `yes-and-fail` (stop with results), or `yes-and-fix` (create one repair prompt first); defaults to `npm test` when enabled without `--test`. |
+| `--thinking-level <level>` | Sets reasoning effort for supported runners.                                                                                                                                           |
+| `--no-auto`                | Waits for user confirmation before each prompt instead of running automatically through the queue.                                                                                     |
+| `--ignore-git-changes`     | Disables the clean-working-tree guard.                                                                                                                                                 |
+| `--priority <n>`           | Runs only prompts at or above the given priority.                                                                                                                                      |
+| `--dry-run`                | Prints which prompts are ready instead of executing them.                                                                                                                              |
+| `--allow-credits`          | Lets OpenAI Codex spend credits when required.                                                                                                                                         |
+| `--auto-push`              | Pushes each successful coding-agent commit to the configured remote.                                                                                                                   |
+| `--auto-migrate`           | Runs testing-server database migrations after each successful prompt.                                                                                                                  |
 
 #### Typical usage pattern
 

package/esm/index.es.js

@@ -1,6 +1,4 @@
 import { spaceTrim as spaceTrim$1 } from 'spacetrim';
-import { Readability } from '@mozilla/readability';
-import { JSDOM } from 'jsdom';
 import CryptoJS from 'crypto-js';
 import { Subject } from 'rxjs';
 import { forTime } from 'waitasecond';
@@ -26,7 +24,7 @@ const BOOK_LANGUAGE_VERSION = '2.0.0';
  * @generated
  * @see https://github.com/webgptorg/promptbook
  */
-const PROMPTBOOK_ENGINE_VERSION = '0.114.0-1';
+const PROMPTBOOK_ENGINE_VERSION = '0.114.0-3';
 /**
  * TODO: string_promptbook_version should be constrained to the all versions of Promptbook engine
  * Note: [💞] Ignore a discrepancy between file name and entity name
@@ -1601,6 +1599,31 @@ class KnowledgeScrapeError extends Error {
     }
 }
 
+/**
+ * Creates a loader which imports one module on the first call and reuses the very same module afterwards
+ *
+ * Note: [🐌] Heavy third-party dependencies are imported lazily to keep the startup of the Promptbook CLI fast.
+ *       A statically imported dependency is loaded every single time the bundle is loaded, even when the running
+ *       command never touches it. A lazily imported dependency is loaded only when the feature is really used.
+ *
+ * @example
+ * const loadJsdomModule = createLazyModuleLoader(() => import('jsdom'));
+ * const { JSDOM } = await loadJsdomModule();
+ *
+ * @private internal utility of Promptbook
+ */
+function createLazyModuleLoader(importModule) {
+    let importedModulePromise = null;
+    return function loadModule() {
+        if (importedModulePromise === null) {
+            importedModulePromise = importModule();
+        }
+        return importedModulePromise;
+    };
+}
+// Note: [🐌] Do not convert the lazy `import(...)` calls back to static `import` statements, it would bring back the
+//       slow startup of the `ptbk` CLI utility
+
 var PipelineCollection = [{title:"Prepare Knowledge from Markdown",pipelineUrl:"https://promptbook.studio/promptbook/prepare-knowledge-from-markdown.book",formfactorName:"GENERIC",parameters:[{name:"knowledgeContent",description:"Markdown document content",isInput:true,isOutput:false},{name:"knowledgePieces",description:"The knowledge JSON object",isInput:false,isOutput:true}],tasks:[{taskType:"PROMPT_TASK",name:"knowledge",title:"Knowledge",content:"You are experienced data researcher, extract the important knowledge from the document.\n\n# Rules\n\n-   Make pieces of information concise, clear, and easy to understand\n-   One piece of information should be approximately 1 paragraph\n-   Divide the paragraphs by markdown horizontal lines ---\n-   Omit irrelevant information\n-   Group redundant information\n-   Write just extracted information, nothing else\n\n# The document\n\nTake information from this document:\n\n> {knowledgeContent}",resultingParameterName:"knowledgePieces",dependentParameterNames:["knowledgeContent"]}],personas:[],preparations:[],knowledgeSources:[],knowledgePieces:[],sources:[{type:"BOOK",path:null,content:"# Prepare Knowledge from Markdown\n\n-   PIPELINE URL `https://promptbook.studio/promptbook/prepare-knowledge-from-markdown.book`\n-   INPUT PARAMETER `{knowledgeContent}` Markdown document content\n-   OUTPUT PARAMETER `{knowledgePieces}` The knowledge JSON object\n\n## Knowledge\n\n<!-- TODO: [🍆] -FORMAT JSON -->\n\n```markdown\nYou are experienced data researcher, extract the important knowledge from the document.\n\n# Rules\n\n-   Make pieces of information concise, clear, and easy to understand\n-   One piece of information should be approximately 1 paragraph\n-   Divide the paragraphs by markdown horizontal lines ---\n-   Omit irrelevant information\n-   Group redundant information\n-   Write just extracted information, nothing else\n\n# The document\n\nTake information from this document:\n\n> {knowledgeContent}\n```\n\n`-> {knowledgePieces}`\n"}],sourceFile:"./books/prepare-knowledge-from-markdown.book"},{title:"Prepare Keywords",pipelineUrl:"https://promptbook.studio/promptbook/prepare-knowledge-keywords.book",formfactorName:"GENERIC",parameters:[{name:"knowledgePieceContent",description:"The content",isInput:true,isOutput:false},{name:"keywords",description:"Keywords separated by comma",isInput:false,isOutput:true}],tasks:[{taskType:"PROMPT_TASK",name:"knowledge",title:"Knowledge",content:"You are experienced data researcher, detect the important keywords in the document.\n\n# Rules\n\n-   Write just keywords separated by comma\n\n# The document\n\nTake information from this document:\n\n> {knowledgePieceContent}",resultingParameterName:"keywords",dependentParameterNames:["knowledgePieceContent"]}],personas:[],preparations:[],knowledgeSources:[],knowledgePieces:[],sources:[{type:"BOOK",path:null,content:"# Prepare Keywords\n\n-   PIPELINE URL `https://promptbook.studio/promptbook/prepare-knowledge-keywords.book`\n-   INPUT PARAMETER `{knowledgePieceContent}` The content\n-   OUTPUT PARAMETER `{keywords}` Keywords separated by comma\n\n## Knowledge\n\n<!-- TODO: [🍆] -FORMAT JSON -->\n\n```markdown\nYou are experienced data researcher, detect the important keywords in the document.\n\n# Rules\n\n-   Write just keywords separated by comma\n\n# The document\n\nTake information from this document:\n\n> {knowledgePieceContent}\n```\n\n`-> {keywords}`\n"}],sourceFile:"./books/prepare-knowledge-keywords.book"},{title:"Prepare Knowledge-piece Title",pipelineUrl:"https://promptbook.studio/promptbook/prepare-knowledge-title.book",formfactorName:"GENERIC",parameters:[{name:"knowledgePieceContent",description:"The content",isInput:true,isOutput:false},{name:"title",description:"The title of the document",isInput:false,isOutput:true}],tasks:[{taskType:"PROMPT_TASK",name:"knowledge",title:"Knowledge",content:"You are experienced content creator, write best title for the document.\n\n# Rules\n\n-   Write just title, nothing else\n-   Write maximum 5 words for the title\n\n# The document\n\n> {knowledgePieceContent}",resultingParameterName:"title",expectations:{words:{min:1,max:8}},dependentParameterNames:["knowledgePieceContent"]}],personas:[],preparations:[],knowledgeSources:[],knowledgePieces:[],sources:[{type:"BOOK",path:null,content:"# Prepare Knowledge-piece Title\n\n-   PIPELINE URL `https://promptbook.studio/promptbook/prepare-knowledge-title.book`\n-   INPUT PARAMETER `{knowledgePieceContent}` The content\n-   OUTPUT PARAMETER `{title}` The title of the document\n\n## Knowledge\n\n-   EXPECT MIN 1 WORD\n-   EXPECT MAX 8 WORDS\n\n```markdown\nYou are experienced content creator, write best title for the document.\n\n# Rules\n\n-   Write just title, nothing else\n-   Write maximum 5 words for the title\n\n# The document\n\n> {knowledgePieceContent}\n```\n\n`-> {title}`\n"}],sourceFile:"./books/prepare-knowledge-title.book"},{title:"Prepare Persona",pipelineUrl:"https://promptbook.studio/promptbook/prepare-persona.book",formfactorName:"GENERIC",parameters:[{name:"availableModels",description:"List of available model names together with their descriptions as JSON",isInput:true,isOutput:false},{name:"personaDescription",description:"Description of the persona",isInput:true,isOutput:false},{name:"modelsRequirements",description:"Specific requirements for the model",isInput:false,isOutput:true}],tasks:[{taskType:"PROMPT_TASK",name:"make-model-requirements",title:"Make modelRequirements",content:"You are an experienced AI engineer, you need to find the best models for virtual assistants:\n\n## Example\n\n```json\n[\n  {\n  \"modelName\": \"gpt-4o\",\n  \"systemMessage\": \"You are experienced AI engineer and helpful assistant.\",\n  \"temperature\": 0.7\n  },\n  {\n    \"modelName\": \"claude-3-5-sonnet\",\n    \"systemMessage\": \"You are a friendly and knowledgeable chatbot.\",\n    \"temperature\": 0.5\n  }\n]\n```\n\n## Instructions\n\n-   Your output format is JSON array\n-   Sort best-fitting models first\n-   Omit any models that are not suitable\n-   Write just the JSON, no other text should be present\n-   Array contain items with following keys:\n    -   `modelName`: The name of the model to use\n    -   `systemMessage`: The system message to provide context to the model\n    -   `temperature`: The sampling temperature to use\n\n### Key `modelName`\n\nHere are the available models:\n\n```json\n{availableModels}\n```\n\n### Key `systemMessage`\n\nThe system message is used to communicate instructions or provide context to the model at the beginning of a conversation. It is displayed in a different format compared to user messages, helping the model understand its role in the conversation. The system message typically guides the model's behavior, sets the tone, or specifies desired output from the model. By utilizing the system message effectively, users can steer the model towards generating more accurate and relevant responses.\n\nFor example:\n\n> You are an experienced AI engineer and helpful assistant.\n\n> You are a friendly and knowledgeable chatbot.\n\n### Key `temperature`\n\nThe sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. If set to 0, the model will use log probability to automatically increase the temperature until certain thresholds are hit.\n\nYou can pick a value between 0 and 2. For example:\n\n-   `0.1`: Low temperature, extremely conservative and deterministic\n-   `0.5`: Medium temperature, balanced between conservative and creative\n-   `1.0`: High temperature, creative and bit random\n-   `1.5`: Very high temperature, extremely creative and often chaotic and unpredictable\n-   `2.0`: Maximum temperature, completely random and unpredictable, for some extreme creative use cases\n\n# The assistant\n\nTake this description of the persona:\n\n> {personaDescription}",resultingParameterName:"modelsRequirements",format:"JSON",dependentParameterNames:["availableModels","personaDescription"]}],personas:[],preparations:[],knowledgeSources:[],knowledgePieces:[],sources:[{type:"BOOK",path:null,content:"# Prepare Persona\n\n-   PIPELINE URL `https://promptbook.studio/promptbook/prepare-persona.book`\n-   INPUT PARAMETER `{availableModels}` List of available model names together with their descriptions as JSON\n-   INPUT PARAMETER `{personaDescription}` Description of the persona\n-   OUTPUT PARAMETER `{modelsRequirements}` Specific requirements for the model\n\n## Make modelRequirements\n\n-   FORMAT JSON\n\n```markdown\nYou are an experienced AI engineer, you need to find the best models for virtual assistants:\n\n## Example\n\n\\`\\`\\`json\n[\n  {\n  \"modelName\": \"gpt-4o\",\n  \"systemMessage\": \"You are experienced AI engineer and helpful assistant.\",\n  \"temperature\": 0.7\n  },\n  {\n    \"modelName\": \"claude-3-5-sonnet\",\n    \"systemMessage\": \"You are a friendly and knowledgeable chatbot.\",\n    \"temperature\": 0.5\n  }\n]\n\\`\\`\\`\n\n## Instructions\n\n-   Your output format is JSON array\n-   Sort best-fitting models first\n-   Omit any models that are not suitable\n-   Write just the JSON, no other text should be present\n-   Array contain items with following keys:\n    -   `modelName`: The name of the model to use\n    -   `systemMessage`: The system message to provide context to the model\n    -   `temperature`: The sampling temperature to use\n\n### Key `modelName`\n\nHere are the available models:\n\n\\`\\`\\`json\n{availableModels}\n\\`\\`\\`\n\n### Key `systemMessage`\n\nThe system message is used to communicate instructions or provide context to the model at the beginning of a conversation. It is displayed in a different format compared to user messages, helping the model understand its role in the conversation. The system message typically guides the model's behavior, sets the tone, or specifies desired output from the model. By utilizing the system message effectively, users can steer the model towards generating more accurate and relevant responses.\n\nFor example:\n\n> You are an experienced AI engineer and helpful assistant.\n\n> You are a friendly and knowledgeable chatbot.\n\n### Key `temperature`\n\nThe sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. If set to 0, the model will use log probability to automatically increase the temperature until certain thresholds are hit.\n\nYou can pick a value between 0 and 2. For example:\n\n-   `0.1`: Low temperature, extremely conservative and deterministic\n-   `0.5`: Medium temperature, balanced between conservative and creative\n-   `1.0`: High temperature, creative and bit random\n-   `1.5`: Very high temperature, extremely creative and often chaotic and unpredictable\n-   `2.0`: Maximum temperature, completely random and unpredictable, for some extreme creative use cases\n\n# The assistant\n\nTake this description of the persona:\n\n> {personaDescription}\n```\n\n`-> {modelsRequirements}`\n"}],sourceFile:"./books/prepare-persona.book"},{title:"Prepare Title",pipelineUrl:"https://promptbook.studio/promptbook/prepare-title.book",formfactorName:"GENERIC",parameters:[{name:"book",description:"The book to prepare the title for",isInput:true,isOutput:false},{name:"title",description:"Best title for the book",isInput:false,isOutput:true}],tasks:[{taskType:"PROMPT_TASK",name:"make-title",title:"Make title",content:"Make best title for given text which describes the workflow:\n\n## Rules\n\n-   Write just title, nothing else\n-   Title should be concise and clear - Write maximum ideally 2 words, maximum 5 words\n-   Title starts with emoticon\n-   Title should not mention the input and output of the workflow but the main purpose of the workflow\n    _For example, not \"✍ Convert Knowledge-piece to title\" but \"✍ Title\"_\n\n## The workflow\n\n> {book}",resultingParameterName:"title",expectations:{words:{min:1,max:8},lines:{min:1,max:1}},dependentParameterNames:["book"]}],personas:[],preparations:[],knowledgeSources:[],knowledgePieces:[],sources:[{type:"BOOK",path:null,content:"# Prepare Title\n\n-   PIPELINE URL `https://promptbook.studio/promptbook/prepare-title.book`\n-   INPUT PARAMETER `{book}` The book to prepare the title for\n-   OUTPUT PARAMETER `{title}` Best title for the book\n\n## Make title\n\n-   EXPECT MIN 1 Word\n-   EXPECT MAX 8 Words\n-   EXPECT EXACTLY 1 Line\n\n```markdown\nMake best title for given text which describes the workflow:\n\n## Rules\n\n-   Write just title, nothing else\n-   Title should be concise and clear - Write maximum ideally 2 words, maximum 5 words\n-   Title starts with emoticon\n-   Title should not mention the input and output of the workflow but the main purpose of the workflow\n    _For example, not \"✍ Convert Knowledge-piece to title\" but \"✍ Title\"_\n\n## The workflow\n\n> {book}\n```\n\n`-> {title}`\n"}],sourceFile:"./books/prepare-title.book"}];
 
 /**
@@ -8497,7 +8520,23 @@ function createShowdownConverter() {
     });
 }
 
-// TODO: [🏳‍🌈] Finally take pick of .json vs .ts
+/**
+ * Loads `jsdom` on demand
+ *
+ * Note: [🐌] `jsdom` is by far the heaviest dependency of Promptbook, loading it eagerly would slow down every single
+ *       run of the `ptbk` CLI utility even when no website is scraped
+ *
+ * @private internal utility of `WebsiteScraper`
+ */
+const loadJsdomModule = createLazyModuleLoader(() => import('jsdom'));
+/**
+ * Loads `@mozilla/readability` on demand
+ *
+ * Note: [🐌] Loaded lazily to keep the startup of the `ptbk` CLI utility fast
+ *
+ * @private internal utility of `WebsiteScraper`
+ */
+const loadReadabilityModule = createLazyModuleLoader(() => import('@mozilla/readability'));
 /**
  * Scraper for websites
  *
@@ -8533,6 +8572,7 @@ class WebsiteScraper {
         if (this.tools.fs === undefined) {
             throw new EnvironmentMismatchError('Can not scrape websites without filesystem tools');
         }
+        const [{ JSDOM }, { Readability }] = await Promise.all([loadJsdomModule(), loadReadabilityModule()]);
         const jsdom = new JSDOM(await source.asText(), {
             url: source.url,
         });