@promptbook/website-crawler 0.100.0-8 → 0.100.0

package/README.md

@@ -10,14 +10,18 @@ Write AI applications using plain human language across multiple models and plat
 [![NPM Version of ![Promptbook logo - cube with letters P and B](./design/logo-h1.png) Promptbook](https://badge.fury.io/js/promptbook.svg)](https://www.npmjs.com/package/promptbook)
 [![Quality of package ![Promptbook logo - cube with letters P and B](./design/logo-h1.png) Promptbook](https://packagequality.com/shield/promptbook.svg)](https://packagequality.com/#?package=promptbook)
 [![Known Vulnerabilities](https://snyk.io/test/github/webgptorg/promptbook/badge.svg)](https://snyk.io/test/github/webgptorg/promptbook)
-[![Build Status](https://github.com/webgptorg/promptbook/actions/workflows/ci.yml/badge.svg)](https://github.com/webgptorg/promptbook/actions)
-[![Coverage Status](https://coveralls.io/repos/github/webgptorg/promptbook/badge.svg?branch=main)](https://coveralls.io/github/webgptorg/promptbook?branch=main)
+[![🧪 Test Books](https://github.com/webgptorg/promptbook/actions/workflows/test-books.yml/badge.svg)](https://github.com/webgptorg/promptbook/actions/workflows/test-books.yml)
+[![🧪 Test build](https://github.com/webgptorg/promptbook/actions/workflows/test-build.yml/badge.svg)](https://github.com/webgptorg/promptbook/actions/workflows/test-build.yml)
+[![🧪 Lint](https://github.com/webgptorg/promptbook/actions/workflows/test-lint.yml/badge.svg)](https://github.com/webgptorg/promptbook/actions/workflows/test-lint.yml)
+[![🧪 Spell check](https://github.com/webgptorg/promptbook/actions/workflows/test-spell-check.yml/badge.svg)](https://github.com/webgptorg/promptbook/actions/workflows/test-spell-check.yml)
+[![🧪 Test types](https://github.com/webgptorg/promptbook/actions/workflows/test-types.yml/badge.svg)](https://github.com/webgptorg/promptbook/actions/workflows/test-types.yml)
 [![Issues](https://img.shields.io/github/issues/webgptorg/promptbook.svg?style=flat)](https://github.com/webgptorg/promptbook/issues)
 
 
 
 ## 🌟 New Features
 
+-   🚀 **GPT-5 Support** - Now includes OpenAI's most advanced language model with unprecedented reasoning capabilities and 200K context window
 -   💡 VS Code support for `.book` files with syntax highlighting and IntelliSense
 -   🐳 Official Docker image (`hejny/promptbook`) for seamless containerized usage
 -   🔥 Native support for OpenAI `o3-mini`, GPT-4 and other leading LLMs
@@ -25,10 +29,6 @@ Write AI applications using plain human language across multiple models and plat
 
 
 
-<blockquote style="color: #ff8811">
-    <b>⚠ Warning:</b> This is a pre-release version of the library. It is not yet ready for production use. Please look at <a href="https://www.npmjs.com/package/@promptbook/core?activeTab=versions">latest stable release</a>.
-</blockquote>
-
 ## 📦 Package `@promptbook/website-crawler`
 
 - Promptbooks are [divided into several](#-packages) packages, all are published from [single monorepo](https://github.com/webgptorg/promptbook).
@@ -60,8 +60,6 @@ Rest of the documentation is common for **entire promptbook ecosystem**:
 
 During the computer revolution, we have seen [multiple generations of computer languages](https://github.com/webgptorg/promptbook/discussions/180), from the physical rewiring of the vacuum tubes through low-level machine code to the high-level languages like Python or JavaScript. And now, we're on the edge of the **next revolution**!
 
-
-
 It's a revolution of writing software in **plain human language** that is understandable and executable by both humans and machines – and it's going to change everything!
 
 The incredible growth in power of microprocessors and the Moore's Law have been the driving force behind the ever-more powerful languages, and it's been an amazing journey! Similarly, the large language models (like GPT or Claude) are the next big thing in language technology, and they're set to transform the way we interact with computers.
@@ -187,8 +185,6 @@ Join our growing community of developers and users:
 
 _A concise, Markdown-based DSL for crafting AI workflows and automations._
 
-
-
 ### Introduction
 
 Book is a Markdown-based language that simplifies the creation of AI applications, workflows, and automations. With human-readable commands, you can define inputs, outputs, personas, knowledge sources, and actions—without needing model-specific details.
@@ -238,8 +234,6 @@ Personas can have access to different knowledge, tools and actions. They can als
 
 -   [PERSONA](https://github.com/webgptorg/promptbook/blob/main/documents/commands/PERSONA.md)
 
-
-
 ### **3. How:** Knowledge, Instruments and Actions
 
 The resources used by the personas are used to do the work.
@@ -314,6 +308,7 @@ Or you can install them separately:
 -   **[@promptbook/editable](https://www.npmjs.com/package/@promptbook/editable)** - Editable book as native javascript object with imperative object API
 -   **[@promptbook/templates](https://www.npmjs.com/package/@promptbook/templates)** - Useful templates and examples of books which can be used as a starting point
 -   **[@promptbook/types](https://www.npmjs.com/package/@promptbook/types)** - Just typescript types used in the library
+-   **[@promptbook/color](https://www.npmjs.com/package/@promptbook/color)** - Color manipulation library
 -   ⭐ **[@promptbook/cli](https://www.npmjs.com/package/@promptbook/cli)** - Command line interface utilities for promptbooks
 -   🐋 **[Docker image](https://hub.docker.com/r/hejny/promptbook/)** - Promptbook server
 
@@ -339,8 +334,6 @@ The following glossary is used to clarify certain concepts:
 
 _Note: This section is not a complete dictionary, more list of general AI / LLM terms that has connection with Promptbook_
 
-
-
 ### 💯 Core concepts
 
 -   [📚 Collection of pipelines](https://github.com/webgptorg/promptbook/discussions/65)

package/esm/index.es.js

@@ -5,8 +5,9 @@ import { SHA256 } from 'crypto-js';
 import hexEncoder from 'crypto-js/enc-hex';
 import { mkdir, rm } from 'fs/promises';
 import { basename, join, dirname } from 'path';
-import { format } from 'prettier';
 import parserHtml from 'prettier/parser-html';
+import parserMarkdown from 'prettier/parser-markdown';
+import { format } from 'prettier/standalone';
 import { randomBytes } from 'crypto';
 import { Subject } from 'rxjs';
 import { forTime } from 'waitasecond';
@@ -29,7 +30,7 @@ const BOOK_LANGUAGE_VERSION = '1.0.0';
  * @generated
  * @see https://github.com/webgptorg/promptbook
  */
-const PROMPTBOOK_ENGINE_VERSION = '0.100.0-8';
+const PROMPTBOOK_ENGINE_VERSION = '0.100.0';
 /**
  * TODO: string_promptbook_version should be constrained to the all versions of Promptbook engine
  * Note: [💞] Ignore a discrepancy between file name and entity name
@@ -248,6 +249,19 @@ let DEFAULT_IS_VERBOSE = false;
  * @public exported from `@promptbook/core`
  */
 const DEFAULT_IS_AUTO_INSTALLED = false;
+/**
+ * Default simulated duration for a task in milliseconds (used for progress reporting)
+ *
+ * @public exported from `@promptbook/core`
+ */
+const DEFAULT_TASK_SIMULATED_DURATION_MS = 5 * 60 * 1000; // 5 minutes
+/**
+ * API request timeout in milliseconds
+ * Can be overridden via API_REQUEST_TIMEOUT environment variable
+ *
+ * @public exported from `@promptbook/core`
+ */
+parseInt(process.env.API_REQUEST_TIMEOUT || '90000');
 /**
  * Indicates whether pipeline logic validation is enabled. When true, the pipeline logic is checked for consistency.
  *
@@ -1200,7 +1214,7 @@ function prettifyMarkdown(content) {
     try {
         return format(content, {
             parser: 'markdown',
-            plugins: [parserHtml],
+            plugins: [parserMarkdown, parserHtml],
             // TODO: DRY - make some import or auto-copy of .prettierrc
             endOfLine: 'lf',
             tabWidth: 4,
@@ -2652,6 +2666,7 @@ function createTask(options) {
     const errors = [];
     const warnings = [];
     let currentValue = {};
+    let customTldr = null;
     const partialResultSubject = new Subject();
     // <- Note: Not using `BehaviorSubject` because on error we can't access the last value
     const finalResultPromise = /* not await */ taskProcessCallback((newOngoingResult) => {
@@ -2662,6 +2677,9 @@ function createTask(options) {
         Object.assign(currentValue, newOngoingResult);
         // <- TODO: assign deep
         partialResultSubject.next(newOngoingResult);
+    }, (tldrInfo) => {
+        customTldr = tldrInfo;
+        updatedAt = new Date();
     });
     finalResultPromise
         .catch((error) => {
@@ -2715,6 +2733,78 @@ function createTask(options) {
             return status;
             // <- Note: [1] --||--
         },
+        get tldr() {
+            var _a, _b, _c, _d, _e, _f, _g, _h, _j, _k;
+            // Use custom tldr if available
+            if (customTldr) {
+                return customTldr;
+            }
+            // Fallback to default implementation
+            const cv = currentValue;
+            // If explicit percent is provided, use it
+            let percentRaw = (_f = (_d = (_b = (_a = cv === null || cv === void 0 ? void 0 : cv.tldr) === null || _a === void 0 ? void 0 : _a.percent) !== null && _b !== void 0 ? _b : (_c = cv === null || cv === void 0 ? void 0 : cv.usage) === null || _c === void 0 ? void 0 : _c.percent) !== null && _d !== void 0 ? _d : (_e = cv === null || cv === void 0 ? void 0 : cv.progress) === null || _e === void 0 ? void 0 : _e.percent) !== null && _f !== void 0 ? _f : cv === null || cv === void 0 ? void 0 : cv.percent;
+            // Simulate progress if not provided
+            if (typeof percentRaw !== 'number') {
+                // Simulate progress: evenly split across subtasks, based on elapsed time
+                const now = new Date();
+                const elapsedMs = now.getTime() - createdAt.getTime();
+                const totalMs = DEFAULT_TASK_SIMULATED_DURATION_MS;
+                // If subtasks are defined, split progress evenly
+                const subtaskCount = Array.isArray(cv === null || cv === void 0 ? void 0 : cv.subtasks) ? cv.subtasks.length : 1;
+                const completedSubtasks = Array.isArray(cv === null || cv === void 0 ? void 0 : cv.subtasks)
+                    ? cv.subtasks.filter((s) => s.done || s.completed).length
+                    : 0;
+                // Progress from completed subtasks
+                const subtaskProgress = subtaskCount > 0 ? completedSubtasks / subtaskCount : 0;
+                // Progress from elapsed time for current subtask
+                const timeProgress = Math.min(elapsedMs / totalMs, 1);
+                // Combine: completed subtasks + time progress for current subtask
+                percentRaw = Math.min(subtaskProgress + (1 / subtaskCount) * timeProgress, 1);
+                if (status === 'FINISHED')
+                    percentRaw = 1;
+                if (status === 'ERROR')
+                    percentRaw = 0;
+            }
+            // Clamp to [0,1]
+            let percent = Number(percentRaw) || 0;
+            if (percent < 0)
+                percent = 0;
+            if (percent > 1)
+                percent = 1;
+            // Build a short message: prefer explicit tldr.message, then common summary/message fields, then errors/warnings, then status
+            const messageFromResult = (_k = (_j = (_h = (_g = cv === null || cv === void 0 ? void 0 : cv.tldr) === null || _g === void 0 ? void 0 : _g.message) !== null && _h !== void 0 ? _h : cv === null || cv === void 0 ? void 0 : cv.message) !== null && _j !== void 0 ? _j : cv === null || cv === void 0 ? void 0 : cv.summary) !== null && _k !== void 0 ? _k : cv === null || cv === void 0 ? void 0 : cv.statusMessage;
+            let message = messageFromResult;
+            if (!message) {
+                // If subtasks, show current subtask
+                if (Array.isArray(cv === null || cv === void 0 ? void 0 : cv.subtasks) && cv.subtasks.length > 0) {
+                    const current = cv.subtasks.find((s) => !s.done && !s.completed);
+                    if (current && current.title) {
+                        message = `Working on ${current.title}`;
+                    }
+                }
+                if (!message) {
+                    if (errors.length) {
+                        message = errors[errors.length - 1].message || 'Error';
+                    }
+                    else if (warnings.length) {
+                        message = warnings[warnings.length - 1].message || 'Warning';
+                    }
+                    else if (status === 'FINISHED') {
+                        message = 'Finished';
+                    }
+                    else if (status === 'ERROR') {
+                        message = 'Error';
+                    }
+                    else {
+                        message = 'Running';
+                    }
+                }
+            }
+            return {
+                percent: percent,
+                message,
+            };
+        },
         get createdAt() {
             return createdAt;
             // <- Note: [1] --||--
@@ -2963,6 +3053,76 @@ function countUsage(llmTools) {
  * TODO: [👷‍♂️] @@@ Manual about construction of llmTools
  */
 
+/**
+ * Predefined profiles for LLM providers to maintain consistency across the application
+ * These profiles represent each provider as a virtual persona in chat interfaces
+ *
+ * @private !!!!
+ */
+const LLM_PROVIDER_PROFILES = {
+    OPENAI: {
+        name: 'OPENAI',
+        fullname: 'OpenAI GPT',
+        color: '#10a37f', // OpenAI's signature green
+        // Note: avatarSrc could be added when we have provider logos available
+    },
+    ANTHROPIC: {
+        name: 'ANTHROPIC',
+        fullname: 'Anthropic Claude',
+        color: '#d97706', // Anthropic's orange/amber color
+    },
+    AZURE_OPENAI: {
+        name: 'AZURE_OPENAI',
+        fullname: 'Azure OpenAI',
+        color: '#0078d4', // Microsoft Azure blue
+    },
+    GOOGLE: {
+        name: 'GOOGLE',
+        fullname: 'Google Gemini',
+        color: '#4285f4', // Google blue
+    },
+    DEEPSEEK: {
+        name: 'DEEPSEEK',
+        fullname: 'DeepSeek',
+        color: '#7c3aed', // Purple color for DeepSeek
+    },
+    OLLAMA: {
+        name: 'OLLAMA',
+        fullname: 'Ollama',
+        color: '#059669', // Emerald green for local models
+    },
+    REMOTE: {
+        name: 'REMOTE',
+        fullname: 'Remote Server',
+        color: '#6b7280', // Gray for remote/proxy connections
+    },
+    MOCKED_ECHO: {
+        name: 'MOCKED_ECHO',
+        fullname: 'Echo (Test)',
+        color: '#8b5cf6', // Purple for test/mock tools
+    },
+    MOCKED_FAKE: {
+        name: 'MOCKED_FAKE',
+        fullname: 'Fake LLM (Test)',
+        color: '#ec4899', // Pink for fake/test tools
+    },
+    VERCEL: {
+        name: 'VERCEL',
+        fullname: 'Vercel AI',
+        color: '#000000', // Vercel's black
+    },
+    MULTIPLE: {
+        name: 'MULTIPLE',
+        fullname: 'Multiple Providers',
+        color: '#6366f1', // Indigo for combined/multiple providers
+    },
+};
+/**
+ * TODO: Refactor this - each profile must be alongside the provider definition
+ * TODO: Unite `AvatarProfileProps` and `ChatParticipant`
+ * Note: [💞] Ignore a discrepancy between file name and entity name
+ */
+
 /**
  * Multiple LLM Execution Tools is a proxy server that uses multiple execution tools internally and exposes the executor interface externally.
  *
@@ -2988,7 +3148,7 @@ class MultipleLlmExecutionTools {
             }
             return spaceTrim$1((block) => `
                         ${headLine}
-                        
+
                           ${ /* <- Note: Indenting the description: */block(description)}
                     `);
         })
@@ -2999,6 +3159,9 @@ class MultipleLlmExecutionTools {
                 ${block(innerModelsTitlesAndDescriptions)}
             `);
     }
+    get profile() {
+        return LLM_PROVIDER_PROFILES.MULTIPLE;
+    }
     /**
      * Check the configuration of all execution tools
      */
@@ -3043,25 +3206,22 @@ class MultipleLlmExecutionTools {
         const errors = [];
         llm: for (const llmExecutionTools of this.llmExecutionTools) {
             try {
-                variant: switch (prompt.modelRequirements.modelVariant) {
+                switch (prompt.modelRequirements.modelVariant) {
                     case 'CHAT':
                         if (llmExecutionTools.callChatModel === undefined) {
                             continue llm;
                         }
                         return await llmExecutionTools.callChatModel(prompt);
-                        break variant;
                     case 'COMPLETION':
                         if (llmExecutionTools.callCompletionModel === undefined) {
                             continue llm;
                         }
                         return await llmExecutionTools.callCompletionModel(prompt);
-                        break variant;
                     case 'EMBEDDING':
                         if (llmExecutionTools.callEmbeddingModel === undefined) {
                             continue llm;
                         }
                         return await llmExecutionTools.callEmbeddingModel(prompt);
-                        break variant;
                     // <- case [🤖]:
                     default:
                         throw new UnexpectedError(`Unknown model variant "${prompt.modelRequirements.modelVariant}"`);
@@ -3214,7 +3374,7 @@ async function preparePersona(personaDescription, tools, options) {
     const result = await preparePersonaExecutor({
         availableModels /* <- Note: Passing as JSON */,
         personaDescription,
-    }).asPromise();
+    }).asPromise({ isCrashedOnError: true });
     const { outputParameters } = result;
     const { modelsRequirements: modelsRequirementsJson } = outputParameters;
     let modelsRequirementsUnchecked = jsonParse(modelsRequirementsJson);
@@ -3863,7 +4023,7 @@ async function preparePipeline(pipeline, tools, options) {
         });
         const result = await prepareTitleExecutor({
             book: sources.map(({ content }) => content).join('\n\n'),
-        }).asPromise();
+        }).asPromise({ isCrashedOnError: true });
         const { outputParameters } = result;
         const { title: titleRaw } = outputParameters;
         if (isVerbose) {
@@ -4957,7 +5117,7 @@ function validatePromptResult(options) {
  */
 async function executeAttempts(options) {
     const { jokerParameterNames, priority, maxAttempts, // <- Note: [💂]
-    preparedContent, parameters, task, preparedPipeline, tools, $executionReport, pipelineIdentification, maxExecutionAttempts, } = options;
+    preparedContent, parameters, task, preparedPipeline, tools, $executionReport, pipelineIdentification, maxExecutionAttempts, onProgress, } = options;
     const $ongoingTaskResult = {
         $result: null,
         $resultString: null,
@@ -5201,6 +5361,10 @@ async function executeAttempts(options) {
                 result: $ongoingTaskResult.$resultString,
                 error: error,
             });
+            // Report failed attempt
+            onProgress({
+                errors: [error],
+            });
         }
         finally {
             if (!isJokerAttempt &&
@@ -6074,15 +6238,74 @@ function createPipelineExecutor(options) {
             });
         });
     };
-    const pipelineExecutor = (inputParameters) => createTask({
-        taskType: 'EXECUTION',
-        title: pipeline.title,
-        taskProcessCallback(updateOngoingResult) {
-            return pipelineExecutorWithCallback(inputParameters, async (newOngoingResult) => {
-                updateOngoingResult(newOngoingResult);
-            });
-        },
-    });
+    const pipelineExecutor = (inputParameters) => {
+        const startTime = new Date().getTime();
+        return createTask({
+            taskType: 'EXECUTION',
+            title: pipeline.title,
+            taskProcessCallback(updateOngoingResult, updateTldr) {
+                return pipelineExecutorWithCallback(inputParameters, async (newOngoingResult) => {
+                    var _a, _b;
+                    updateOngoingResult(newOngoingResult);
+                    // Calculate and update tldr based on pipeline progress
+                    const cv = newOngoingResult;
+                    // Calculate progress based on parameters resolved vs total parameters
+                    const totalParameters = pipeline.parameters.filter(p => !p.isInput).length;
+                    let resolvedParameters = 0;
+                    let currentTaskTitle = '';
+                    // Get the resolved parameters from output parameters
+                    if (cv === null || cv === void 0 ? void 0 : cv.outputParameters) {
+                        // Count how many output parameters have non-empty values
+                        resolvedParameters = Object.values(cv.outputParameters).filter(value => value !== undefined && value !== null && String(value).trim() !== '').length;
+                    }
+                    // Try to determine current task from execution report
+                    if (((_a = cv === null || cv === void 0 ? void 0 : cv.executionReport) === null || _a === void 0 ? void 0 : _a.promptExecutions) && cv.executionReport.promptExecutions.length > 0) {
+                        const lastExecution = cv.executionReport.promptExecutions[cv.executionReport.promptExecutions.length - 1];
+                        if ((_b = lastExecution === null || lastExecution === void 0 ? void 0 : lastExecution.prompt) === null || _b === void 0 ? void 0 : _b.title) {
+                            currentTaskTitle = lastExecution.prompt.title;
+                        }
+                    }
+                    // Calculate base progress percentage
+                    let percent = totalParameters > 0 ? resolvedParameters / totalParameters : 0;
+                    // Add time-based progress for current task if we haven't completed all parameters
+                    if (resolvedParameters < totalParameters) {
+                        const elapsedMs = new Date().getTime() - startTime;
+                        const estimatedTotalMs = totalParameters * 30 * 1000; // Estimate 30 seconds per parameter
+                        const timeProgress = Math.min(elapsedMs / estimatedTotalMs, 0.9); // Cap at 90% for time-based progress
+                        // If we have time progress but no parameter progress, show time progress
+                        if (percent === 0 && timeProgress > 0) {
+                            percent = Math.min(timeProgress, 0.1); // Show some progress but not more than 10%
+                        }
+                        else if (percent < 1) {
+                            // Add partial progress for current task
+                            const taskProgress = totalParameters > 0 ? (1 / totalParameters) * 0.5 : 0; // 50% of task progress
+                            percent = Math.min(percent + taskProgress, 0.95); // Cap at 95% until fully complete
+                        }
+                    }
+                    // Clamp to [0,1]
+                    percent = Math.min(Math.max(percent, 0), 1);
+                    // Generate message
+                    let message = '';
+                    if (currentTaskTitle) {
+                        message = `Executing: ${currentTaskTitle}`;
+                    }
+                    else if (resolvedParameters === 0) {
+                        message = 'Starting pipeline execution';
+                    }
+                    else if (resolvedParameters < totalParameters) {
+                        message = `Processing pipeline (${resolvedParameters}/${totalParameters} parameters resolved)`;
+                    }
+                    else {
+                        message = 'Completing pipeline execution';
+                    }
+                    updateTldr({
+                        percent: percent,
+                        message,
+                    });
+                });
+            },
+        });
+    };
     //        <- TODO: Make types such as there is no need to do `as` for `createTask`
     return pipelineExecutor;
 }
@@ -6167,7 +6390,9 @@ class MarkdownScraper {
             },
         });
         const knowledgeContent = await source.asText();
-        const result = await prepareKnowledgeFromMarkdownExecutor({ knowledgeContent }).asPromise();
+        const result = await prepareKnowledgeFromMarkdownExecutor({ knowledgeContent }).asPromise({
+            isCrashedOnError: true,
+        });
         const { outputParameters } = result;
         const { knowledgePieces: knowledgePiecesRaw } = outputParameters;
         const knowledgeTextPieces = (knowledgePiecesRaw || '').split('\n---\n');
@@ -6191,12 +6416,16 @@ class MarkdownScraper {
           ];
           */
             try {
-                const titleResult = await prepareTitleExecutor({ knowledgePieceContent }).asPromise();
+                const titleResult = await prepareTitleExecutor({ knowledgePieceContent }).asPromise({
+                    isCrashedOnError: true,
+                });
                 const { title: titleRaw = 'Untitled' } = titleResult.outputParameters;
                 title = spaceTrim$1(titleRaw) /* <- TODO: Maybe do in pipeline */;
                 name = titleToName(title);
                 // --- Keywords
-                const keywordsResult = await prepareKeywordsExecutor({ knowledgePieceContent }).asPromise();
+                const keywordsResult = await prepareKeywordsExecutor({ knowledgePieceContent }).asPromise({
+                    isCrashedOnError: true,
+                });
                 const { keywords: keywordsRaw = '' } = keywordsResult.outputParameters;
                 keywords = (keywordsRaw || '')
                     .split(',')