@promptbook/core 0.101.0-0 → 0.101.0-1

package/README.md

@@ -48,9 +48,246 @@ npm i ptbk
 npm install @promptbook/core
 ```
 
-Core of the library, it contains the main logic for promptbooks.
-
-
+The core package contains the fundamental logic and infrastructure for Promptbook. It provides the essential building blocks for creating, parsing, validating, and executing promptbooks, along with comprehensive error handling, LLM provider integrations, and execution utilities.
+
+## 🎯 Purpose and Motivation
+
+The core package serves as the foundation of the Promptbook ecosystem. It abstracts away the complexity of working with different LLM providers, provides a unified interface for prompt execution, and handles all the intricate details of pipeline management, parameter validation, and result processing.
+
+## 🔧 High-Level Functionality
+
+This package orchestrates the entire promptbook execution lifecycle:
+- **Pipeline Management**: Parse, validate, and compile promptbook definitions
+- **Execution Engine**: Create and manage pipeline executors with comprehensive error handling
+- **LLM Integration**: Unified interface for multiple LLM providers (OpenAI, Anthropic, Google, etc.)
+- **Parameter Processing**: Template parameter substitution and validation
+- **Knowledge Management**: Handle knowledge sources and scraping
+- **Storage Abstraction**: Flexible storage backends for caching and persistence
+- **Format Support**: Parse and validate various data formats (JSON, CSV, XML)
+
+## ✨ Key Features
+
+- 🚀 **Universal Pipeline Executor** - Execute promptbooks with any supported LLM provider
+- 🔄 **Multi-Provider Support** - Seamlessly switch between OpenAI, Anthropic, Google, and other providers
+- 📊 **Comprehensive Validation** - Validate promptbooks, parameters, and execution results
+- 🎯 **Expectation Checking** - Built-in validation for output format, length, and content expectations
+- 🧠 **Knowledge Integration** - Scrape and process knowledge from various sources
+- 💾 **Flexible Storage** - Memory, filesystem, and custom storage backends
+- 🔧 **Error Handling** - Detailed error types for debugging and monitoring
+- 📈 **Usage Tracking** - Monitor token usage, costs, and performance metrics
+- 🎨 **Format Parsers** - Support for JSON, CSV, XML, and text formats
+- 🔀 **Pipeline Migration** - Upgrade and migrate pipeline definitions
+
+## 📦 Exported Entities
+
+### Version Information
+- `BOOK_LANGUAGE_VERSION` - Current book language version
+- `PROMPTBOOK_ENGINE_VERSION` - Current engine version
+
+### Agent and Book Management
+- `createAgentModelRequirements` - Create model requirements for agents
+- `parseAgentSource` - Parse agent source code
+- `isValidBook` - Validate book format
+- `validateBook` - Comprehensive book validation
+- `DEFAULT_BOOK` - Default book template
+
+### Commitment System
+- `createEmptyAgentModelRequirements` - Create empty model requirements
+- `createBasicAgentModelRequirements` - Create basic model requirements
+- `NotYetImplementedCommitmentDefinition` - Placeholder for future commitments
+- `getCommitmentDefinition` - Get specific commitment definition
+- `getAllCommitmentDefinitions` - Get all available commitment definitions
+- `getAllCommitmentTypes` - Get all commitment types
+- `isCommitmentSupported` - Check if commitment is supported
+
+### Collection Management
+- `collectionToJson` - Convert collection to JSON
+- `createCollectionFromJson` - Create collection from JSON data
+- `createCollectionFromPromise` - Create collection from async source
+- `createCollectionFromUrl` - Create collection from URL
+- `createSubcollection` - Create filtered subcollection
+
+### Configuration Constants
+- `NAME` - Project name
+- `ADMIN_EMAIL` - Administrator email
+- `ADMIN_GITHUB_NAME` - GitHub username
+- `CLAIM` - Project claim/tagline
+- `DEFAULT_BOOK_TITLE` - Default book title
+- `DEFAULT_TASK_TITLE` - Default task title
+- `DEFAULT_PROMPT_TASK_TITLE` - Default prompt task title
+- `DEFAULT_BOOK_OUTPUT_PARAMETER_NAME` - Default output parameter name
+- `DEFAULT_MAX_FILE_SIZE` - Maximum file size limit
+- `BIG_DATASET_TRESHOLD` - Threshold for large datasets
+- `FAILED_VALUE_PLACEHOLDER` - Placeholder for failed values
+- `PENDING_VALUE_PLACEHOLDER` - Placeholder for pending values
+- `MAX_FILENAME_LENGTH` - Maximum filename length
+- `DEFAULT_INTERMEDIATE_FILES_STRATEGY` - Strategy for intermediate files
+- `DEFAULT_MAX_PARALLEL_COUNT` - Maximum parallel executions
+- `DEFAULT_MAX_EXECUTION_ATTEMPTS` - Maximum execution attempts
+- `DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_DEPTH` - Knowledge scraping depth limit
+- `DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_TOTAL` - Knowledge scraping total limit
+- `DEFAULT_BOOKS_DIRNAME` - Default books directory name
+- `DEFAULT_DOWNLOAD_CACHE_DIRNAME` - Default download cache directory
+- `DEFAULT_EXECUTION_CACHE_DIRNAME` - Default execution cache directory
+- `DEFAULT_SCRAPE_CACHE_DIRNAME` - Default scrape cache directory
+- `CLI_APP_ID` - CLI application identifier
+- `PLAYGROUND_APP_ID` - Playground application identifier
+- `DEFAULT_PIPELINE_COLLECTION_BASE_FILENAME` - Default collection filename
+- `DEFAULT_REMOTE_SERVER_URL` - Default remote server URL
+- `DEFAULT_CSV_SETTINGS` - Default CSV parsing settings
+- `DEFAULT_IS_VERBOSE` - Default verbosity setting
+- `SET_IS_VERBOSE` - Verbosity setter
+- `DEFAULT_IS_AUTO_INSTALLED` - Default auto-install setting
+- `DEFAULT_TASK_SIMULATED_DURATION_MS` - Default task simulation duration
+- `DEFAULT_GET_PIPELINE_COLLECTION_FUNCTION_NAME` - Default collection function name
+- `DEFAULT_MAX_REQUESTS_PER_MINUTE` - Rate limiting configuration
+- `API_REQUEST_TIMEOUT` - API request timeout
+- `PROMPTBOOK_LOGO_URL` - Official logo URL
+
+### Model and Provider Constants
+- `MODEL_TRUST_LEVELS` - Trust levels for different models
+- `MODEL_ORDERS` - Ordering preferences for models
+- `ORDER_OF_PIPELINE_JSON` - JSON property ordering
+- `RESERVED_PARAMETER_NAMES` - Reserved parameter names
+
+### Pipeline Processing
+- `compilePipeline` - Compile pipeline from source
+- `parsePipeline` - Parse pipeline definition
+- `pipelineJsonToString` - Convert pipeline JSON to string
+- `prettifyPipelineString` - Format pipeline string
+- `extractParameterNamesFromTask` - Extract parameter names
+- `validatePipeline` - Validate pipeline structure
+
+### Dialog and Interface Tools
+- `CallbackInterfaceTools` - Callback-based interface tools
+- `CallbackInterfaceToolsOptions` - Options for callback tools (type)
+
+### Error Handling
+- `BoilerplateError` - Base error class
+- `PROMPTBOOK_ERRORS` - All error types registry
+- `AbstractFormatError` - Abstract format validation error
+- `AuthenticationError` - Authentication failure error
+- `CollectionError` - Collection-related error
+- `EnvironmentMismatchError` - Environment compatibility error
+- `ExpectError` - Expectation validation error
+- `KnowledgeScrapeError` - Knowledge scraping error
+- `LimitReachedError` - Resource limit error
+- `MissingToolsError` - Missing tools error
+- `NotFoundError` - Resource not found error
+- `NotYetImplementedError` - Feature not implemented error
+- `ParseError` - Parsing error
+- `PipelineExecutionError` - Pipeline execution error
+- `PipelineLogicError` - Pipeline logic error
+- `PipelineUrlError` - Pipeline URL error
+- `PromptbookFetchError` - Fetch operation error
+- `UnexpectedError` - Unexpected error
+- `WrappedError` - Wrapped error container
+
+### Execution Engine
+- `createPipelineExecutor` - Create pipeline executor
+- `computeCosineSimilarity` - Compute cosine similarity for embeddings
+- `embeddingVectorToString` - Convert embedding vector to string
+- `executionReportJsonToString` - Convert execution report to string
+- `ExecutionReportStringOptions` - Report formatting options (type)
+- `ExecutionReportStringOptionsDefaults` - Default report options
+
+### Usage and Metrics
+- `addUsage` - Add usage metrics
+- `isPassingExpectations` - Check if expectations are met
+- `ZERO_VALUE` - Zero usage value constant
+- `UNCERTAIN_ZERO_VALUE` - Uncertain zero value constant
+- `ZERO_USAGE` - Zero usage object
+- `UNCERTAIN_USAGE` - Uncertain usage object
+- `usageToHuman` - Convert usage to human-readable format
+- `usageToWorktime` - Convert usage to work time estimate
+
+### Format Parsers
+- `CsvFormatError` - CSV format error
+- `CsvFormatParser` - CSV format parser
+- `MANDATORY_CSV_SETTINGS` - Required CSV settings
+- `TextFormatParser` - Text format parser
+
+### Form Factor Definitions
+- `BoilerplateFormfactorDefinition` - Boilerplate form factor
+- `ChatbotFormfactorDefinition` - Chatbot form factor
+- `CompletionFormfactorDefinition` - Completion form factor
+- `GeneratorFormfactorDefinition` - Generator form factor
+- `GenericFormfactorDefinition` - Generic form factor
+- `ImageGeneratorFormfactorDefinition` - Image generator form factor
+- `FORMFACTOR_DEFINITIONS` - All form factor definitions
+- `MatcherFormfactorDefinition` - Matcher form factor
+- `SheetsFormfactorDefinition` - Sheets form factor
+- `TranslatorFormfactorDefinition` - Translator form factor
+
+### LLM Provider Integration
+- `filterModels` - Filter available models
+- `$llmToolsMetadataRegister` - LLM tools metadata registry
+- `$llmToolsRegister` - LLM tools registry
+- `createLlmToolsFromConfiguration` - Create tools from config
+- `cacheLlmTools` - Cache LLM tools
+- `countUsage` - Count total usage
+- `limitTotalUsage` - Limit total usage
+- `joinLlmExecutionTools` - Join multiple LLM tools
+- `MultipleLlmExecutionTools` - Multiple LLM tools container
+
+### Provider Registrations
+- `_AnthropicClaudeMetadataRegistration` - Anthropic Claude registration
+- `_AzureOpenAiMetadataRegistration` - Azure OpenAI registration
+- `_DeepseekMetadataRegistration` - Deepseek registration
+- `_GoogleMetadataRegistration` - Google registration
+- `_OllamaMetadataRegistration` - Ollama registration
+- `_OpenAiMetadataRegistration` - OpenAI registration
+- `_OpenAiAssistantMetadataRegistration` - OpenAI Assistant registration
+- `_OpenAiCompatibleMetadataRegistration` - OpenAI Compatible registration
+
+### Pipeline Management
+- `migratePipeline` - Migrate pipeline to newer version
+- `preparePersona` - Prepare persona for execution
+- `book` - Book notation utilities
+- `isValidPipelineString` - Validate pipeline string
+- `GENERIC_PIPELINE_INTERFACE` - Generic pipeline interface
+- `getPipelineInterface` - Get pipeline interface
+- `isPipelineImplementingInterface` - Check interface implementation
+- `isPipelineInterfacesEqual` - Compare pipeline interfaces
+- `EXPECTATION_UNITS` - Units for expectations
+- `validatePipelineString` - Validate pipeline string format
+
+### Pipeline Preparation
+- `isPipelinePrepared` - Check if pipeline is prepared
+- `preparePipeline` - Prepare pipeline for execution
+- `unpreparePipeline` - Unprepare pipeline
+
+### Remote Server Integration
+- `identificationToPromptbookToken` - Convert ID to token
+- `promptbookTokenToIdentification` - Convert token to ID
+
+### Knowledge Scraping
+- `_BoilerplateScraperMetadataRegistration` - Boilerplate scraper registration
+- `prepareKnowledgePieces` - Prepare knowledge pieces
+- `$scrapersMetadataRegister` - Scrapers metadata registry
+- `$scrapersRegister` - Scrapers registry
+- `makeKnowledgeSourceHandler` - Create knowledge source handler
+- `promptbookFetch` - Fetch with promptbook context
+- `_LegacyDocumentScraperMetadataRegistration` - Legacy document scraper
+- `_DocumentScraperMetadataRegistration` - Document scraper registration
+- `_MarkdownScraperMetadataRegistration` - Markdown scraper registration
+- `_MarkitdownScraperMetadataRegistration` - Markitdown scraper registration
+- `_PdfScraperMetadataRegistration` - PDF scraper registration
+- `_WebsiteScraperMetadataRegistration` - Website scraper registration
+
+### Storage Backends
+- `BlackholeStorage` - Blackhole storage (discards data)
+- `MemoryStorage` - In-memory storage
+- `PrefixStorage` - Prefixed storage wrapper
+
+### Type Definitions
+- `MODEL_VARIANTS` - Available model variants
+- `NonTaskSectionTypes` - Non-task section types
+- `SectionTypes` - All section types
+- `TaskTypes` - Task types
+
+### Server Configuration
+- `REMOTE_SERVER_URLS` - Remote server URLs
 
 > 💡 This package does not make sense on its own, look at [all promptbook packages](#-packages) or just install all by `npm i ptbk`
 

package/esm/index.es.js

@@ -1,7 +1,4 @@
 import spaceTrim$1, { spaceTrim } from 'spacetrim';
-import parserHtml from 'prettier/parser-html';
-import parserMarkdown from 'prettier/parser-markdown';
-import { format } from 'prettier/standalone';
 import { randomBytes } from 'crypto';
 import { Subject } from 'rxjs';
 import { forTime } from 'waitasecond';
@@ -28,7 +25,7 @@ const BOOK_LANGUAGE_VERSION = '1.0.0';
  * @generated
  * @see https://github.com/webgptorg/promptbook
  */
-const PROMPTBOOK_ENGINE_VERSION = '0.101.0-0';
+const PROMPTBOOK_ENGINE_VERSION = '0.101.0-1';
 /**
  * TODO: string_promptbook_version should be constrained to the all versions of Promptbook engine
  * Note: [💞] Ignore a discrepancy between file name and entity name
@@ -1895,6 +1892,7 @@ function isValidFilePath(filename) {
 /**
  * Tests if given string is valid URL.
  *
+ * Note: [🔂] This function is idempotent.
  * Note: Dataurl are considered perfectly valid.
  * Note: There are two similar functions:
  * - `isValidUrl` which tests any URL
@@ -2481,8 +2479,18 @@ function validatePipelineString(pipelineString) {
  * @private withing the package because of HUGE size of prettier dependency
  */
 function prettifyMarkdown(content) {
+    // In browser/Next.js environments, just return the original content
+    // since prettier parsers are not available and would cause bundling issues
+    if (typeof window !== 'undefined') {
+        return content;
+    }
     try {
-        return format(content, {
+        // Use dynamic require to avoid static imports that cause bundling issues
+        // This will only work in Node.js environments
+        const prettierStandalone = eval('require')('prettier/standalone');
+        const parserMarkdown = eval('require')('prettier/parser-markdown');
+        const parserHtml = eval('require')('prettier/parser-html');
+        return prettierStandalone.format(content, {
             parser: 'markdown',
             plugins: [parserMarkdown, parserHtml],
             // TODO: DRY - make some import or auto-copy of .prettierrc
@@ -2510,6 +2518,8 @@ function prettifyMarkdown(content) {
 /**
  * Makes first letter of a string uppercase
  *
+ * Note: [🔂] This function is idempotent.
+ *
  * @public exported from `@promptbook/utils`
  */
 function capitalize(word) {
@@ -2812,6 +2822,7 @@ function checkSerializableAsJson(options) {
 /**
  * Creates a deep clone of the given object
  *
+ * Note: [🔂] This function is idempotent.
  * Note: This method only works for objects that are fully serializable to JSON and do not contain functions, Dates, or special types.
  *
  * @param objectValue The object to clone.
@@ -5863,6 +5874,8 @@ for (let i = 0; i < defaultDiacriticsRemovalMap.length; i++) {
 /**
  * Removes diacritic marks (accents) from characters in a string.
  *
+ * Note: [🔂] This function is idempotent.
+ *
  * @param input The string containing diacritics to be normalized.
  * @returns The string with diacritics removed or normalized.
  * @public exported from `@promptbook/utils`
@@ -7414,6 +7427,8 @@ function $getGlobalScope() {
 /**
  * Normalizes a text string to SCREAMING_CASE (all uppercase with underscores).
  *
+ * Note: [🔂] This function is idempotent.
+ *
  * @param text The text string to be converted to SCREAMING_CASE format.
  * @returns The normalized text in SCREAMING_CASE format.
  * @example 'HELLO_WORLD'
@@ -7469,6 +7484,8 @@ function normalizeTo_SCREAMING_CASE(text) {
 /**
  * Normalizes a text string to snake_case format.
  *
+ * Note: [🔂] This function is idempotent.
+ *
  * @param text The text string to be converted to snake_case format.
  * @returns The normalized text in snake_case format.
  * @example 'hello_world'
@@ -7785,6 +7802,8 @@ function mimeTypeToExtension(value) {
 /**
  * Removes emojis from a string and fix whitespaces
  *
+ * Note: [🔂] This function is idempotent.
+ *
  * @param text with emojis
  * @returns text without emojis
  * @public exported from `@promptbook/utils`
@@ -8874,6 +8893,7 @@ const EXPECTATION_UNITS = ['CHARACTERS', 'WORDS', 'SENTENCES', 'LINES', 'PARAGRA
 /**
  * Function parseNumber will parse number from string
  *
+ * Note: [🔂] This function is idempotent.
  * Unlike Number.parseInt, Number.parseFloat it will never ever result in NaN
  * Note: it also works only with decimal numbers
  *
@@ -9136,6 +9156,7 @@ function normalizeTo_camelCase(text, _isFirstLetterCapital = false) {
 /**
  * Removes quotes from a string
  *
+ * Note: [🔂] This function is idempotent.
  * Tip: This is very useful for post-processing of the result of the LLM model
  * Note: This function removes only the same quotes from the beginning and the end of the string
  * Note: There are two similar functions:
@@ -12174,6 +12195,8 @@ function createMarkdownChart(options) {
  * Function escapeMarkdownBlock will escape markdown block if needed
  * It is useful when you want have block in block
  *
+ * Note: [🔂] This function is idempotent.
+ *
  * @public exported from `@promptbook/markdown-utils`
  */
 function escapeMarkdownBlock(value) {