@promptbook/core 0.88.0 → 0.89.0-11

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (52) hide show
  1. package/README.md +7 -1
  2. package/esm/index.es.js +197 -108
  3. package/esm/index.es.js.map +1 -1
  4. package/esm/typings/src/_packages/core.index.d.ts +18 -6
  5. package/esm/typings/src/_packages/remote-client.index.d.ts +6 -8
  6. package/esm/typings/src/_packages/remote-server.index.d.ts +6 -6
  7. package/esm/typings/src/_packages/types.index.d.ts +18 -10
  8. package/esm/typings/src/cli/cli-commands/login.d.ts +15 -0
  9. package/esm/typings/src/cli/common/$addGlobalOptionsToCommand.d.ts +7 -0
  10. package/esm/typings/src/cli/common/$provideLlmToolsForCli.d.ts +15 -0
  11. package/esm/typings/src/config.d.ts +15 -8
  12. package/esm/typings/src/errors/0-index.d.ts +6 -0
  13. package/esm/typings/src/errors/AuthenticationError.d.ts +9 -0
  14. package/esm/typings/src/errors/PromptbookFetchError.d.ts +9 -0
  15. package/esm/typings/src/execution/PipelineExecutorResult.d.ts +2 -2
  16. package/esm/typings/src/execution/PromptResult.d.ts +2 -2
  17. package/esm/typings/src/execution/{PromptResultUsage.d.ts → Usage.d.ts} +5 -5
  18. package/esm/typings/src/execution/utils/addUsage.d.ts +2 -2
  19. package/esm/typings/src/execution/utils/computeUsageCounts.d.ts +3 -3
  20. package/esm/typings/src/execution/utils/usage-constants.d.ts +77 -60
  21. package/esm/typings/src/execution/utils/usageToHuman.d.ts +5 -5
  22. package/esm/typings/src/execution/utils/usageToWorktime.d.ts +5 -5
  23. package/esm/typings/src/llm-providers/_common/register/$provideEnvFilename.d.ts +12 -0
  24. package/esm/typings/src/llm-providers/_common/register/$provideLlmToolsConfigurationFromEnv.d.ts +2 -8
  25. package/esm/typings/src/llm-providers/_common/register/$provideLlmToolsForTestingAndScriptsAndPlayground.d.ts +2 -0
  26. package/esm/typings/src/llm-providers/_common/register/$provideLlmToolsForWizzardOrCli.d.ts +36 -1
  27. package/esm/typings/src/llm-providers/_common/register/$provideLlmToolsFromEnv.d.ts +1 -0
  28. package/esm/typings/src/llm-providers/_common/utils/count-total-usage/LlmExecutionToolsWithTotalUsage.d.ts +9 -2
  29. package/esm/typings/src/llm-providers/_common/utils/count-total-usage/{countTotalUsage.d.ts → countUsage.d.ts} +1 -1
  30. package/esm/typings/src/llm-providers/_common/utils/count-total-usage/limitTotalUsage.d.ts +2 -2
  31. package/esm/typings/src/llm-providers/anthropic-claude/AnthropicClaudeExecutionToolsOptions.d.ts +1 -1
  32. package/esm/typings/src/llm-providers/anthropic-claude/computeAnthropicClaudeUsage.d.ts +2 -2
  33. package/esm/typings/src/llm-providers/anthropic-claude/register-configuration.d.ts +1 -1
  34. package/esm/typings/src/llm-providers/openai/OpenAiExecutionTools.d.ts +0 -9
  35. package/esm/typings/src/llm-providers/openai/computeOpenAiUsage.d.ts +2 -2
  36. package/esm/typings/src/pipeline/PipelineJson/PreparationJson.d.ts +2 -2
  37. package/esm/typings/src/playground/playground.d.ts +5 -0
  38. package/esm/typings/src/remote-server/RemoteServer.d.ts +23 -0
  39. package/esm/typings/src/remote-server/socket-types/_subtypes/{PromptbookServer_Identification.d.ts → Identification.d.ts} +5 -4
  40. package/esm/typings/src/remote-server/socket-types/listModels/PromptbookServer_ListModels_Request.d.ts +2 -2
  41. package/esm/typings/src/remote-server/socket-types/prepare/PromptbookServer_PreparePipeline_Request.d.ts +2 -2
  42. package/esm/typings/src/remote-server/socket-types/prompt/PromptbookServer_Prompt_Request.d.ts +2 -2
  43. package/esm/typings/src/remote-server/startRemoteServer.d.ts +2 -2
  44. package/esm/typings/src/remote-server/types/RemoteClientOptions.d.ts +4 -12
  45. package/esm/typings/src/remote-server/types/RemoteServerOptions.d.ts +88 -6
  46. package/esm/typings/src/scrapers/_common/utils/{scraperFetch.d.ts → promptbookFetch.d.ts} +2 -2
  47. package/esm/typings/src/storage/env-storage/$EnvStorage.d.ts +37 -0
  48. package/esm/typings/src/types/typeAliases.d.ts +8 -2
  49. package/esm/typings/src/utils/organization/TODO_narrow.d.ts +6 -0
  50. package/package.json +1 -1
  51. package/umd/index.umd.js +205 -110
  52. package/umd/index.umd.js.map +1 -1
package/README.md CHANGED
@@ -23,6 +23,10 @@
23
23
 
24
24
 
25
25
 
26
+ <blockquote style="color: #ff8811">
27
+ <b>⚠ Warning:</b> This is a pre-release version of the library. It is not yet ready for production use. Please look at <a href="https://www.npmjs.com/package/@promptbook/core?activeTab=versions">latest stable release</a>.
28
+ </blockquote>
29
+
26
30
  ## 📦 Package `@promptbook/core`
27
31
 
28
32
  - Promptbooks are [divided into several](#-packages) packages, all are published from [single monorepo](https://github.com/webgptorg/promptbook).
@@ -200,7 +204,7 @@ Each part of the book defines one of 3 circles:
200
204
 
201
205
  ### **What:** Workflows, Tasks and Parameters
202
206
 
203
- What work needs to be done. Each book defines a workflow, which is one or more tasks. Each workflow has a fixed input and output. For example, you have a book that generates an article from a topic. Once it generates an article about AI, once about marketing, once about cooking. The workflow (= your AI program) is the same, only the input and output change.
207
+ What work needs to be done. Each book defines a [workflow *(scenario or pipeline)*](https://github.com/webgptorg/promptbook/discussions/88), which is one or more tasks. Each workflow has a fixed input and output. For example, you have a book that generates an article from a topic. Once it generates an article about AI, once about marketing, once about cooking. The workflow (= your AI program) is the same, only the input and output change.
204
208
 
205
209
  **Related commands:**
206
210
 
@@ -310,6 +314,8 @@ The following glossary is used to clarify certain concepts:
310
314
 
311
315
 
312
316
 
317
+
318
+
313
319
  _Note: This section is not complete dictionary, more list of general AI / LLM terms that has connection with Promptbook_
314
320
 
315
321
 
package/esm/index.es.js CHANGED
@@ -27,7 +27,7 @@ const BOOK_LANGUAGE_VERSION = '1.0.0';
27
27
  * @generated
28
28
  * @see https://github.com/webgptorg/promptbook
29
29
  */
30
- const PROMPTBOOK_ENGINE_VERSION = '0.88.0';
30
+ const PROMPTBOOK_ENGINE_VERSION = '0.89.0-11';
31
31
  /**
32
32
  * TODO: string_promptbook_version should be constrained to the all versions of Promptbook engine
33
33
  * Note: [💞] Ignore a discrepancy between file name and entity name
@@ -572,6 +572,7 @@ const DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_TOTAL = 200;
572
572
  */
573
573
  const DEFAULT_BOOKS_DIRNAME = './books';
574
574
  // <- TODO: [🕝] Make also `BOOKS_DIRNAME_ALTERNATIVES`
575
+ // TODO: !!!!!! Just .promptbook dir, hardocode others
575
576
  /**
576
577
  * Where to store the temporary downloads
577
578
  *
@@ -596,6 +597,27 @@ const DEFAULT_EXECUTION_CACHE_DIRNAME = './.promptbook/execution-cache';
596
597
  * @public exported from `@promptbook/core`
597
598
  */
598
599
  const DEFAULT_SCRAPE_CACHE_DIRNAME = './.promptbook/scrape-cache';
600
+ /**
601
+ * Id of application for the CLI when using remote server
602
+ *
603
+ * @public exported from `@promptbook/core`
604
+ */
605
+ const CLI_APP_ID = 'cli';
606
+ /**
607
+ * Id of application for the playground
608
+ *
609
+ * @public exported from `@promptbook/core`
610
+ */
611
+ const PLAYGROUND_APP_ID = 'playground';
612
+ /*
613
+ TODO: [🌃]
614
+ /**
615
+ * Id of application for the wizzard when using remote server
616
+ *
617
+ * @public exported from `@promptbook/core`
618
+ * /
619
+ ex-port const WIZZARD_APP_ID: string_app_id = 'wizzard';
620
+ */
599
621
  /**
600
622
  * The name of the builded pipeline collection made by CLI `ptbk make` and for lookup in `createCollectionFromDirectory`
601
623
  *
@@ -616,13 +638,7 @@ const MOMENT_ARG_THRESHOLDS = {
616
638
  *
617
639
  * @public exported from `@promptbook/core`
618
640
  */
619
- const DEFAULT_REMOTE_URL = 'https://api.pavolhejny.com/';
620
- /**
621
- * @@@
622
- *
623
- * @public exported from `@promptbook/core`
624
- */
625
- const DEFAULT_REMOTE_URL_PATH = '/promptbook/socket.io';
641
+ const DEFAULT_REMOTE_SERVER_URL = 'https://api.pavolhejny.com/promptbook';
626
642
  // <- TODO: [🧜‍♂️]
627
643
  /**
628
644
  * @@@
@@ -674,7 +690,7 @@ const IS_PIPELINE_LOGIC_VALIDATED = just(
674
690
  true);
675
691
  /**
676
692
  * Note: [💞] Ignore a discrepancy between file name and entity name
677
- * TODO: [🧠][🧜‍♂️] Maybe join remoteUrl and path into single value
693
+ * TODO: [🧠][🧜‍♂️] Maybe join remoteServerUrl and path into single value
678
694
  */
679
695
 
680
696
  /**
@@ -1893,6 +1909,19 @@ class CsvFormatError extends AbstractFormatError {
1893
1909
  }
1894
1910
  }
1895
1911
 
1912
+ /**
1913
+ * AuthenticationError is thrown from login function which is dependency of remote server
1914
+ *
1915
+ * @public exported from `@promptbook/core`
1916
+ */
1917
+ class AuthenticationError extends Error {
1918
+ constructor(message) {
1919
+ super(message);
1920
+ this.name = 'AuthenticationError';
1921
+ Object.setPrototypeOf(this, AuthenticationError.prototype);
1922
+ }
1923
+ }
1924
+
1896
1925
  /**
1897
1926
  * This error indicates that the pipeline collection cannot be propperly loaded
1898
1927
  *
@@ -1984,6 +2013,19 @@ class NotYetImplementedError extends Error {
1984
2013
  }
1985
2014
  }
1986
2015
 
2016
+ /**
2017
+ * Error thrown when a fetch request fails
2018
+ *
2019
+ * @public exported from `@promptbook/core`
2020
+ */
2021
+ class PromptbookFetchError extends Error {
2022
+ constructor(message) {
2023
+ super(message);
2024
+ this.name = 'PromptbookFetchError';
2025
+ Object.setPrototypeOf(this, PromptbookFetchError.prototype);
2026
+ }
2027
+ }
2028
+
1987
2029
  /**
1988
2030
  * Index of all custom errors
1989
2031
  *
@@ -2021,6 +2063,8 @@ const COMMON_JAVASCRIPT_ERRORS = {
2021
2063
  TypeError,
2022
2064
  URIError,
2023
2065
  AggregateError,
2066
+ AuthenticationError,
2067
+ PromptbookFetchError,
2024
2068
  /*
2025
2069
  Note: Not widely supported
2026
2070
  > InternalError,
@@ -2082,6 +2126,7 @@ function assertsTaskSuccessful(executionResult) {
2082
2126
  const { isSuccessful, errors, warnings } = executionResult;
2083
2127
  for (const warning of warnings) {
2084
2128
  console.warn(warning.message);
2129
+ // <- TODO: [🏮] Some standard way how to transform errors into warnings and how to handle non-critical fails during the tasks
2085
2130
  }
2086
2131
  if (isSuccessful === true) {
2087
2132
  return;
@@ -2219,6 +2264,10 @@ function serializeError(error) {
2219
2264
 
2220
2265
  Cannot serialize error with name "${name}"
2221
2266
 
2267
+ Authors of Promptbook probably forgot to add this error into the list of errors:
2268
+ https://github.com/webgptorg/promptbook/blob/main/src/errors/0-index.ts
2269
+
2270
+
2222
2271
  ${block(stack || message)}
2223
2272
 
2224
2273
  `));
@@ -2306,30 +2355,42 @@ function valueToString(value) {
2306
2355
  }
2307
2356
  }
2308
2357
 
2358
+ /**
2359
+ * Represents the uncertain value
2360
+ *
2361
+ * @public exported from `@promptbook/core`
2362
+ */
2363
+ const ZERO_VALUE = $deepFreeze({ value: 0 });
2364
+ /**
2365
+ * Represents the uncertain value
2366
+ *
2367
+ * @public exported from `@promptbook/core`
2368
+ */
2369
+ const UNCERTAIN_ZERO_VALUE = $deepFreeze({ value: 0, isUncertain: true });
2309
2370
  /**
2310
2371
  * Represents the usage with no resources consumed
2311
2372
  *
2312
2373
  * @public exported from `@promptbook/core`
2313
2374
  */
2314
2375
  const ZERO_USAGE = $deepFreeze({
2315
- price: { value: 0 },
2376
+ price: ZERO_VALUE,
2316
2377
  input: {
2317
- tokensCount: { value: 0 },
2318
- charactersCount: { value: 0 },
2319
- wordsCount: { value: 0 },
2320
- sentencesCount: { value: 0 },
2321
- linesCount: { value: 0 },
2322
- paragraphsCount: { value: 0 },
2323
- pagesCount: { value: 0 },
2378
+ tokensCount: ZERO_VALUE,
2379
+ charactersCount: ZERO_VALUE,
2380
+ wordsCount: ZERO_VALUE,
2381
+ sentencesCount: ZERO_VALUE,
2382
+ linesCount: ZERO_VALUE,
2383
+ paragraphsCount: ZERO_VALUE,
2384
+ pagesCount: ZERO_VALUE,
2324
2385
  },
2325
2386
  output: {
2326
- tokensCount: { value: 0 },
2327
- charactersCount: { value: 0 },
2328
- wordsCount: { value: 0 },
2329
- sentencesCount: { value: 0 },
2330
- linesCount: { value: 0 },
2331
- paragraphsCount: { value: 0 },
2332
- pagesCount: { value: 0 },
2387
+ tokensCount: ZERO_VALUE,
2388
+ charactersCount: ZERO_VALUE,
2389
+ wordsCount: ZERO_VALUE,
2390
+ sentencesCount: ZERO_VALUE,
2391
+ linesCount: ZERO_VALUE,
2392
+ paragraphsCount: ZERO_VALUE,
2393
+ pagesCount: ZERO_VALUE,
2333
2394
  },
2334
2395
  });
2335
2396
  /**
@@ -2338,24 +2399,24 @@ const ZERO_USAGE = $deepFreeze({
2338
2399
  * @public exported from `@promptbook/core`
2339
2400
  */
2340
2401
  const UNCERTAIN_USAGE = $deepFreeze({
2341
- price: { value: 0, isUncertain: true },
2402
+ price: UNCERTAIN_ZERO_VALUE,
2342
2403
  input: {
2343
- tokensCount: { value: 0, isUncertain: true },
2344
- charactersCount: { value: 0, isUncertain: true },
2345
- wordsCount: { value: 0, isUncertain: true },
2346
- sentencesCount: { value: 0, isUncertain: true },
2347
- linesCount: { value: 0, isUncertain: true },
2348
- paragraphsCount: { value: 0, isUncertain: true },
2349
- pagesCount: { value: 0, isUncertain: true },
2404
+ tokensCount: UNCERTAIN_ZERO_VALUE,
2405
+ charactersCount: UNCERTAIN_ZERO_VALUE,
2406
+ wordsCount: UNCERTAIN_ZERO_VALUE,
2407
+ sentencesCount: UNCERTAIN_ZERO_VALUE,
2408
+ linesCount: UNCERTAIN_ZERO_VALUE,
2409
+ paragraphsCount: UNCERTAIN_ZERO_VALUE,
2410
+ pagesCount: UNCERTAIN_ZERO_VALUE,
2350
2411
  },
2351
2412
  output: {
2352
- tokensCount: { value: 0, isUncertain: true },
2353
- charactersCount: { value: 0, isUncertain: true },
2354
- wordsCount: { value: 0, isUncertain: true },
2355
- sentencesCount: { value: 0, isUncertain: true },
2356
- linesCount: { value: 0, isUncertain: true },
2357
- paragraphsCount: { value: 0, isUncertain: true },
2358
- pagesCount: { value: 0, isUncertain: true },
2413
+ tokensCount: UNCERTAIN_ZERO_VALUE,
2414
+ charactersCount: UNCERTAIN_ZERO_VALUE,
2415
+ wordsCount: UNCERTAIN_ZERO_VALUE,
2416
+ sentencesCount: UNCERTAIN_ZERO_VALUE,
2417
+ linesCount: UNCERTAIN_ZERO_VALUE,
2418
+ paragraphsCount: UNCERTAIN_ZERO_VALUE,
2419
+ pagesCount: UNCERTAIN_ZERO_VALUE,
2359
2420
  },
2360
2421
  });
2361
2422
  /**
@@ -2526,7 +2587,7 @@ function extractParameterNamesFromTask(task) {
2526
2587
  if (parameterNames.has(subparameterName)) {
2527
2588
  parameterNames.delete(subparameterName);
2528
2589
  parameterNames.add(foreach.parameterName);
2529
- // <- TODO: [🚎] Warn/logic error when `subparameterName` not used
2590
+ // <- TODO: [🏮] Warn/logic error when `subparameterName` not used
2530
2591
  }
2531
2592
  }
2532
2593
  }
@@ -3020,6 +3081,7 @@ function joinLlmExecutionTools(...llmExecutionTools) {
3020
3081
  `);
3021
3082
  // TODO: [🟥] Detect browser / node and make it colorfull
3022
3083
  console.warn(warningMessage);
3084
+ // <- TODO: [🏮] Some standard way how to transform errors into warnings and how to handle non-critical fails during the tasks
3023
3085
  /*
3024
3086
  return {
3025
3087
  async listModels() {
@@ -4631,6 +4693,7 @@ function createPipelineExecutor(options) {
4631
4693
 
4632
4694
  @see more at https://ptbk.io/prepare-pipeline
4633
4695
  `));
4696
+ // <- TODO: [🏮] Some standard way how to transform errors into warnings and how to handle non-critical fails during the tasks
4634
4697
  }
4635
4698
  let runCount = 0;
4636
4699
  const pipelineExecutorWithCallback = async (inputParameters, onProgress) => {
@@ -4707,8 +4770,9 @@ async function forEachAsync(array, options, callbackfunction) {
4707
4770
  * @returns LLM tools with same functionality with added total cost counting
4708
4771
  * @public exported from `@promptbook/core`
4709
4772
  */
4710
- function countTotalUsage(llmTools) {
4773
+ function countUsage(llmTools) {
4711
4774
  let totalUsage = ZERO_USAGE;
4775
+ const spending = new Subject();
4712
4776
  const proxyTools = {
4713
4777
  get title() {
4714
4778
  // TODO: [🧠] Maybe put here some suffix
@@ -4718,12 +4782,15 @@ function countTotalUsage(llmTools) {
4718
4782
  // TODO: [🧠] Maybe put here some suffix
4719
4783
  return llmTools.description;
4720
4784
  },
4721
- async checkConfiguration() {
4785
+ checkConfiguration() {
4722
4786
  return /* not await */ llmTools.checkConfiguration();
4723
4787
  },
4724
4788
  listModels() {
4725
4789
  return /* not await */ llmTools.listModels();
4726
4790
  },
4791
+ spending() {
4792
+ return spending.asObservable();
4793
+ },
4727
4794
  getTotalUsage() {
4728
4795
  // <- Note: [🥫] Not using getter `get totalUsage` but `getTotalUsage` to allow this object to be proxied
4729
4796
  return totalUsage;
@@ -4734,6 +4801,7 @@ function countTotalUsage(llmTools) {
4734
4801
  // console.info('[🚕] callChatModel through countTotalUsage');
4735
4802
  const promptResult = await llmTools.callChatModel(prompt);
4736
4803
  totalUsage = addUsage(totalUsage, promptResult.usage);
4804
+ spending.next(promptResult.usage);
4737
4805
  return promptResult;
4738
4806
  };
4739
4807
  }
@@ -4742,6 +4810,7 @@ function countTotalUsage(llmTools) {
4742
4810
  // console.info('[🚕] callCompletionModel through countTotalUsage');
4743
4811
  const promptResult = await llmTools.callCompletionModel(prompt);
4744
4812
  totalUsage = addUsage(totalUsage, promptResult.usage);
4813
+ spending.next(promptResult.usage);
4745
4814
  return promptResult;
4746
4815
  };
4747
4816
  }
@@ -4750,6 +4819,7 @@ function countTotalUsage(llmTools) {
4750
4819
  // console.info('[🚕] callEmbeddingModel through countTotalUsage');
4751
4820
  const promptResult = await llmTools.callEmbeddingModel(prompt);
4752
4821
  totalUsage = addUsage(totalUsage, promptResult.usage);
4822
+ spending.next(promptResult.usage);
4753
4823
  return promptResult;
4754
4824
  };
4755
4825
  }
@@ -4920,7 +4990,7 @@ class $Register {
4920
4990
  const existingRegistration = this.storage[existingRegistrationIndex];
4921
4991
  if (!existingRegistration) {
4922
4992
  if (DEFAULT_IS_VERBOSE) {
4923
- console.warn(`[📦] Registering \`${packageName}.${className}\` to \`${this.registerName}\``);
4993
+ console.info(`[📦] Registering \`${packageName}.${className}\` to \`${this.registerName}\``);
4924
4994
  }
4925
4995
  this.storage.push(registered);
4926
4996
  }
@@ -5234,17 +5304,24 @@ function titleToName(value) {
5234
5304
  /**
5235
5305
  * The built-in `fetch' function with a lightweight error handling wrapper as default fetch function used in Promptbook scrapers
5236
5306
  *
5237
- * @private as default `fetch` function used in Promptbook scrapers
5307
+ * @public exported from `@promptbook/core`
5238
5308
  */
5239
- const scraperFetch = async (url, init) => {
5309
+ const promptbookFetch = async (urlOrRequest, init) => {
5240
5310
  try {
5241
- return await fetch(url, init);
5311
+ return await fetch(urlOrRequest, init);
5242
5312
  }
5243
5313
  catch (error) {
5244
5314
  if (!(error instanceof Error)) {
5245
5315
  throw error;
5246
5316
  }
5247
- throw new KnowledgeScrapeError(spaceTrim((block) => `
5317
+ let url;
5318
+ if (typeof urlOrRequest === 'string') {
5319
+ url = urlOrRequest;
5320
+ }
5321
+ else if (urlOrRequest instanceof Request) {
5322
+ url = urlOrRequest.url;
5323
+ }
5324
+ throw new PromptbookFetchError(spaceTrim((block) => `
5248
5325
  Can not fetch "${url}"
5249
5326
 
5250
5327
  Fetch error:
@@ -5265,7 +5342,7 @@ const scraperFetch = async (url, init) => {
5265
5342
  async function makeKnowledgeSourceHandler(knowledgeSource, tools, options) {
5266
5343
  // console.log('!! makeKnowledgeSourceHandler', knowledgeSource);
5267
5344
  var _a;
5268
- const { fetch = scraperFetch } = tools;
5345
+ const { fetch = promptbookFetch } = tools;
5269
5346
  const { knowledgeSourceContent } = knowledgeSource;
5270
5347
  let { name } = knowledgeSource;
5271
5348
  const { rootDirname = null,
@@ -5406,63 +5483,73 @@ async function prepareKnowledgePieces(knowledgeSources, tools, options) {
5406
5483
  const { maxParallelCount = DEFAULT_MAX_PARALLEL_COUNT, rootDirname, isVerbose = DEFAULT_IS_VERBOSE } = options;
5407
5484
  const knowledgePreparedUnflatten = new Array(knowledgeSources.length);
5408
5485
  await forEachAsync(knowledgeSources, { maxParallelCount }, async (knowledgeSource, index) => {
5409
- let partialPieces = null;
5410
- const sourceHandler = await makeKnowledgeSourceHandler(knowledgeSource, tools, { rootDirname, isVerbose });
5411
- const scrapers = arrayableToArray(tools.scrapers);
5412
- for (const scraper of scrapers) {
5413
- if (!scraper.metadata.mimeTypes.includes(sourceHandler.mimeType)
5414
- // <- TODO: [🦔] Implement mime-type wildcards
5415
- ) {
5416
- continue;
5417
- }
5418
- const partialPiecesUnchecked = await scraper.scrape(sourceHandler);
5419
- if (partialPiecesUnchecked !== null) {
5420
- partialPieces = [...partialPiecesUnchecked];
5421
- // <- TODO: [🪓] Here should be no need for spreading new array, just `partialPieces = partialPiecesUnchecked`
5422
- break;
5423
- }
5424
- console.warn(spaceTrim((block) => `
5425
- Cannot scrape knowledge from source despite the scraper \`${scraper.metadata.className}\` supports the mime type "${sourceHandler.mimeType}".
5486
+ try {
5487
+ let partialPieces = null;
5488
+ const sourceHandler = await makeKnowledgeSourceHandler(knowledgeSource, tools, { rootDirname, isVerbose });
5489
+ const scrapers = arrayableToArray(tools.scrapers);
5490
+ for (const scraper of scrapers) {
5491
+ if (!scraper.metadata.mimeTypes.includes(sourceHandler.mimeType)
5492
+ // <- TODO: [🦔] Implement mime-type wildcards
5493
+ ) {
5494
+ continue;
5495
+ }
5496
+ const partialPiecesUnchecked = await scraper.scrape(sourceHandler);
5497
+ if (partialPiecesUnchecked !== null) {
5498
+ partialPieces = [...partialPiecesUnchecked];
5499
+ // <- TODO: [🪓] Here should be no need for spreading new array, just `partialPieces = partialPiecesUnchecked`
5500
+ break;
5501
+ }
5502
+ console.warn(spaceTrim((block) => `
5503
+ Cannot scrape knowledge from source despite the scraper \`${scraper.metadata.className}\` supports the mime type "${sourceHandler.mimeType}".
5426
5504
 
5427
- The source:
5428
- ${block(knowledgeSource.knowledgeSourceContent
5429
- .split('\n')
5430
- .map((line) => `> ${line}`)
5431
- .join('\n'))}
5505
+ The source:
5506
+ ${block(knowledgeSource.knowledgeSourceContent
5507
+ .split('\n')
5508
+ .map((line) => `> ${line}`)
5509
+ .join('\n'))}
5432
5510
 
5433
- ${block($registeredScrapersMessage(scrapers))}
5511
+ ${block($registeredScrapersMessage(scrapers))}
5434
5512
 
5435
5513
 
5436
- `));
5437
- }
5438
- if (partialPieces === null) {
5439
- throw new KnowledgeScrapeError(spaceTrim((block) => `
5440
- Cannot scrape knowledge
5514
+ `));
5515
+ // <- TODO: [🏮] Some standard way how to transform errors into warnings and how to handle non-critical fails during the tasks
5516
+ }
5517
+ if (partialPieces === null) {
5518
+ throw new KnowledgeScrapeError(spaceTrim((block) => `
5519
+ Cannot scrape knowledge
5441
5520
 
5442
- The source:
5443
- > ${block(knowledgeSource.knowledgeSourceContent
5444
- .split('\n')
5445
- .map((line) => `> ${line}`)
5446
- .join('\n'))}
5521
+ The source:
5522
+ > ${block(knowledgeSource.knowledgeSourceContent
5523
+ .split('\n')
5524
+ .map((line) => `> ${line}`)
5525
+ .join('\n'))}
5447
5526
 
5448
- No scraper found for the mime type "${sourceHandler.mimeType}"
5527
+ No scraper found for the mime type "${sourceHandler.mimeType}"
5449
5528
 
5450
- ${block($registeredScrapersMessage(scrapers))}
5529
+ ${block($registeredScrapersMessage(scrapers))}
5451
5530
 
5452
5531
 
5453
- `));
5532
+ `));
5533
+ }
5534
+ const pieces = partialPieces.map((partialPiece) => ({
5535
+ ...partialPiece,
5536
+ sources: [
5537
+ {
5538
+ name: knowledgeSource.name,
5539
+ // line, column <- TODO: [☀]
5540
+ // <- TODO: [❎]
5541
+ },
5542
+ ],
5543
+ }));
5544
+ knowledgePreparedUnflatten[index] = pieces;
5545
+ }
5546
+ catch (error) {
5547
+ if (!(error instanceof Error)) {
5548
+ throw error;
5549
+ }
5550
+ console.warn(error);
5551
+ // <- TODO: [🏮] Some standard way how to transform errors into warnings and how to handle non-critical fails during the tasks
5454
5552
  }
5455
- const pieces = partialPieces.map((partialPiece) => ({
5456
- ...partialPiece,
5457
- sources: [
5458
- {
5459
- name: knowledgeSource.name,
5460
- // line, column <- TODO: [☀]
5461
- // <- TODO: [❎]
5462
- },
5463
- ],
5464
- }));
5465
- knowledgePreparedUnflatten[index] = pieces;
5466
5553
  });
5467
5554
  const knowledgePrepared = knowledgePreparedUnflatten.flat();
5468
5555
  return knowledgePrepared;
@@ -5568,7 +5655,7 @@ async function preparePipeline(pipeline, tools, options) {
5568
5655
  // TODO: [🚐] Make arrayable LLMs -> single LLM DRY
5569
5656
  const _llms = arrayableToArray(tools.llm);
5570
5657
  const llmTools = _llms.length === 1 ? _llms[0] : joinLlmExecutionTools(..._llms);
5571
- const llmToolsWithUsage = countTotalUsage(llmTools);
5658
+ const llmToolsWithUsage = countUsage(llmTools);
5572
5659
  // <- TODO: [🌯]
5573
5660
  /*
5574
5661
  TODO: [🧠][🪑][🔃] Should this be done or not
@@ -7287,7 +7374,8 @@ const modelCommandParser = {
7287
7374
  if ($pipelineJson.defaultModelRequirements[command.key] !== undefined) {
7288
7375
  if ($pipelineJson.defaultModelRequirements[command.key] === command.value) {
7289
7376
  console.warn(`Multiple commands \`MODEL ${command.key} ${command.value}\` in the pipeline head`);
7290
- // <- TODO: [🚎][💩] Some better way how to get warnings from pipeline parsing / logic
7377
+ // <- TODO: [🏮] Some better way how to get warnings from pipeline parsing / logic
7378
+ // <- TODO: [🏮] Some standard way how to transform errors into warnings and how to handle non-critical fails during the tasks
7291
7379
  }
7292
7380
  else {
7293
7381
  throw new ParseError(spaceTrim(`
@@ -7319,6 +7407,7 @@ const modelCommandParser = {
7319
7407
  modelVariant: 'VARIANT',
7320
7408
  maxTokens: '???',
7321
7409
  }[command.key]} ${command.value}\` in the task "${$taskJson.title || $taskJson.name}"`);
7410
+ // <- TODO: [🏮] Some standard way how to transform errors into warnings and how to handle non-critical fails during the tasks
7322
7411
  }
7323
7412
  else {
7324
7413
  throw new ParseError(spaceTrim(`
@@ -7598,15 +7687,15 @@ function $applyToTaskJson(command, $taskJson, $pipelineJson) {
7598
7687
  }
7599
7688
  console.warn(spaceTrim(`
7600
7689
 
7601
- Persona "${personaName}" is defined multiple times with different description:
7690
+ Persona "${personaName}" is defined multiple times with different description:
7602
7691
 
7603
- First definition:
7604
- ${persona.description}
7692
+ First definition:
7693
+ ${persona.description}
7605
7694
 
7606
- Second definition:
7607
- ${personaDescription}
7695
+ Second definition:
7696
+ ${personaDescription}
7608
7697
 
7609
- `));
7698
+ `));
7610
7699
  persona.description += spaceTrim('\n\n' + personaDescription);
7611
7700
  }
7612
7701
 
@@ -9168,7 +9257,8 @@ function addAutoGeneratedSection(content, options) {
9168
9257
  `));
9169
9258
  }
9170
9259
  console.warn(`No place where to put the section <!--${sectionName}-->, using the end of the file`);
9171
- // <- TODO: [🚎][💩] Some better way how to get warnings from pipeline parsing / logic
9260
+ // <- TODO: [🏮] Some standard way how to transform errors into warnings and how to handle non-critical fails during the tasks
9261
+ // <- TODO: [🏮] Some better way how to get warnings from pipeline parsing / logic
9172
9262
  return spaceTrim$1((block) => `
9173
9263
  ${block(content)}
9174
9264
 
@@ -10079,7 +10169,7 @@ function cacheLlmTools(llmTools, options = {}) {
10079
10169
  */
10080
10170
  function limitTotalUsage(llmTools, options = {}) {
10081
10171
  const { maxTotalUsage = ZERO_USAGE, storage = new MemoryStorage() } = options;
10082
- const proxyTools = countTotalUsage(llmTools);
10172
+ const proxyTools = countUsage(llmTools);
10083
10173
  if (maxTotalUsage.price.value !== 0) {
10084
10174
  throw new NotYetImplementedError('`limitTotalUsage` is not yet implemented for non-zero price');
10085
10175
  // TODO: "Cannot call `callChatModel` because the total cost limit is reached"
@@ -10132,8 +10222,7 @@ const _AnthropicClaudeMetadataRegistration = $llmToolsMetadataRegister.register(
10132
10222
  options: {
10133
10223
  apiKey: 'sk-ant-api03-',
10134
10224
  isProxied: true,
10135
- remoteUrl: DEFAULT_REMOTE_URL,
10136
- path: DEFAULT_REMOTE_URL_PATH,
10225
+ remoteServerUrl: DEFAULT_REMOTE_SERVER_URL,
10137
10226
  },
10138
10227
  };
10139
10228
  },
@@ -10828,5 +10917,5 @@ class PrefixStorage {
10828
10917
  }
10829
10918
  }
10830
10919
 
10831
- export { $llmToolsMetadataRegister, $llmToolsRegister, $scrapersMetadataRegister, $scrapersRegister, ADMIN_EMAIL, ADMIN_GITHUB_NAME, AbstractFormatError, BOOK_LANGUAGE_VERSION, BlackholeStorage, BoilerplateError, BoilerplateFormfactorDefinition, CLAIM, CallbackInterfaceTools, ChatbotFormfactorDefinition, CollectionError, CsvFormatDefinition, CsvFormatError, DEFAULT_BOOKS_DIRNAME, DEFAULT_BOOK_OUTPUT_PARAMETER_NAME, DEFAULT_BOOK_TITLE, DEFAULT_CSV_SETTINGS, DEFAULT_DOWNLOAD_CACHE_DIRNAME, DEFAULT_EXECUTION_CACHE_DIRNAME, DEFAULT_GET_PIPELINE_COLLECTION_FUNCTION_NAME, DEFAULT_INTERMEDIATE_FILES_STRATEGY, DEFAULT_IS_AUTO_INSTALLED, DEFAULT_IS_VERBOSE, DEFAULT_MAX_EXECUTION_ATTEMPTS, DEFAULT_MAX_FILE_SIZE, DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_DEPTH, DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_TOTAL, DEFAULT_MAX_PARALLEL_COUNT, DEFAULT_PIPELINE_COLLECTION_BASE_FILENAME, DEFAULT_PROMPT_TASK_TITLE, DEFAULT_REMOTE_URL, DEFAULT_REMOTE_URL_PATH, DEFAULT_SCRAPE_CACHE_DIRNAME, DEFAULT_TASK_TITLE, EXPECTATION_UNITS, EnvironmentMismatchError, ExecutionReportStringOptionsDefaults, ExpectError, FORMFACTOR_DEFINITIONS, GENERIC_PIPELINE_INTERFACE, GeneratorFormfactorDefinition, GenericFormfactorDefinition, ImageGeneratorFormfactorDefinition, KnowledgeScrapeError, LOGO_DARK_SRC, LOGO_LIGHT_SRC, LimitReachedError, MANDATORY_CSV_SETTINGS, MAX_FILENAME_LENGTH, MODEL_VARIANTS, MatcherFormfactorDefinition, MemoryStorage, MissingToolsError, MultipleLlmExecutionTools, NAME, NonTaskSectionTypes, NotFoundError, NotYetImplementedError, ORDER_OF_PIPELINE_JSON, PROMPTBOOK_ENGINE_VERSION, PROMPTBOOK_ERRORS, ParseError, PipelineExecutionError, PipelineLogicError, PipelineUrlError, PrefixStorage, RESERVED_PARAMETER_NAMES, SET_IS_VERBOSE, SectionTypes, SheetsFormfactorDefinition, TaskTypes, TextFormatDefinition, TranslatorFormfactorDefinition, UNCERTAIN_USAGE, UnexpectedError, ZERO_USAGE, _AnthropicClaudeMetadataRegistration, _AzureOpenAiMetadataRegistration, _BoilerplateScraperMetadataRegistration, _DeepseekMetadataRegistration, _DocumentScraperMetadataRegistration, _GoogleMetadataRegistration, _LegacyDocumentScraperMetadataRegistration, _MarkdownScraperMetadataRegistration, _MarkitdownScraperMetadataRegistration, _OpenAiAssistantMetadataRegistration, _OpenAiMetadataRegistration, _PdfScraperMetadataRegistration, _WebsiteScraperMetadataRegistration, addUsage, book, cacheLlmTools, collectionToJson, compilePipeline, countTotalUsage, createCollectionFromJson, createCollectionFromPromise, createCollectionFromUrl, createLlmToolsFromConfiguration, createPipelineExecutor, createSubcollection, embeddingVectorToString, executionReportJsonToString, extractParameterNamesFromTask, getPipelineInterface, isPassingExpectations, isPipelineImplementingInterface, isPipelineInterfacesEqual, isPipelinePrepared, isValidPipelineString, joinLlmExecutionTools, limitTotalUsage, makeKnowledgeSourceHandler, parsePipeline, pipelineJsonToString, prepareKnowledgePieces, preparePersona, preparePipeline, prepareTasks, prettifyPipelineString, unpreparePipeline, usageToHuman, usageToWorktime, validatePipeline, validatePipelineString };
10920
+ export { $llmToolsMetadataRegister, $llmToolsRegister, $scrapersMetadataRegister, $scrapersRegister, ADMIN_EMAIL, ADMIN_GITHUB_NAME, AbstractFormatError, AuthenticationError, BOOK_LANGUAGE_VERSION, BlackholeStorage, BoilerplateError, BoilerplateFormfactorDefinition, CLAIM, CLI_APP_ID, CallbackInterfaceTools, ChatbotFormfactorDefinition, CollectionError, CsvFormatDefinition, CsvFormatError, DEFAULT_BOOKS_DIRNAME, DEFAULT_BOOK_OUTPUT_PARAMETER_NAME, DEFAULT_BOOK_TITLE, DEFAULT_CSV_SETTINGS, DEFAULT_DOWNLOAD_CACHE_DIRNAME, DEFAULT_EXECUTION_CACHE_DIRNAME, DEFAULT_GET_PIPELINE_COLLECTION_FUNCTION_NAME, DEFAULT_INTERMEDIATE_FILES_STRATEGY, DEFAULT_IS_AUTO_INSTALLED, DEFAULT_IS_VERBOSE, DEFAULT_MAX_EXECUTION_ATTEMPTS, DEFAULT_MAX_FILE_SIZE, DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_DEPTH, DEFAULT_MAX_KNOWLEDGE_SOURCES_SCRAPING_TOTAL, DEFAULT_MAX_PARALLEL_COUNT, DEFAULT_PIPELINE_COLLECTION_BASE_FILENAME, DEFAULT_PROMPT_TASK_TITLE, DEFAULT_REMOTE_SERVER_URL, DEFAULT_SCRAPE_CACHE_DIRNAME, DEFAULT_TASK_TITLE, EXPECTATION_UNITS, EnvironmentMismatchError, ExecutionReportStringOptionsDefaults, ExpectError, FORMFACTOR_DEFINITIONS, GENERIC_PIPELINE_INTERFACE, GeneratorFormfactorDefinition, GenericFormfactorDefinition, ImageGeneratorFormfactorDefinition, KnowledgeScrapeError, LOGO_DARK_SRC, LOGO_LIGHT_SRC, LimitReachedError, MANDATORY_CSV_SETTINGS, MAX_FILENAME_LENGTH, MODEL_VARIANTS, MatcherFormfactorDefinition, MemoryStorage, MissingToolsError, MultipleLlmExecutionTools, NAME, NonTaskSectionTypes, NotFoundError, NotYetImplementedError, ORDER_OF_PIPELINE_JSON, PLAYGROUND_APP_ID, PROMPTBOOK_ENGINE_VERSION, PROMPTBOOK_ERRORS, ParseError, PipelineExecutionError, PipelineLogicError, PipelineUrlError, PrefixStorage, PromptbookFetchError, RESERVED_PARAMETER_NAMES, SET_IS_VERBOSE, SectionTypes, SheetsFormfactorDefinition, TaskTypes, TextFormatDefinition, TranslatorFormfactorDefinition, UNCERTAIN_USAGE, UNCERTAIN_ZERO_VALUE, UnexpectedError, ZERO_USAGE, ZERO_VALUE, _AnthropicClaudeMetadataRegistration, _AzureOpenAiMetadataRegistration, _BoilerplateScraperMetadataRegistration, _DeepseekMetadataRegistration, _DocumentScraperMetadataRegistration, _GoogleMetadataRegistration, _LegacyDocumentScraperMetadataRegistration, _MarkdownScraperMetadataRegistration, _MarkitdownScraperMetadataRegistration, _OpenAiAssistantMetadataRegistration, _OpenAiMetadataRegistration, _PdfScraperMetadataRegistration, _WebsiteScraperMetadataRegistration, addUsage, book, cacheLlmTools, collectionToJson, compilePipeline, countUsage, createCollectionFromJson, createCollectionFromPromise, createCollectionFromUrl, createLlmToolsFromConfiguration, createPipelineExecutor, createSubcollection, embeddingVectorToString, executionReportJsonToString, extractParameterNamesFromTask, getPipelineInterface, isPassingExpectations, isPipelineImplementingInterface, isPipelineInterfacesEqual, isPipelinePrepared, isValidPipelineString, joinLlmExecutionTools, limitTotalUsage, makeKnowledgeSourceHandler, parsePipeline, pipelineJsonToString, prepareKnowledgePieces, preparePersona, preparePipeline, prepareTasks, prettifyPipelineString, promptbookFetch, unpreparePipeline, usageToHuman, usageToWorktime, validatePipeline, validatePipelineString };
10832
10921
  //# sourceMappingURL=index.es.js.map