@memberjunction/content-autotagging 3.4.0 → 4.1.0

package/README.md

@@ -1,20 +1,42 @@
 # @memberjunction/content-autotagging
 
-A powerful AI-driven package for automatically processing, analyzing, and tagging content from various sources including RSS feeds, websites, local files, and cloud storage.
+AI-powered content ingestion and autotagging engine for MemberJunction. Scans content from multiple sources (local files, websites, RSS feeds, cloud storage), extracts text from documents, and uses LLMs to generate tags, summaries, and metadata attributes.
 
 ## Overview
 
-The `@memberjunction/content-autotagging` package provides an extensible framework for ingesting content from diverse sources and leveraging AI models to extract meaningful tags, summaries, and metadata. Built on the MemberJunction platform, it helps organizations automatically organize and categorize their content for improved searchability and insights.
-
-## Features
-
-- **Multiple Content Sources**: Support for RSS feeds, websites, local file systems, and cloud storage (Azure Blob)
-- **AI-Powered Processing**: Integrates with various AI models to generate tags, summaries, and metadata
-- **Extensible Architecture**: Easy to add new content sources and processing strategies
-- **Smart Content Detection**: Validates content types and filters out irrelevant content
-- **Incremental Processing**: Only processes new or modified content since last run
-- **File Format Support**: Handles PDFs, Office documents, HTML, and plain text
-- **Chunking Strategy**: Intelligently chunks large content to fit within AI model token limits
+The `@memberjunction/content-autotagging` package provides an extensible framework for ingesting content from diverse sources and leveraging AI models to extract meaningful tags, summaries, and metadata. Built on the MemberJunction platform, it helps organizations automatically organize and categorize their content.
+
+```mermaid
+graph TD
+    A["AutotagBaseEngine<br/>(Orchestrator)"] --> B["Content Sources"]
+    B --> C["Local File System"]
+    B --> D["Websites"]
+    B --> E["RSS Feeds"]
+    B --> F["Cloud Storage<br/>(Azure Blob)"]
+
+    A --> G["Text Extraction"]
+    G --> H["PDF Parser"]
+    G --> I["Office Parser"]
+    G --> J["HTML Parser<br/>(Cheerio)"]
+
+    A --> K["LLM Processing"]
+    K --> L["Tag Generation"]
+    K --> M["Attribute Extraction"]
+
+    A --> N["Content Items<br/>(Database)"]
+    A --> O["Content Item Attributes<br/>(Database)"]
+
+    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
+    style B fill:#7c5295,stroke:#563a6b,color:#fff
+    style C fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style D fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style E fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style F fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style G fill:#b8762f,stroke:#8a5722,color:#fff
+    style K fill:#7c5295,stroke:#563a6b,color:#fff
+    style N fill:#2d6a9f,stroke:#1a4971,color:#fff
+    style O fill:#2d6a9f,stroke:#1a4971,color:#fff
+```
 
 ## Installation
 
@@ -22,39 +44,46 @@ The `@memberjunction/content-autotagging` package provides an extensible framewo
 npm install @memberjunction/content-autotagging
 ```
 
-## Dependencies
+## Content Processing Pipeline
+
+```mermaid
+sequenceDiagram
+    participant Source as Content Source
+    participant Engine as AutotagBaseEngine
+    participant Extract as Text Extractor
+    participant AI as LLM
+    participant DB as Database
+
+    Source->>Engine: Provide content items
+    Engine->>Engine: Change detection (checksum)
+    Engine->>Extract: Extract text (PDF/Office/HTML)
+    Extract-->>Engine: Raw text
+    Engine->>Engine: Chunk text for token limits
+    Engine->>AI: Generate tags and attributes
+    AI-->>Engine: Structured metadata
+    Engine->>DB: Save ContentItem + Attributes
+    Engine->>DB: Create ProcessRun record
+```
 
-### MemberJunction Dependencies
-- `@memberjunction/ai` (2.43.0) - AI model integration
-- `@memberjunction/aiengine` (2.43.0) - AI processing pipeline  
-- `@memberjunction/core` (2.43.0) - Core MemberJunction functionality
-- `@memberjunction/core-entities` (2.43.0) - Entity models
-- `@memberjunction/global` (2.43.0) - Global utilities
-
-### External Dependencies
-- `axios` - HTTP requests
-- `cheerio` - HTML parsing and web scraping
-- `pdf-parse` - PDF document parsing
-- `officeparser` - Microsoft Office document parsing
-- `rss-parser` - RSS feed parsing
-- `date-fns` & `date-fns-tz` - Date manipulation and timezone handling
-- `openai` - OpenAI API integration
-- `xml2js` - XML parsing
-- `crypto` - Checksum generation
-
-## Architecture
-
-The package follows a modular architecture with these key components:
-
-### Core Classes
-
-1. **AutotagBase** - Abstract base class defining the autotagging interface
-2. **AutotagBaseEngine** - Central processing engine handling AI interactions and content processing
-3. **Content Source Implementations**:
-   - `AutotagRSSFeed` - RSS feed processing
-   - `AutotagWebsite` - Website crawling and processing
-   - `AutotagLocalFileSystem` - Local file processing
-   - `AutotagAzureBlob` - Azure Blob Storage integration
+## Content Sources
+
+| Source | Class | Description |
+|--------|-------|-------------|
+| Local Files | `AutotagLocalFileSystem` | Scans local directories for documents |
+| Websites | `AutotagWebsite` | Crawls web pages and extracts content |
+| RSS Feeds | `AutotagRSSFeed` | Parses RSS/Atom feeds for articles |
+| Azure Blob | `AutotagAzureBlob` | Processes files from Azure Blob Storage |
+
+All sources extend `AutotagBase`, which provides the common interface for content discovery and ingestion.
+
+## Supported File Formats
+
+| Format | Library | Extensions |
+|--------|---------|------------|
+| PDF | `pdf-parse` | .pdf |
+| Office Documents | `officeparser` | .docx, .xlsx, .pptx |
+| HTML/Web Pages | `cheerio` | .html, .htm |
+| Plain Text | Native | .txt, .md, .csv |
 
 ## Usage
 
@@ -62,238 +91,105 @@ The package follows a modular architecture with these key components:
 
 ```typescript
 import { AutotagRSSFeed } from '@memberjunction/content-autotagging';
-import { UserInfo } from '@memberjunction/core';
 
 const rssTagger = new AutotagRSSFeed();
-const userContext: UserInfo = { /* your user context */ };
-
-// Process all configured RSS feeds
-await rssTagger.Autotag(userContext);
+await rssTagger.Autotag(contextUser);
 ```
 
 ### Website Content Processing
 
 ```typescript
 import { AutotagWebsite } from '@memberjunction/content-autotagging';
-import { UserInfo } from '@memberjunction/core';
 
 const websiteTagger = new AutotagWebsite();
-const userContext: UserInfo = { /* your user context */ };
-
-// Process all configured websites with crawling options
-await websiteTagger.Autotag(userContext);
+await websiteTagger.Autotag(contextUser);
 ```
 
 ### Local File System Processing
 
 ```typescript
 import { AutotagLocalFileSystem } from '@memberjunction/content-autotagging';
-import { UserInfo } from '@memberjunction/core';
 
 const fileTagger = new AutotagLocalFileSystem();
-const userContext: UserInfo = { /* your user context */ };
-
-// Process files from configured local directories
-await fileTagger.Autotag(userContext);
+await fileTagger.Autotag(contextUser);
 ```
 
 ### Azure Blob Storage Processing
 
 ```typescript
 import { AutotagAzureBlob } from '@memberjunction/content-autotagging';
-import { UserInfo } from '@memberjunction/core';
 
 const blobTagger = new AutotagAzureBlob(
   process.env.AZURE_STORAGE_CONNECTION_STRING,
   'your-container-name'
 );
-
 await blobTagger.Authenticate();
-await blobTagger.Autotag(userContext);
+await blobTagger.Autotag(contextUser);
 ```
 
 ### Direct Engine Usage
 
-For more control over the processing pipeline:
-
 ```typescript
 import { AutotagBaseEngine } from '@memberjunction/content-autotagging';
-import { ContentItemEntity } from '@memberjunction/core-entities';
 
 const engine = AutotagBaseEngine.Instance;
-
-// Process specific content items
-const contentItems: ContentItemEntity[] = [ /* your content items */ ];
-await engine.ExtractTextAndProcessWithLLM(contentItems, userContext);
+await engine.ExtractTextAndProcessWithLLM(contentItems, contextUser);
 ```
 
-## Content Processing Pipeline
-
-1. **Content Source Discovery**: Retrieves configured content sources from the database
-2. **Content Acquisition**: Fetches content from each source (RSS, web, files, etc.)
-3. **Change Detection**: Compares checksums to identify new or modified content
-4. **Text Extraction**: Extracts text from various formats (HTML, PDF, Office docs)
-5. **AI Processing**: 
-   - Chunks content to fit model token limits
-   - Validates content type
-   - Generates title, summary, and keywords
-   - Extracts custom attributes based on content type
-6. **Storage**: Saves results to MemberJunction entities:
-   - Content Items
-   - Content Item Tags
-   - Content Item Attributes
-
-## Configuration
-
-### Content Source Configuration
-
-Content sources are configured in the MemberJunction database with these key fields:
-- `Name`: Display name
-- `URL`: Source location (RSS URL, website URL, file path, etc.)
-- `ContentTypeID`: Type of content (article, blog post, etc.)
-- `ContentSourceTypeID`: Source type (RSS Feed, Website, etc.)
-- `ContentFileTypeID`: Expected file format
-
-### AI Model Configuration
-
-The package uses AI models configured in MemberJunction. Key parameters:
-- `modelID`: Specific AI model to use
-- `minTags`: Minimum number of tags to generate
-- `maxTags`: Maximum number of tags to generate
-- Token limits are automatically handled based on model configuration
-
-### Website Crawling Options
-
-For website sources, these parameters can be configured:
-- `CrawlOtherSitesInTopLevelDomain`: Whether to crawl other subdomains
-- `CrawlSitesInLowerLevelDomain`: Whether to crawl child paths
-- `MaxDepth`: Maximum crawl depth
-- `RootURL`: Base URL for crawling
-- `URLPattern`: Regex pattern for URL filtering
-
-## Extending the Package
-
-### Creating a Custom Content Source
+## Creating a Custom Content Source
 
 ```typescript
 import { AutotagBase } from '@memberjunction/content-autotagging';
 import { RegisterClass } from '@memberjunction/global';
-import { ContentSourceEntity, ContentItemEntity } from '@memberjunction/core-entities';
-import { UserInfo } from '@memberjunction/core';
 
 @RegisterClass(AutotagBase, 'AutotagCustomSource')
 export class AutotagCustomSource extends AutotagBase {
-  public async SetContentItemsToProcess(
-    contentSources: ContentSourceEntity[]
-  ): Promise<ContentItemEntity[]> {
-    // Implement logic to fetch and create content items
-    const contentItems: ContentItemEntity[] = [];
-    
-    // Your custom source logic here
-    
+  public async SetContentItemsToProcess(contentSources) {
+    // Fetch and create content items from your custom source
     return contentItems;
   }
 
-  public async Autotag(contextUser: UserInfo): Promise<void> {
-    // Set up content source type
+  public async Autotag(contextUser) {
     const contentSourceTypeID = await this.engine.setSubclassContentSourceType(
-      'Custom Source', 
-      contextUser
+      'Custom Source', contextUser
     );
-    
-    // Get configured sources
     const contentSources = await this.engine.getAllContentSources(
-      contextUser, 
-      contentSourceTypeID
+      contextUser, contentSourceTypeID
     );
-    
-    // Process content
     const contentItems = await this.SetContentItemsToProcess(contentSources);
     await this.engine.ExtractTextAndProcessWithLLM(contentItems, contextUser);
   }
 }
 ```
 
-### Custom Content Type Attributes
-
-Add custom prompts for specific content types by creating Content Type Attributes in the database. These will be automatically included in the AI processing prompts.
-
-## API Reference
-
-### AutotagBase (Abstract)
-
-```typescript
-abstract class AutotagBase {
-  abstract SetContentItemsToProcess(
-    contentSources: ContentSourceEntity[]
-  ): Promise<ContentItemEntity[]>;
-  
-  abstract Autotag(contextUser: UserInfo): Promise<void>;
-}
-```
-
-### AutotagBaseEngine
-
-```typescript
-class AutotagBaseEngine extends AIEngine {
-  // Process content items with AI
-  async ExtractTextAndProcessWithLLM(
-    contentItems: ContentItemEntity[], 
-    contextUser: UserInfo
-  ): Promise<void>;
-  
-  // Process individual content item text
-  async ProcessContentItemText(
-    params: ContentItemProcessParams, 
-    contextUser: UserInfo
-  ): Promise<void>;
-  
-  // Get all content sources for a type
-  async getAllContentSources(
-    contextUser: UserInfo, 
-    contentSourceTypeID: string
-  ): Promise<ContentSourceEntity[]>;
-}
-```
-
-## Environment Variables
-
-```bash
-# For Azure Blob Storage
-AZURE_STORAGE_CONNECTION_STRING=your_connection_string
-
-# AI Model API Keys (handled by @memberjunction/ai)
-OPENAI_API_KEY=your_openai_key
-# Other AI provider keys as needed
-```
-
-## Error Handling
+## Database Entities
 
-The package includes comprehensive error handling:
-- Invalid content detection with automatic cleanup
-- Checksum-based duplicate detection
-- Graceful handling of parsing failures
-- Token limit management with automatic chunking
-- Network retry logic for external sources
+| Entity | Purpose |
+|--------|---------|
+| Content Sources | Configuration for each content source |
+| Content Items | Individual pieces of content with extracted text |
+| Content Item Tags | AI-generated tags |
+| Content Item Attributes | Additional extracted metadata |
+| Content Process Runs | Processing history and audit trail |
+| Content Types | Content categorization definitions |
+| Content Source Types | Source type definitions |
+| Content File Types | Supported file format definitions |
 
-## Performance Considerations
-
-- **Incremental Processing**: Only new/modified content is processed
-- **Parallel Processing**: Content items can be processed in parallel
-- **Chunking**: Large documents are automatically chunked for AI processing
-- **Caching**: Processed content checksums prevent reprocessing
-
-## Database Schema
+## Dependencies
 
-The package works with these MemberJunction entities:
-- `Content Sources` - Configuration for each source
-- `Content Items` - Individual pieces of content
-- `Content Item Tags` - Generated tags
-- `Content Item Attributes` - Additional extracted metadata
-- `Content Process Runs` - Processing history
-- `Content Types` - Content categorization
-- `Content Source Types` - Source type definitions
+| Package | Purpose |
+|---------|---------|
+| `@memberjunction/core` | Entity system and metadata |
+| `@memberjunction/global` | Class registration |
+| `@memberjunction/core-entities` | Content entity types |
+| `@memberjunction/ai` | LLM integration |
+| `@memberjunction/aiengine` | AI Engine base class |
+| `pdf-parse` | PDF text extraction |
+| `officeparser` | Office document parsing |
+| `cheerio` | HTML parsing |
+| `axios` | HTTP requests for web content |
+| `rss-parser` | RSS feed parsing |
 
 ## License
 
-ISC
+ISC

package/dist/CloudStorage/generic/CloudStorageBase.d.ts

@@ -0,0 +1,28 @@
+import { AutotagBase } from "../../Core/index.js";
+import { AutotagBaseEngine } from "../../Engine/index.js";
+import { ContentSourceParams } from "../../Engine/index.js";
+import { UserInfo } from "@memberjunction/core";
+import { ContentSourceEntity, ContentItemEntity } from "@memberjunction/core-entities";
+export declare abstract class CloudStorageBase extends AutotagBase {
+    protected contextUser: UserInfo;
+    protected engine: AutotagBaseEngine;
+    protected contentSourceTypeID: string;
+    constructor();
+    /**
+     * Abstract method to be implemented in the subclass. This method authenticates the user to the cloud storage.
+    */
+    abstract Authenticate(): Promise<void>;
+    /**
+     * Abstract method to be implemented in the subclass. Given a list of content sources, this method should return a list
+     * of content source items that have been modified or added after the most recent process run for that content source.
+     * Each cloud storage provider has a different way of accessing files and checking for modified or added items.
+     * @param contentSources - An array of content sources to check for modified or added content source items
+     * @param lastRunDate - The date of the most recent process run for the content source
+     * @param contextUser - The user context for the autotagging process
+     * @returns - An array of content source items that have been modified or added after the most recent process run for that content source
+    */
+    abstract SetNewAndModifiedContentItems(contentSourceParams: ContentSourceParams, lastRunDate: Date, contextUser: UserInfo): Promise<ContentItemEntity[]>;
+    Autotag(contextUser: UserInfo): Promise<void>;
+    SetContentItemsToProcess(contentSources: ContentSourceEntity[]): Promise<ContentItemEntity[]>;
+}
+//# sourceMappingURL=CloudStorageBase.d.ts.map

package/dist/CloudStorage/generic/CloudStorageBase.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"CloudStorageBase.d.ts","sourceRoot":"","sources":["../../../src/CloudStorage/generic/CloudStorageBase.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,iBAAiB,EAAE,MAAM,cAAc,CAAC;AACjD,OAAO,EAAE,mBAAmB,EAAE,MAAM,cAAc,CAAC;AACnD,OAAO,EAAE,QAAQ,EAAE,MAAM,sBAAsB,CAAC;AAChD,OAAO,EAAE,mBAAmB,EAAE,iBAAiB,EAAE,MAAM,+BAA+B,CAAC;AAIvF,8BAAsB,gBAAiB,SAAQ,WAAW;IACtD,SAAS,CAAC,WAAW,EAAE,QAAQ,CAAC;IAChC,SAAS,CAAC,MAAM,EAAE,iBAAiB,CAAC;IACpC,SAAS,CAAC,mBAAmB,EAAE,MAAM,CAAA;;IAOrC;;MAEE;aACc,YAAY,IAAI,OAAO,CAAC,IAAI,CAAC;IAE7C;;;;;;;;MAQE;aACc,6BAA6B,CAAC,mBAAmB,EAAE,mBAAmB,EAAE,WAAW,EAAE,IAAI,EAAE,WAAW,EAAE,QAAQ,GAAG,OAAO,CAAC,iBAAiB,EAAE,CAAC;IAElJ,OAAO,CAAC,WAAW,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IAQ7C,wBAAwB,CAAC,cAAc,EAAE,mBAAmB,EAAE,GAAG,OAAO,CAAC,iBAAiB,EAAE,CAAC;CAyB7G"}

package/dist/{src/CloudStorage → CloudStorage}/generic/CloudStorageBase.js

@@ -1,20 +1,11 @@
-"use strict";
-var __importDefault = (this && this.__importDefault) || function (mod) {
-    return (mod && mod.__esModule) ? mod : { "default": mod };
-};
-Object.defineProperty(exports, "__esModule", { value: true });
-exports.CloudStorageBase = void 0;
-const Core_1 = require("../../Core");
-const Engine_1 = require("../../Engine");
-const dotenv_1 = __importDefault(require("dotenv"));
-dotenv_1.default.config();
-class CloudStorageBase extends Core_1.AutotagBase {
-    contextUser;
-    engine;
-    contentSourceTypeID;
+import { AutotagBase } from "../../Core/index.js";
+import { AutotagBaseEngine } from "../../Engine/index.js";
+import dotenv from 'dotenv';
+dotenv.config({ quiet: true });
+export class CloudStorageBase extends AutotagBase {
     constructor() {
         super();
-        this.engine = Engine_1.AutotagBaseEngine.Instance;
+        this.engine = AutotagBaseEngine.Instance;
     }
     async Autotag(contextUser) {
         this.contextUser = contextUser;
@@ -44,5 +35,4 @@ class CloudStorageBase extends Core_1.AutotagBase {
         return contentItemsToProcess;
     }
 }
-exports.CloudStorageBase = CloudStorageBase;
 //# sourceMappingURL=CloudStorageBase.js.map

package/dist/CloudStorage/generic/CloudStorageBase.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"CloudStorageBase.js","sourceRoot":"","sources":["../../../src/CloudStorage/generic/CloudStorageBase.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,iBAAiB,EAAE,MAAM,cAAc,CAAC;AAIjD,OAAO,MAAM,MAAM,QAAQ,CAAC;AAC5B,MAAM,CAAC,MAAM,CAAC,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAA;AAE9B,MAAM,OAAgB,gBAAiB,SAAQ,WAAW;IAKtD;QACI,KAAK,EAAE,CAAC;QACR,IAAI,CAAC,MAAM,GAAG,iBAAiB,CAAC,QAAQ,CAAC;IAC7C,CAAC;IAkBM,KAAK,CAAC,OAAO,CAAC,WAAqB;QACtC,IAAI,CAAC,WAAW,GAAG,WAAW,CAAC;QAC/B,IAAI,CAAC,mBAAmB,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,4BAA4B,CAAC,eAAe,EAAE,IAAI,CAAC,WAAW,CAAC,CAAC;QAC7G,MAAM,cAAc,GAA0B,MAAM,IAAI,CAAC,MAAM,CAAC,oBAAoB,CAAC,IAAI,CAAC,WAAW,EAAE,IAAI,CAAC,mBAAmB,CAAC,IAAI,EAAE,CAAC;QACvI,MAAM,qBAAqB,GAAwB,MAAM,IAAI,CAAC,wBAAwB,CAAC,cAAc,CAAC,CAAA;QACtG,MAAM,IAAI,CAAC,MAAM,CAAC,4BAA4B,CAAC,qBAAqB,EAAE,IAAI,CAAC,WAAW,CAAC,CAAC;IAC5F,CAAC;IAEM,KAAK,CAAC,wBAAwB,CAAC,cAAqC;QACvE,MAAM,qBAAqB,GAAwB,EAAE,CAAA;QAErD,KAAK,MAAM,aAAa,IAAI,cAAc,EAAE,CAAC;YACzC,MAAM,IAAI,CAAC,YAAY,EAAE,CAAC;YAE1B,MAAM,mBAAmB,GAAwB;gBAC7C,eAAe,EAAE,aAAa,CAAC,EAAE;gBACjC,IAAI,EAAE,aAAa,CAAC,IAAI;gBACxB,aAAa,EAAE,aAAa,CAAC,aAAa;gBAC1C,mBAAmB,EAAE,aAAa,CAAC,mBAAmB;gBACtD,iBAAiB,EAAE,aAAa,CAAC,iBAAiB;gBAClD,GAAG,EAAE,aAAa,CAAC,GAAG;aACzB,CAAA;YAED,MAAM,WAAW,GAAS,MAAM,IAAI,CAAC,MAAM,CAAC,2BAA2B,CAAC,mBAAmB,CAAC,eAAe,EAAE,IAAI,CAAC,WAAW,CAAC,CAAC;YAE/H,IAAI,WAAW,EAAE,CAAC;gBACd,MAAM,YAAY,GAAG,MAAM,IAAI,CAAC,6BAA6B,CAAC,mBAAmB,EAAE,WAAW,EAAE,IAAI,CAAC,WAAW,CAAC,CAAC;gBAClH,qBAAqB,CAAC,IAAI,CAAC,GAAG,YAAY,CAAC,CAAC;YAChD,CAAC;QACL,CAAC;QAED,OAAO,qBAAqB,CAAC;IACjC,CAAC;CACJ"}

package/dist/CloudStorage/index.d.ts

@@ -0,0 +1,3 @@
+export * from './generic/CloudStorageBase.js';
+export * from './providers/AutotagAzureBlob.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/CloudStorage/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/CloudStorage/index.ts"],"names":[],"mappings":"AAAA,cAAc,4BAA4B,CAAA;AAC1C,cAAc,8BAA8B,CAAA"}

package/dist/CloudStorage/index.js

@@ -0,0 +1,3 @@
+export * from './generic/CloudStorageBase.js';
+export * from './providers/AutotagAzureBlob.js';
+//# sourceMappingURL=index.js.map

package/dist/CloudStorage/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/CloudStorage/index.ts"],"names":[],"mappings":"AAAA,cAAc,4BAA4B,CAAA;AAC1C,cAAc,8BAA8B,CAAA"}

package/dist/{src/CloudStorage → CloudStorage}/providers/AutotagAzureBlob.d.ts

@@ -1,15 +1,17 @@
-/// <reference types="node" />
-/// <reference types="node" />
-import { CloudStorageBase } from "../generic/CloudStorageBase";
+import { CloudStorageBase } from "../generic/CloudStorageBase.js";
 import { UserInfo } from "@memberjunction/core";
 import { ContentItemEntity } from "@memberjunction/core-entities";
-import { ContentSourceParams } from "../../Engine";
+import { ContentSourceParams } from "../../Engine/index.js";
 export declare class AutotagAzureBlob extends CloudStorageBase {
     private blobServiceClient;
     private containerClient;
     private connectionString;
     private containerName;
     constructor(connectionString: string, containerName: string);
+    /**
+     * Implemented abstract method from the CloudStorageBase class for cloud storage authentication. This method authenticates the user to the Azure Blob Storage.
+     * @returns void
+    */
     Authenticate(): Promise<void>;
     SetNewAndModifiedContentItems(contentSourceParams: ContentSourceParams, lastRunDate: Date, contextUser: UserInfo, prefix?: string): Promise<ContentItemEntity[]>;
     extractText(file: string): Promise<string>;

package/dist/CloudStorage/providers/AutotagAzureBlob.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"AutotagAzureBlob.d.ts","sourceRoot":"","sources":["../../../src/CloudStorage/providers/AutotagAzureBlob.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,MAAM,6BAA6B,CAAC;AAC/D,OAAO,EAAE,QAAQ,EAAE,MAAM,sBAAsB,CAAC;AAGhD,OAAO,EAAE,iBAAiB,EAAE,MAAM,+BAA+B,CAAC;AAGlE,OAAO,EAAE,mBAAmB,EAAE,MAAM,cAAc,CAAC;AAGnD,qBAAa,gBAAiB,SAAQ,gBAAgB;IAClD,OAAO,CAAC,iBAAiB,CAAoB;IAC7C,OAAO,CAAC,eAAe,CAAkB;IACzC,OAAO,CAAC,gBAAgB,CAAS;IACjC,OAAO,CAAC,aAAa,CAAS;gBAElB,gBAAgB,EAAE,MAAM,EAAE,aAAa,EAAE,MAAM;IAO3D;;;MAGE;IACW,YAAY,IAAI,OAAO,CAAC,IAAI,CAAC;IAc7B,6BAA6B,CAAC,mBAAmB,EAAE,mBAAmB,EAAE,WAAW,EAAE,IAAI,EAAE,WAAW,EAAE,QAAQ,EAAE,MAAM,SAAG,GAAG,OAAO,CAAC,iBAAiB,EAAE,CAAC;IAwC1J,WAAW,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;IAQ1C,cAAc,CAAC,cAAc,EAAE,MAAM,CAAC,cAAc,GAAG,OAAO,CAAC,MAAM,CAAC;CAYtF"}

package/dist/{src/CloudStorage → CloudStorage}/providers/AutotagAzureBlob.js

@@ -1,28 +1,22 @@
-"use strict";
-var __importDefault = (this && this.__importDefault) || function (mod) {
-    return (mod && mod.__esModule) ? mod : { "default": mod };
-};
-Object.defineProperty(exports, "__esModule", { value: true });
-exports.AutotagAzureBlob = void 0;
-const CloudStorageBase_1 = require("../generic/CloudStorageBase");
-const storage_blob_1 = require("@azure/storage-blob");
-const dotenv_1 = __importDefault(require("dotenv"));
-const core_1 = require("@memberjunction/core");
-const path_1 = __importDefault(require("path"));
-dotenv_1.default.config();
-class AutotagAzureBlob extends CloudStorageBase_1.CloudStorageBase {
-    blobServiceClient;
-    containerClient;
-    connectionString;
-    containerName;
+import { CloudStorageBase } from "../generic/CloudStorageBase.js";
+import { BlobServiceClient } from '@azure/storage-blob';
+import dotenv from 'dotenv';
+import { Metadata } from "@memberjunction/core";
+import path from "path";
+dotenv.config({ quiet: true });
+export class AutotagAzureBlob extends CloudStorageBase {
     constructor(connectionString, containerName) {
         super();
         this.connectionString = connectionString;
         this.containerName = containerName;
     }
+    /**
+     * Implemented abstract method from the CloudStorageBase class for cloud storage authentication. This method authenticates the user to the Azure Blob Storage.
+     * @returns void
+    */
     async Authenticate() {
         try {
-            this.blobServiceClient = storage_blob_1.BlobServiceClient.fromConnectionString(this.connectionString);
+            this.blobServiceClient = BlobServiceClient.fromConnectionString(this.connectionString);
             for await (const container of this.blobServiceClient.listContainers()) {
                 console.log(`Container: ${container.name}`);
             }
@@ -36,9 +30,10 @@ class AutotagAzureBlob extends CloudStorageBase_1.CloudStorageBase {
     async SetNewAndModifiedContentItems(contentSourceParams, lastRunDate, contextUser, prefix = '') {
         const contentItemsToProcess = [];
         for await (const blob of this.containerClient.listBlobsFlat()) {
-            const filePath = path_1.default.join(this.containerName, blob.name);
+            const filePath = path.join(this.containerName, blob.name);
             if (blob.properties.createdOn && blob.properties.createdOn > lastRunDate) {
-                const md = new core_1.Metadata();
+                // The file has been created, add a new record for this file
+                const md = new Metadata();
                 const contentItem = await md.GetEntityObject('Content Items', contextUser);
                 const text = await this.extractText(blob.name);
                 contentItem.ContentSourceID = contentSourceParams.contentSourceID;
@@ -54,7 +49,8 @@ class AutotagAzureBlob extends CloudStorageBase_1.CloudStorageBase {
                 contentItemsToProcess.push(contentItem);
             }
             else if (blob.properties.lastModified && blob.properties.lastModified > lastRunDate) {
-                const md = new core_1.Metadata();
+                // The file has been modified, update the record for this file
+                const md = new Metadata();
                 const contentItem = await md.GetEntityObject('Content Items', contextUser);
                 const contentItemID = await this.engine.getContentItemIDFromURL(contentSourceParams, contextUser);
                 await contentItem.Load(contentItemID);
@@ -87,5 +83,4 @@ class AutotagAzureBlob extends CloudStorageBase_1.CloudStorageBase {
         });
     }
 }
-exports.AutotagAzureBlob = AutotagAzureBlob;
 //# sourceMappingURL=AutotagAzureBlob.js.map

package/dist/CloudStorage/providers/AutotagAzureBlob.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"AutotagAzureBlob.js","sourceRoot":"","sources":["../../../src/CloudStorage/providers/AutotagAzureBlob.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,MAAM,6BAA6B,CAAC;AAE/D,OAAO,EAAE,iBAAiB,EAAmB,MAAM,qBAAqB,CAAC;AACzE,OAAO,MAAM,MAAM,QAAQ,CAAC;AAE5B,OAAO,EAAE,QAAQ,EAAE,MAAM,sBAAsB,CAAC;AAChD,OAAO,IAAI,MAAM,MAAM,CAAC;AAExB,MAAM,CAAC,MAAM,CAAC,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAA;AAE9B,MAAM,OAAO,gBAAiB,SAAQ,gBAAgB;IAMlD,YAAY,gBAAwB,EAAE,aAAqB;QACvD,KAAK,EAAE,CAAC;QACR,IAAI,CAAC,gBAAgB,GAAG,gBAAgB,CAAA;QACxC,IAAI,CAAC,aAAa,GAAG,aAAa,CAAA;IACtC,CAAC;IAGD;;;MAGE;IACK,KAAK,CAAC,YAAY;QACrB,IAAI,CAAC;YACD,IAAI,CAAC,iBAAiB,GAAG,iBAAiB,CAAC,oBAAoB,CAAC,IAAI,CAAC,gBAAgB,CAAC,CAAC;YACvF,IAAI,KAAK,EAAE,MAAM,SAAS,IAAI,IAAI,CAAC,iBAAiB,CAAC,cAAc,EAAE,EAAE,CAAC;gBACpE,OAAO,CAAC,GAAG,CAAC,cAAc,SAAS,CAAC,IAAI,EAAE,CAAC,CAAC;YAChD,CAAC;YAED,IAAI,CAAC,eAAe,GAAG,IAAI,CAAC,iBAAiB,CAAC,kBAAkB,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC;QACzF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACb,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAA;YACpB,MAAM,IAAI,KAAK,CAAC,4CAA4C,CAAC,CAAA;QACjE,CAAC;IACL,CAAC;IAEM,KAAK,CAAC,6BAA6B,CAAC,mBAAwC,EAAE,WAAiB,EAAE,WAAqB,EAAE,MAAM,GAAC,EAAE;QACpI,MAAM,qBAAqB,GAAwB,EAAE,CAAA;QAErD,IAAI,KAAK,EAAE,MAAM,IAAI,IAAI,IAAI,CAAC,eAAe,CAAC,aAAa,EAAE,EAAE,CAAC;YAC5D,MAAM,QAAQ,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,aAAa,EAAE,IAAI,CAAC,IAAI,CAAC,CAAA;YACzD,IAAI,IAAI,CAAC,UAAU,CAAC,SAAS,IAAI,IAAI,CAAC,UAAU,CAAC,SAAS,GAAG,WAAW,EAAE,CAAC;gBACvE,4DAA4D;gBAC5D,MAAM,EAAE,GAAG,IAAI,QAAQ,EAAE,CAAA;gBACzB,MAAM,WAAW,GAAG,MAAM,EAAE,CAAC,eAAe,CAAoB,eAAe,EAAE,WAAW,CAAC,CAAA;gBAC7F,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;gBAC9C,WAAW,CAAC,eAAe,GAAG,mBAAmB,CAAC,eAAe,CAAA;gBACjE,WAAW,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,CAAA;gBAC5B,WAAW,CAAC,WAAW,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,yBAAyB,CAAC,mBAAmB,EAAE,WAAW,CAAC,CAAA;gBACvG,WAAW,CAAC,GAAG,GAAG,QAAQ,CAAA;gBAC1B,WAAW,CAAC,aAAa,GAAG,mBAAmB,CAAC,aAAa,CAAA;gBAC7D,WAAW,CAAC,mBAAmB,GAAI,mBAAmB,CAAC,mBAAmB,CAAA;gBAC1E,WAAW,CAAC,iBAAiB,GAAG,mBAAmB,CAAC,iBAAiB,CAAA;gBACrE,WAAW,CAAC,QAAQ,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,CAAA;gBAClE,WAAW,CAAC,IAAI,GAAG,IAAI,CAAA;gBAEvB,MAAM,WAAW,CAAC,IAAI,EAAE,CAAA;gBACxB,qBAAqB,CAAC,IAAI,CAAC,WAAW,CAAC,CAAA;YAC3C,CAAC;iBACI,IAAI,IAAI,CAAC,UAAU,CAAC,YAAY,IAAI,IAAI,CAAC,UAAU,CAAC,YAAY,GAAG,WAAW,EAAE,CAAC;gBAClF,8DAA8D;gBAC9D,MAAM,EAAE,GAAG,IAAI,QAAQ,EAAE,CAAA;gBACzB,MAAM,WAAW,GAAG,MAAM,EAAE,CAAC,eAAe,CAAoB,eAAe,EAAE,WAAW,CAAC,CAAA;gBAC7F,MAAM,aAAa,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,uBAAuB,CAAC,mBAAmB,EAAE,WAAW,CAAC,CAAA;gBACjG,MAAM,WAAW,CAAC,IAAI,CAAC,aAAa,CAAC,CAAA;gBACrC,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;gBAC9C,WAAW,CAAC,IAAI,GAAG,IAAI,CAAA;gBACvB,WAAW,CAAC,QAAQ,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,CAAA;gBAClE,WAAW,CAAC,IAAI,EAAE,CAAA;gBAClB,qBAAqB,CAAC,IAAI,CAAC,WAAW,CAAC,CAAA;YAC3C,CAAC;QACL,CAAC;QAED,OAAO,qBAAqB,CAAA;IAChC,CAAC;IAEM,KAAK,CAAC,WAAW,CAAC,IAAY;QACjC,MAAM,eAAe,GAAG,IAAI,CAAC,eAAe,CAAC,kBAAkB,CAAC,IAAI,CAAC,CAAA;QACrE,MAAM,yBAAyB,GAAG,MAAM,eAAe,CAAC,QAAQ,EAAE,CAAA;QAClE,MAAM,QAAQ,GAAW,MAAM,IAAI,CAAC,cAAc,CAAC,yBAAyB,CAAC,kBAAkB,CAAC,CAAA;QAChG,MAAM,IAAI,GAAW,MAAM,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAA;QACzD,OAAO,IAAI,CAAA;IACf,CAAC;IAEM,KAAK,CAAC,cAAc,CAAC,cAAqC;QAC7D,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;YACnC,MAAM,MAAM,GAAa,EAAE,CAAC;YAC5B,cAAc,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,EAAE;gBAC/B,MAAM,CAAC,IAAI,CAAC,IAAI,YAAY,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC;YACnE,CAAC,CAAC,CAAC;YACH,cAAc,CAAC,EAAE,CAAC,KAAK,EAAE,GAAG,EAAE;gBAC1B,OAAO,CAAC,MAAM,CAAC,MAAM,CAAC,MAAiC,CAAC,CAAC,CAAC;YAC9D,CAAC,CAAC,CAAC;YACH,cAAc,CAAC,EAAE,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QACvC,CAAC,CAAC,CAAC;IACP,CAAC;CACJ"}

package/dist/Core/generic/AutotagBase.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"AutotagBase.d.ts","sourceRoot":"","sources":["../../../src/Core/generic/AutotagBase.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,sBAAsB,CAAC;AAChD,OAAO,EAAE,mBAAmB,EAAE,iBAAiB,EAAE,MAAM,+BAA+B,CAAC;AAEvF,8BAAsB,WAAW;aACb,wBAAwB,CAAC,cAAc,EAAE,mBAAmB,EAAE,GAAG,OAAO,CAAC,iBAAiB,EAAE,CAAC;aAC7F,OAAO,CAAC,WAAW,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;CAChE"}

package/dist/Core/generic/AutotagBase.js

@@ -0,0 +1,3 @@
+export class AutotagBase {
+}
+//# sourceMappingURL=AutotagBase.js.map

package/dist/Core/generic/AutotagBase.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"AutotagBase.js","sourceRoot":"","sources":["../../../src/Core/generic/AutotagBase.ts"],"names":[],"mappings":"AAGA,MAAM,OAAgB,WAAW;CAGhC"}

package/dist/Core/index.d.ts

@@ -0,0 +1,2 @@
+export * from './generic/AutotagBase.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/Core/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/Core/index.ts"],"names":[],"mappings":"AAAA,cAAc,uBAAuB,CAAC"}

package/dist/Core/index.js

@@ -0,0 +1,2 @@
+export * from './generic/AutotagBase.js';
+//# sourceMappingURL=index.js.map

package/dist/Core/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/Core/index.ts"],"names":[],"mappings":"AAAA,cAAc,uBAAuB,CAAC"}