@byted-las/contextlake-openclaw 1.0.3 → 1.0.4

package/README.md

@@ -1,27 +1,32 @@
 # ContextLake OpenClaw Plugin
 
-A powerful, serverless-first Knowledge Base / Knowledge Lake (知识库/知识湖) plugin for the OpenClaw Agent framework.
+A powerful, multi-modal Knowledge Base / Knowledge Lake (知识库/知识湖) plugin for the OpenClaw Agent framework.
 
-This plugin allows OpenClaw agents to natively understand, index, and retrieve local and remote documents (PDF, Word, Markdown, Text) using vector similarity search. It features a pluggable architecture supporting both completely offline operations (via LanceDB and local GGUF models) and cloud-based operations.
+This plugin allows OpenClaw agents to natively understand, profile, index, and retrieve data from diverse data sources (Local, TOS, S3, etc.). It supports advanced multi-modal document ingestion (Text, PDF, Images, Audio, Video) powered by Large AI Services (LAS) and vector similarity search.
 
 ## Features
 
-- **Multi-modal Document Ingestion**: Automatically extracts text from PDF, DOCX, TXT, and MD files.
-- **Agentic Tools**: Exposes `contextlake-ingest`, `contextlake-retrieve`, `contextlake-list`, and `contextlake-delete` skills directly to the LLM.
-- **Slash Commands**: Quick chat actions (`/contextlake-search`, `/contextlake-ingest`) for fast human-in-the-loop operations.
-- **CLI Management**: Full command-line interface for bulk operations and CI/CD integration.
-- **Local-First & Pluggable**: 
-  - Uses `LanceDB` for local embedded vector storage.
-  - Uses `node-llama-cpp` for offline, private embedding generation (`bge-small-zh` by default).
-  - Can be easily configured to use remote embedding APIs (like OpenAI or Doubao ARK) or remote storage (TOS).
+- **Data Profiling**: Connect to heterogeneous data sources (TOS/OSS/COS/S3/Local) and auto-profile schemas and media metadata.
+- **Multi-modal Ingestion**: Automatically extracts text, converts PDFs, parses audio (ASR), understands video, and embeds images directly using LAS multi-modal models.
+- **Agentic Tools**: Exposes `contextlake-ingest`, `contextlake-retrieve`, `contextlake-list`, `contextlake-delete`, `las-data-profiler`, and `contextlake-list-datasource` skills directly to the LLM.
+- **Slash Commands**: Quick chat actions (`/contextlake-search`, `/contextlake-ingest`, `/contextlake-profiler`) for fast human-in-the-loop operations.
+- **CLI Management**: Full command-line interface for bulk operations and pipeline integration.
+- **Pluggable Architecture**: 
+  - Uses `LanceDB` for local embedded multi-modal vector storage.
+  - Pluggable storage providers (Local and TOS).
+
+## Documentation
+
+For a comprehensive guide on installation, initialization (`onboard`), connecting data sources (`connect`/`profiler`), ingestion, and searching, please refer to the [**USAGE.md**](./USAGE.md).
+
+For detailed architecture and internal workflows, please refer to [AGENTS.md](./AGENTS.md).
 
 ## Installation
 
 Assuming you are in an OpenClaw environment:
 
 ```bash
-npm install
-npm run build
+npm install @byted-las/contextlake-openclaw
 ```
 
 Then register the plugin in your OpenClaw configuration:
@@ -30,7 +35,7 @@ Then register the plugin in your OpenClaw configuration:
 {
   "plugins": [
     {
-      "package": "contextlake-openclaw",
+      "package": "@byted-las/contextlake-openclaw",
       "config": {
         "metadata_storage": {
           "type": "local",
@@ -46,19 +51,36 @@ Then register the plugin in your OpenClaw configuration:
 }
 ```
 
+## Quick Start (Slash Commands)
+
+1. **Initialize Credentials** (Run in terminal):
+   ```bash
+   openclaw contextlake onboard
+   ```
+2. **Profile a Data Source**:
+   ```text
+   /contextlake-profiler my_data local /path/to/my/files .
+   ```
+3. **Ingest the Data Source** (Processes PDF, Audio, Images, Text):
+   ```text
+   /contextlake-ingest my_data
+   ```
+4. **Search your Knowledge Base**:
+   ```text
+   /contextlake-search How do I configure the API?
+   ```
+
 ## Agent Tools (Skills)
 
 The LLM agent automatically has access to the following tools:
 
-1. **`contextlake-ingest`**: Uploads and indexes files.
-2. **`contextlake-retrieve`**: Performs vector search on the indexed documents based on a query.
-3. **`contextlake-list`**: Lists all currently indexed documents.
-4. **`contextlake-delete`**: Removes documents by ID or filter.
+1. **`las-data-profiler`**: Connect to a data source and profile its structure.
+2. **`contextlake-list-datasource`**: List all connected and profiled data sources.
+3. **`contextlake-ingest`**: Process and ingest all profiled files into the knowledge base via multi-modal embeddings.
+4. **`contextlake-retrieve`**: Performs vector search on the indexed documents based on a query.
+5. **`contextlake-list`**: Lists all currently indexed documents.
+6. **`contextlake-delete`**: Removes documents by ID or filter.
 
 You can simply talk to the agent:
-> "帮我把这个 `manual.pdf` 注入到知识库中。"
-> "在知识湖中检索关于架构设计的文档。"
-
-## Development
-
-For detailed architecture and internal workflows, please refer to [AGENTS.md](./AGENTS.md).
+> "帮我把 `my_data` 数据源注入到知识库中。"
+> "在知识湖中检索关于架构设计的文档。"

package/dist/src/commands/cli.js

@@ -141,12 +141,14 @@ function getCliCommands(pluginConfig, logger) {
                 // eslint-disable-next-line no-console
                 console.log('Please provide your credentials below. Press enter to keep the current value.');
                 const lasApiKey = await (0, credentials_1.promptForInput)('LAS_API_KEY', currentCreds.LAS_API_KEY);
-                const volcengineAccessKey = await (0, credentials_1.promptForInput)('VOLCENGINE_ACCESS_KEY', currentCreds.VOLCENGINE_ACCESS_KEY);
-                const volcengineSecretKey = await (0, credentials_1.promptForInput)('VOLCENGINE_SECRET_KEY', currentCreds.VOLCENGINE_SECRET_KEY);
+                const accessKey = await (0, credentials_1.promptForInput)('ACCESS_KEY', currentCreds.ACCESS_KEY || currentCreds.VOLCENGINE_ACCESS_KEY);
+                const secretKey = await (0, credentials_1.promptForInput)('SECRET_KEY', currentCreds.SECRET_KEY || currentCreds.VOLCENGINE_SECRET_KEY);
+                const region = await (0, credentials_1.promptForInput)('REGION', currentCreds.REGION || currentCreds.VOLCENGINE_REGION || 'cn-beijing');
                 const newCreds = {
                     LAS_API_KEY: lasApiKey,
-                    VOLCENGINE_ACCESS_KEY: volcengineAccessKey,
-                    VOLCENGINE_SECRET_KEY: volcengineSecretKey
+                    ACCESS_KEY: accessKey,
+                    SECRET_KEY: secretKey,
+                    REGION: region
                 };
                 (0, credentials_1.saveCredentials)(newCreds);
                 // eslint-disable-next-line no-console

package/dist/src/commands/index.js

@@ -20,6 +20,12 @@ function registerAll(ctx, logger) {
         logger.info(`[${new Date().toISOString()}] [ContextLake] Tool registered: ${tools.deleteTool.name}`);
         ctx.registerTool(tools.lasDataProfilerTool);
         logger.info(`[${new Date().toISOString()}] [ContextLake] Tool registered: ${tools.lasDataProfilerTool.name}`);
+        ctx.registerTool(tools.listS3ObjectsTool);
+        logger.info(`[${new Date().toISOString()}] [ContextLake] Tool registered: ${tools.listS3ObjectsTool.name}`);
+        ctx.registerTool(tools.readS3ObjectTool);
+        logger.info(`[${new Date().toISOString()}] [ContextLake] Tool registered: ${tools.readS3ObjectTool.name}`);
+        ctx.registerTool(tools.writeLanceCatalogTool);
+        logger.info(`[${new Date().toISOString()}] [ContextLake] Tool registered: ${tools.writeLanceCatalogTool.name}`);
         ctx.registerTool(tools.listDatasourceTool);
         logger.info(`[${new Date().toISOString()}] [ContextLake] Tool registered: ${tools.listDatasourceTool.name}`);
         for (const lasTool of tools.lasTools) {

package/dist/src/commands/tools.d.ts

@@ -7,5 +7,8 @@ export declare function getAgentTools(pluginConfig: ContextLakeConfig, logger: a
     deleteTool: AnyAgentTool;
     lasDataProfilerTool: AnyAgentTool;
     listDatasourceTool: AnyAgentTool;
+    listS3ObjectsTool: AnyAgentTool;
+    readS3ObjectTool: AnyAgentTool;
+    writeLanceCatalogTool: AnyAgentTool;
     lasTools: AnyAgentTool[];
 };

package/dist/src/commands/tools.js

@@ -6,6 +6,8 @@ const retrieve_1 = require("../lib/actions/retrieve");
 const manage_1 = require("../lib/actions/manage");
 const profiler_1 = require("../lib/actions/profiler");
 const las_tools_1 = require("../lib/actions/las-tools");
+const s3_tools_1 = require("../lib/actions/s3-tools");
+const lance_tools_1 = require("../lib/actions/lance-tools");
 function getAgentTools(pluginConfig, logger) {
     const lasTools = (0, las_tools_1.getLasTools)(pluginConfig, logger);
     return {
@@ -281,11 +283,97 @@ Example User Queries:
                     logger.error(`[${new Date().toISOString()}] [ContextLake] las-data-profiler skill failed`, { error: error.message, stack: error.stack });
                     return {
                         content: [{ type: "text", text: String(error.message) }],
-                        details: { error: error.message
-                        }
+                        details: { error: error.message }
                     };
                 }
             }
+        },
+        listS3ObjectsTool: {
+            name: 'list-s3-objects',
+            label: 'List S3 Objects',
+            description: 'List objects in an S3-compatible bucket or local directory',
+            parameters: {
+                type: 'object',
+                properties: {
+                    vendor: { type: 'string', enum: ['volcengine', 'alibaba', 'tencent', 'aws', 'local'] },
+                    bucket: { type: 'string' },
+                    prefix: { type: 'string' },
+                    endpoint: { type: 'string' },
+                    access_key: { type: 'string' },
+                    secret_key: { type: 'string' },
+                    region: { type: 'string' },
+                    maxKeys: { type: 'integer' },
+                    continuationToken: { type: 'string' }
+                },
+                required: ['vendor', 'bucket'],
+                additionalProperties: false
+            },
+            async execute(toolCallId, params) {
+                let actualParams = params.params || params;
+                try {
+                    const result = await (0, s3_tools_1.listS3Objects)(actualParams, actualParams.prefix || '', actualParams.maxKeys, actualParams.continuationToken);
+                    return { content: [{ type: "text", text: JSON.stringify(result) }], details: result };
+                }
+                catch (e) {
+                    return { content: [{ type: "text", text: String(e.message) }], details: { error: e.message } };
+                }
+            }
+        },
+        readS3ObjectTool: {
+            name: 'read-s3-object',
+            label: 'Read S3 Object',
+            description: 'Read the contents or headers of an S3 object',
+            parameters: {
+                type: 'object',
+                properties: {
+                    vendor: { type: 'string', enum: ['volcengine', 'alibaba', 'tencent', 'aws', 'local'] },
+                    bucket: { type: 'string' },
+                    key: { type: 'string' },
+                    endpoint: { type: 'string' },
+                    access_key: { type: 'string' },
+                    secret_key: { type: 'string' },
+                    region: { type: 'string' },
+                    maxBytes: { type: 'integer' }
+                },
+                required: ['vendor', 'bucket', 'key'],
+                additionalProperties: false
+            },
+            async execute(toolCallId, params) {
+                let actualParams = params.params || params;
+                try {
+                    const buf = await (0, s3_tools_1.readS3Object)(actualParams, actualParams.key, actualParams.maxBytes);
+                    // Return as base64 string
+                    return { content: [{ type: "text", text: buf.toString('base64') }], details: { length: buf.length } };
+                }
+                catch (e) {
+                    return { content: [{ type: "text", text: String(e.message) }], details: { error: e.message } };
+                }
+            }
+        },
+        writeLanceCatalogTool: {
+            name: 'write-lance-catalog',
+            label: 'Write LanceDB Catalog',
+            description: 'Write an array of file records into a local LanceDB table',
+            parameters: {
+                type: 'object',
+                properties: {
+                    db_path: { type: 'string' },
+                    table_name: { type: 'string' },
+                    records: { type: 'array', items: { type: 'object' } }
+                },
+                required: ['db_path', 'table_name', 'records'],
+                additionalProperties: false
+            },
+            async execute(toolCallId, params) {
+                let actualParams = params.params || params;
+                try {
+                    await (0, lance_tools_1.writeLanceCatalog)(actualParams);
+                    return { content: [{ type: "text", text: "Successfully wrote records to LanceDB" }], details: { count: actualParams.records.length } };
+                }
+                catch (e) {
+                    return { content: [{ type: "text", text: String(e.message) }], details: { error: e.message } };
+                }
+            }
         }
     };
 }

package/dist/src/lib/actions/lance-tools.d.ts

@@ -0,0 +1,6 @@
+export interface LanceWriteParams {
+    db_path: string;
+    table_name: string;
+    records: any[];
+}
+export declare function writeLanceCatalog(params: LanceWriteParams): Promise<void>;

package/dist/src/lib/actions/lance-tools.js

@@ -0,0 +1,51 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.writeLanceCatalog = writeLanceCatalog;
+const lancedb = __importStar(require("@lancedb/lancedb"));
+async function writeLanceCatalog(params) {
+    if (!params.records || params.records.length === 0) {
+        return;
+    }
+    const db = await lancedb.connect(params.db_path);
+    const tableNames = await db.tableNames();
+    if (tableNames.includes(params.table_name)) {
+        const table = await db.openTable(params.table_name);
+        await table.add(params.records);
+    }
+    else {
+        await db.createTable(params.table_name, params.records);
+    }
+}

package/dist/src/lib/actions/profiler.js

@@ -38,25 +38,16 @@ exports.listDataSources = listDataSources;
 const path = __importStar(require("path"));
 const fs = __importStar(require("fs"));
 const os = __importStar(require("os"));
-const child_process_1 = require("child_process");
-// ---------------------------------------------------------------------------
-// Constants
-// ---------------------------------------------------------------------------
+const s3_tools_1 = require("./s3-tools");
+const lance_tools_1 = require("./lance-tools");
+const mime = __importStar(require("mime-types"));
 const BASE_DIR = path.join(os.homedir(), '.openclaw', 'contextlake', 'profiler');
-const PYTHON_DEPS = ['boto3', 'lancedb', 'pyarrow', 'pandas', 'Pillow', 'mutagen', 'pymupdf'];
-// ---------------------------------------------------------------------------
-// Helpers
-// ---------------------------------------------------------------------------
 function getDataSourceDir(name) {
     return path.join(BASE_DIR, name);
 }
 function ensureDir(dir) {
     fs.mkdirSync(dir, { recursive: true });
 }
-/**
- * Generate env.sh with all connection parameters for this datasource.
- * This file can be sourced to re-run the profiler or for debugging.
- */
 function writeEnvFile(dir, params) {
     const envPath = path.join(dir, 'env.sh');
     const lines = [
@@ -69,166 +60,154 @@ function writeEnvFile(dir, params) {
         `export LAS_BUCKET="${params.bucket}"`,
         `export LAS_PREFIX="${params.prefix}"`,
     ];
-    if (params.endpoint) {
+    if (params.endpoint)
         lines.push(`export LAS_ENDPOINT="${params.endpoint}"`);
-    }
-    if (params.access_key) {
+    if (params.access_key)
         lines.push(`export LAS_ACCESS_KEY="${params.access_key}"`);
-    }
-    if (params.secret_key) {
+    if (params.secret_key)
         lines.push(`export LAS_SECRET_KEY="${params.secret_key}"`);
-    }
-    if (params.region) {
+    if (params.region)
         lines.push(`export LAS_REGION="${params.region}"`);
-    }
-    if (params.sample_rows) {
+    if (params.sample_rows)
         lines.push(`export LAS_SAMPLE_ROWS="${params.sample_rows}"`);
-    }
     lines.push(`export LAS_DB_PATH="${path.join(dir, 'catalog_db')}"`);
     lines.push(`export LAS_DATASOURCE_NAME="${params.datasource_name}"`);
     lines.push('');
     fs.writeFileSync(envPath, lines.join('\n'), { mode: 0o600 });
     return envPath;
 }
-/**
- * Install Python dependencies if not already available.
- */
-function ensurePythonDeps() {
-    try {
-        (0, child_process_1.execSync)(`python3 -c "import boto3, lancedb, pyarrow, pandas, PIL, mutagen, fitz"`, {
-            stdio: 'pipe',
-        });
-    }
-    catch {
-        console.log('[las-data-profiler] Installing Python dependencies...');
-        (0, child_process_1.execSync)(`pip3 install --user ${PYTHON_DEPS.join(' ')}`, {
-            stdio: 'inherit',
-        });
-    }
-}
-/**
- * Get the path to the bundled Python script.
- */
-function getScriptPath() {
-    // The Python script is located in the scripts directory
-    return path.join(__dirname, '../scripts', 's3_catalog.py');
+function classifyFile(ext) {
+    ext = ext.toLowerCase();
+    const STRUCTURED_EXTS = ['.json', '.jsonl', '.ndjson', '.csv', '.tsv', '.parquet', '.pq'];
+    const IMAGE_EXTS = ['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.webp', '.tiff', '.tif', '.svg', '.ico', '.heic', '.heif'];
+    const AUDIO_EXTS = ['.mp3', '.wav', '.flac', '.aac', '.ogg', '.m4a', '.wma', '.opus'];
+    const VIDEO_EXTS = ['.mp4', '.avi', '.mov', '.mkv', '.webm', '.wmv', '.flv', '.m4v', '.3gp'];
+    const PDF_EXTS = ['.pdf'];
+    if (STRUCTURED_EXTS.includes(ext))
+        return { category: 'structured', mediaType: '' };
+    if (IMAGE_EXTS.includes(ext))
+        return { category: 'non-structured', mediaType: 'image' };
+    if (AUDIO_EXTS.includes(ext))
+        return { category: 'non-structured', mediaType: 'audio' };
+    if (VIDEO_EXTS.includes(ext))
+        return { category: 'non-structured', mediaType: 'video' };
+    if (PDF_EXTS.includes(ext))
+        return { category: 'non-structured', mediaType: 'pdf' };
+    return { category: 'non-structured', mediaType: '' };
 }
-// ---------------------------------------------------------------------------
-// Main Entry
-// ---------------------------------------------------------------------------
 async function connectDataSource(params, _ctx) {
-    // Validate required params
-    if (!params.datasource_name) {
+    if (!params.datasource_name)
         throw new Error('datasource_name is required');
-    }
-    if (!params.vendor) {
+    if (!params.vendor)
         throw new Error('vendor is required');
-    }
-    if (!params.bucket) {
+    if (!params.bucket)
         throw new Error('bucket is required');
-    }
-    if (params.prefix === undefined || params.prefix === null) {
+    if (params.prefix === undefined || params.prefix === null)
         throw new Error('prefix is required');
-    }
-    // For non-local vendors, validate credentials
     if (params.vendor !== 'local') {
-        if (!params.endpoint && params.vendor !== 'aws') {
+        if (!params.endpoint && params.vendor !== 'aws')
             throw new Error(`endpoint is required for vendor "${params.vendor}"`);
+        let ak = params.access_key;
+        let sk = params.secret_key;
+        if (!ak || !sk) {
+            try {
+                const { loadCredentials } = require('../../utils/credentials');
+                const creds = loadCredentials();
+                ak = ak || creds.ACCESS_KEY || creds.VOLCENGINE_ACCESS_KEY;
+                sk = sk || creds.SECRET_KEY || creds.VOLCENGINE_SECRET_KEY;
+            }
+            catch (e) {
+                // ignore
+            }
         }
-        const ak = params.access_key || process.env.TOS_ACCESS_KEY || process.env.S3_ACCESS_KEY || process.env.AWS_ACCESS_KEY_ID;
-        const sk = params.secret_key || process.env.TOS_SECRET_KEY || process.env.S3_SECRET_KEY || process.env.AWS_SECRET_ACCESS_KEY;
         if (!ak || !sk) {
-            throw new Error('access_key and secret_key are required (via params or env vars TOS_ACCESS_KEY/TOS_SECRET_KEY, S3_ACCESS_KEY/S3_SECRET_KEY, AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY)');
+            throw new Error('access_key and secret_key are required');
         }
-        // Normalise into params so env.sh picks them up
         params.access_key = ak;
         params.secret_key = sk;
     }
     const dsDir = getDataSourceDir(params.datasource_name);
     const dbPath = path.join(dsDir, 'catalog_db');
     ensureDir(dsDir);
-    // 1. Write env.sh
     const envPath = writeEnvFile(dsDir, params);
-    // 2. Ensure Python dependencies
-    ensurePythonDeps();
-    // 3. Build CLI args for the Python script
-    const scriptPath = getScriptPath();
-    const args = [
-        scriptPath,
-        '--vendor', params.vendor,
-        '--bucket', params.bucket,
-        '--prefix', params.prefix,
-        '--db-path', dbPath,
-    ];
-    if (params.endpoint) {
-        args.push('--endpoint', params.endpoint);
-    }
-    if (params.access_key) {
-        args.push('--ak', params.access_key);
-    }
-    if (params.secret_key) {
-        args.push('--sk', params.secret_key);
-    }
-    if (params.region) {
-        args.push('--region', params.region);
-    }
-    if (params.sample_rows) {
-        args.push('--sample-rows', String(params.sample_rows));
-    }
-    // 4. Execute the profiling script
-    return new Promise((resolve) => {
-        let stdout = '';
-        let stderr = '';
-        const proc = (0, child_process_1.spawn)('python3', args, {
-            cwd: dsDir,
-            env: { ...process.env },
-        });
-        proc.stdout.on('data', (data) => {
-            stdout += data.toString();
-        });
-        proc.stderr.on('data', (data) => {
-            stderr += data.toString();
-        });
-        proc.on('close', (code) => {
-            if (code !== 0) {
-                resolve({
-                    status: 'error',
-                    datasource_name: params.datasource_name,
-                    db_path: dbPath,
-                    env_path: envPath,
-                    tables: [],
-                    error: stderr || `Python script exited with code ${code}`,
-                });
-                return;
-            }
-            // Try to parse structured output from the script
-            try {
-                const jsonMatch = stdout.match(/\{[\s\S]*"summary"[\s\S]*\}/);
-                const result = jsonMatch ? JSON.parse(jsonMatch[0]) : {};
-                resolve({
-                    status: 'success',
-                    datasource_name: params.datasource_name,
-                    db_path: dbPath,
-                    env_path: envPath,
-                    tables: ['file_catalog', 'structured_schemas', 'media_metadata'],
-                    summary: result.summary || {
-                        total_files: 0,
-                        structured_files: 0,
-                        media_files: 0,
-                    },
-                });
-            }
-            catch {
-                resolve({
-                    status: 'success',
-                    datasource_name: params.datasource_name,
-                    db_path: dbPath,
-                    env_path: envPath,
-                    tables: ['file_catalog', 'structured_schemas', 'media_metadata'],
+    try {
+        let isTruncated = true;
+        let continuationToken = undefined;
+        let total_files = 0;
+        let structured_files = 0;
+        let media_files = 0;
+        const allRecords = [];
+        const scan_ts = new Date().toISOString() + 'Z';
+        while (isTruncated) {
+            const response = await (0, s3_tools_1.listS3Objects)(params, params.prefix, 1000, continuationToken);
+            for (const obj of response.Contents) {
+                const key = obj.Key || '';
+                if (key.endsWith('/'))
+                    continue;
+                const name = path.basename(key);
+                const ext = path.extname(name).toLowerCase();
+                const mimeType = mime.lookup(name) || '';
+                const { category, mediaType } = classifyFile(ext);
+                const depth = (key.match(/\//g) || []).length;
+                const parentDir = key.includes('/') ? path.basename(path.dirname(key)) : '';
+                total_files++;
+                if (category === 'structured')
+                    structured_files++;
+                if (mediaType)
+                    media_files++;
+                allRecords.push({
+                    file_path: key,
+                    file_name: name,
+                    extension: ext,
+                    mime_type: mimeType,
+                    category: category,
+                    media_type: mediaType,
+                    size_bytes: obj.Size || 0,
+                    last_modified: obj.LastModified ? String(obj.LastModified) : '',
+                    created_time: obj._created_time ? String(obj._created_time) : '',
+                    etag: (obj.ETag || '').replace(/"/g, ''),
+                    storage_class: obj.StorageClass || '',
+                    is_multipart: (obj.ETag || '').includes('-'),
+                    depth: depth,
+                    parent_dir: parentDir,
+                    vendor: params.vendor,
+                    bucket: params.bucket,
+                    has_schema: false,
+                    has_media_meta: false,
+                    scan_timestamp: scan_ts
                 });
             }
+            isTruncated = response.IsTruncated || false;
+            continuationToken = response.NextContinuationToken;
+        }
+        await (0, lance_tools_1.writeLanceCatalog)({
+            db_path: dbPath,
+            table_name: 'file_catalog',
+            records: allRecords
         });
-    });
+        return {
+            status: 'success',
+            datasource_name: params.datasource_name,
+            db_path: dbPath,
+            env_path: envPath,
+            tables: ['file_catalog'],
+            summary: {
+                total_files,
+                structured_files,
+                media_files
+            }
+        };
+    }
+    catch (error) {
+        return {
+            status: 'error',
+            datasource_name: params.datasource_name,
+            db_path: dbPath,
+            env_path: envPath,
+            tables: [],
+            error: error.message
+        };
+    }
 }
 async function listDataSources(_ctx) {
     try {

package/dist/src/lib/actions/s3-tools.d.ts

@@ -0,0 +1,18 @@
+export interface S3Params {
+    vendor: 'volcengine' | 'alibaba' | 'tencent' | 'aws' | 'local';
+    endpoint?: string;
+    access_key?: string;
+    secret_key?: string;
+    region?: string;
+    bucket: string;
+}
+export declare function listS3Objects(params: S3Params, prefix: string, maxKeys?: number, continuationToken?: string): Promise<{
+    Contents: any[];
+    IsTruncated: boolean;
+    NextContinuationToken: undefined;
+} | {
+    Contents: import("@aws-sdk/client-s3")._Object[];
+    IsTruncated: boolean | undefined;
+    NextContinuationToken: string | undefined;
+}>;
+export declare function readS3Object(params: S3Params, key: string, maxBytes?: number): Promise<Buffer>;