@sharpapi/sharpapi-node-web-scraping 1.0.0 → 1.0.2

package/README.md

@@ -2,12 +2,12 @@
 
 # Web Scraping API for Node.js
 
-## 🌐 Extract web content and data with ease — powered by SharpAPI.
+## 🌐 Scrape web content with ease — powered by SharpAPI.
 
 [![npm version](https://img.shields.io/npm/v/@sharpapi/sharpapi-node-web-scraping.svg)](https://www.npmjs.com/package/@sharpapi/sharpapi-node-web-scraping)
 [![License](https://img.shields.io/npm/l/@sharpapi/sharpapi-node-web-scraping.svg)](https://github.com/sharpapi/sharpapi-node-client/blob/master/LICENSE.md)
 
-**SharpAPI Web Scraping** provides powerful web scraping capabilities including HTML extraction, text content extraction, structured data parsing, link extraction, and screenshot capture. Perfect for data aggregation, monitoring, and content extraction applications.
+**SharpAPI Web Scraping** fetches and extracts content from web pages, providing structured data including page metadata, content, links, and more in a machine-readable JSON format. Perfect for data collection, content aggregation, SEO analysis, and research.
 
 ---
 
@@ -18,7 +18,12 @@
 3. [Usage](#usage)
 4. [API Documentation](#api-documentation)
 5. [Examples](#examples)
-6. [License](#license)
+6. [Use Cases](#use-cases)
+7. [Response Format](#response-format)
+8. [AI Integration](#ai-integration)
+9. [API Endpoint](#api-endpoint)
+10. [Related Packages](#related-packages)
+11. [License](#license)
 
 ---
 
@@ -33,9 +38,9 @@
 
 ### Step 1. Install the package via npm:
 
-```bash
+\`\`\`bash
 npm install @sharpapi/sharpapi-node-web-scraping
-```
+\`\`\`
 
 ### Step 2. Get your API key
 
@@ -45,248 +50,357 @@ Visit [SharpAPI.com](https://sharpapi.com/) to get your API key.
 
 ## Usage
 
-```javascript
+\`\`\`javascript
 const { SharpApiWebScrapingService } = require('@sharpapi/sharpapi-node-web-scraping');
 
-const apiKey = process.env.SHARP_API_KEY;
+const apiKey = process.env.SHARP_API_KEY; // Store your API key in environment variables
 const service = new SharpApiWebScrapingService(apiKey);
 
 async function scrapeWebsite() {
   try {
-    // Extract content from a URL
-    const content = await service.scrapeUrl('https://example.com');
-    console.log('Title:', content.title);
-    console.log('Content:', content.content);
-
-    // Extract text content
-    const text = await service.extractText('https://example.com');
-    console.log('Text:', text);
-
-    // Extract all links
-    const links = await service.extractLinks('https://example.com');
-    console.log(`Found ${links.length} links`);
+    // Scrape a webpage
+    const data = await service.scrapeUrl('https://sharpapi.com');
+
+    console.log('Title:', data.title);
+    console.log('Description:', data.meta_description);
+    console.log('Keywords:', data.meta_keywords);
+    console.log('Content length:', data.content.length);
+    console.log('Links found:', data.links.length);
   } catch (error) {
     console.error('Error:', error.message);
   }
 }
 
 scrapeWebsite();
-```
+\`\`\`
 
 ---
 
 ## API Documentation
 
-### Methods
+This endpoint is **synchronous** and returns data immediately (no polling required).
 
-#### `scrapeUrl(url: string): Promise<object>`
+### Method
 
-Scrape a webpage and extract its content (synchronous).
+#### `scrapeUrl(url)`
 
-**Parameters:**
-- `url` (string, required): The URL to scrape
-
-**Returns:**
-- Scraped content including title, text, and metadata
-
-#### `scrapeHtml(url: string, options?: object): Promise<object>`
+Scrape a webpage and extract its content in structured format.
 
-Extract raw HTML content from a webpage.
-
-**Options:**
-- `javascript` (boolean): Execute JavaScript (default: false)
-- `timeout` (number): Request timeout in ms (default: 30000)
-- `userAgent` (string): Custom user agent
-- `headers` (object): Custom HTTP headers
-- `proxy` (string): Proxy server URL
-
-#### `extractText(url: string, options?: object): Promise<object>`
+**Parameters:**
+- `url` (string, required): The URL to scrape (e.g., 'https://example.com' or 'example.com')
+
+**Returns:** Promise<object> - Structured page data with metadata, content, and links
+
+**Extracts:**
+- **Page metadata**: Title, description, keywords, author
+- **Open Graph tags**: OG:title, OG:description, OG:image, OG:type
+- **Twitter Card tags**: Twitter:card, Twitter:title, Twitter:description
+- **Content structure**: Headings (H1-H6), paragraphs, main content
+- **Links**: Internal and external links with anchors
+- **Meta information**: Language, charset, viewport, canonical URL
+- **Timestamps**: Extraction date and time
+
+**Example:**
+\`\`\`javascript
+const data = await service.scrapeUrl('https://example.com');
+console.log('Page Title:', data.title);
+console.log('Main Content:', data.content);
+console.log('All Links:', data.links);
+\`\`\`
 
-Extract clean text content from a webpage.
+---
 
-#### `extractLinks(url: string, options?: object): Promise<object>`
+## Examples
 
-Extract all links from a webpage.
+### Basic Web Scraping
 
-#### `extractStructuredData(url: string, options?: object): Promise<object>`
+\`\`\`javascript
+const { SharpApiWebScrapingService } = require('@sharpapi/sharpapi-node-web-scraping');
 
-Extract structured data (JSON-LD, microdata, etc.) from a webpage.
+const service = new SharpApiWebScrapingService(process.env.SHARP_API_KEY);
 
-#### `takeScreenshot(url: string, options?: object): Promise<object>`
+async function scrapeExample() {
+  const result = await service.scrapeUrl('https://sharpapi.com');
 
-Capture a screenshot of a webpage.
+  console.log('=== Page Information ===');
+  console.log('Title:', result.title);
+  console.log('Description:', result.meta_description);
+  console.log('Language:', result.language);
+  console.log('Canonical URL:', result.canonical_url);
 
-**Options:**
-- `fullPage` (boolean): Capture full page (default: false)
-- `width` (number): Viewport width (default: 1280)
-- `height` (number): Viewport height (default: 800)
-- `javascript` (boolean): Execute JavaScript (default: true)
+  console.log('\\n=== Content ===');
+  console.log('Characters:', result.content.length);
+  console.log('Preview:', result.content.substring(0, 200) + '...');
 
----
+  console.log('\\n=== Links Found ===');
+  console.log('Total links:', result.links.length);
+  result.links.slice(0, 5).forEach(link => {
+    console.log(\`- \${link.text || 'No text'}: \${link.url}\`);
+  });
+}
 
-## Examples
+scrapeExample();
+\`\`\`
 
-### Price Monitoring
+### Extract Social Media Metadata
 
-```javascript
+\`\`\`javascript
 const service = new SharpApiWebScrapingService(process.env.SHARP_API_KEY);
 
-async function monitorPrice(productUrl) {
-  const content = await service.scrapeUrl(productUrl);
+async function getSocialMetadata(url) {
+  const data = await service.scrapeUrl(url);
 
-  // Extract price from content
-  const priceMatch = content.text.match(/\$(\d+\.\d{2})/);
-  const price = priceMatch ? parseFloat(priceMatch[1]) : null;
+  console.log('=== Open Graph Tags ===');
+  console.log('OG:Title:', data.og_title);
+  console.log('OG:Description:', data.og_description);
+  console.log('OG:Image:', data.og_image);
+  console.log('OG:Type:', data.og_type);
 
-  return {
-    url: productUrl,
-    price: price,
-    title: content.title,
-    timestamp: new Date().toISOString()
-  };
+  console.log('\\n=== Twitter Card ===');
+  console.log('Card Type:', data.twitter_card);
+  console.log('Title:', data.twitter_title);
+  console.log('Description:', data.twitter_description);
+  console.log('Image:', data.twitter_image);
 }
 
-const priceData = await monitorPrice('https://example.com/product');
-console.log('Price:', priceData.price);
-```
+getSocialMetadata('https://example.com/article');
+\`\`\`
 
-### Content Aggregation
+### SEO Analysis
 
-```javascript
+\`\`\`javascript
 const service = new SharpApiWebScrapingService(process.env.SHARP_API_KEY);
 
-async function aggregateNews(urls) {
-  const articles = await Promise.all(
-    urls.map(async (url) => {
-      const content = await service.scrapeUrl(url);
-      return {
-        title: content.title,
-        text: content.text.substring(0, 200) + '...',
-        url: url,
-        scrapedAt: new Date()
-      };
-    })
-  );
-
-  return articles;
-}
+async function analyzeSEO(url) {
+  const data = await service.scrapeUrl(url);
+
+  console.log('=== SEO Analysis ===');
+  console.log('Title:', data.title, \`(\${data.title.length} chars)\`);
+  console.log('Meta Description:', data.meta_description);
+  console.log('Keywords:', data.meta_keywords);
+  console.log('Canonical URL:', data.canonical_url);
+  console.log('Language:', data.language);
+
+  console.log('\\n=== Headings Structure ===');
+  if (data.headings) {
+    data.headings.forEach(heading => {
+      console.log(\`\${heading.level}: \${heading.text}\`);
+    });
+  }
 
-const newsUrls = [
-  'https://news-site.com/article-1',
-  'https://news-site.com/article-2'
-];
+  console.log('\\n=== Link Analysis ===');
+  const internalLinks = data.links.filter(l => l.type === 'internal');
+  const externalLinks = data.links.filter(l => l.type === 'external');
+  console.log(\`Internal links: \${internalLinks.length}\`);
+  console.log(\`External links: \${externalLinks.length}\`);
+}
 
-const articles = await aggregateNews(newsUrls);
-articles.forEach(article => {
-  console.log(`\n${article.title}`);
-  console.log(article.text);
-});
-```
+analyzeSEO('https://your-website.com');
+\`\`\`
 
-### SEO Analysis
+### Content Extraction for AI Processing
 
-```javascript
+\`\`\`javascript
 const service = new SharpApiWebScrapingService(process.env.SHARP_API_KEY);
 
-async function analyzeSEO(url) {
-  const [html, links, structured] = await Promise.all([
-    service.scrapeHtml(url),
-    service.extractLinks(url),
-    service.extractStructuredData(url)
-  ]);
-
-  const analysis = {
-    url: url,
-    title: html.title,
-    meta: html.meta,
-    wordCount: html.text.split(/\s+/).length,
-    internalLinks: links.filter(l => l.internal).length,
-    externalLinks: links.filter(l => !l.internal).length,
-    hasStructuredData: Object.keys(structured).length > 0,
-    structuredDataTypes: Object.keys(structured)
+async function extractForAI(url) {
+  const data = await service.scrapeUrl(url);
+
+  // Extract clean content for AI processing
+  const cleanContent = {
+    title: data.title,
+    description: data.meta_description,
+    mainContent: data.content,
+    language: data.language,
+    author: data.author,
+    publishedDate: data.published_date,
+    modifiedDate: data.modified_date
   };
 
-  return analysis;
+  console.log('Extracted content ready for AI processing:');
+  console.log(JSON.stringify(cleanContent, null, 2));
+
+  // Now you can pass this to SharpAPI AI endpoints:
+  // - Summarization: @sharpapi/sharpapi-node-summarize-text
+  // - Translation: @sharpapi/sharpapi-node-translate
+  // - Keywords: @sharpapi/sharpapi-node-generate-keywords
+  // - SEO Tags: @sharpapi/sharpapi-node-seo-tags
 }
 
-const seoReport = await analyzeSEO('https://example.com');
-console.log('SEO Analysis:', seoReport);
-```
+extractForAI('https://blog.example.com/article');
+\`\`\`
 
-### Website Screenshots
+### Competitor Analysis
 
-```javascript
+\`\`\`javascript
 const service = new SharpApiWebScrapingService(process.env.SHARP_API_KEY);
-const fs = require('fs');
 
-async function captureWebsite(url, outputPath) {
-  const screenshot = await service.takeScreenshot(url, {
-    fullPage: true,
-    width: 1920,
-    height: 1080
-  });
+async function analyzeCompetitor(url) {
+  const data = await service.scrapeUrl(url);
 
-  // Save screenshot (base64 encoded)
-  const buffer = Buffer.from(screenshot.data, 'base64');
-  fs.writeFileSync(outputPath, buffer);
+  console.log('=== Competitor Analysis ===');
+  console.log('Domain:', new URL(url).hostname);
+  console.log('Title Strategy:', data.title);
+  console.log('Description:', data.meta_description);
+  console.log('Keywords Focus:', data.meta_keywords);
 
-  console.log(`Screenshot saved to ${outputPath}`);
+  console.log('\\n=== Content Strategy ===');
+  console.log('Content Length:', data.content.length, 'characters');
+  console.log('Word Count (approx):', Math.round(data.content.split(' ').length));
+
+  console.log('\\n=== Link Building ===');
+  const externalLinks = data.links.filter(l => l.type === 'external');
+  console.log('External Links:', externalLinks.length);
+  externalLinks.slice(0, 10).forEach(link => {
+    console.log(\`  - \${link.url}\`);
+  });
 }
 
-await captureWebsite('https://example.com', './screenshot.png');
-```
+analyzeCompetitor('https://competitor-website.com');
+\`\`\`
 
 ---
 
 ## Use Cases
 
-- **Price Monitoring**: Track product prices across e-commerce sites
-- **Content Aggregation**: Collect articles and news from multiple sources
-- **SEO Analysis**: Analyze website structure and metadata
-- **Lead Generation**: Extract contact information from websites
-- **Competitive Intelligence**: Monitor competitor websites
-- **Data Collection**: Gather research data from web sources
-- **Website Monitoring**: Track website changes and updates
-- **Screenshot Services**: Generate website previews
+- **Content Aggregation**: Collect content from multiple sources
+- **Price Monitoring**: Track competitor pricing and availability
+- **Research**: Gather data for analysis and insights
+- **Lead Generation**: Extract business information from websites
+- **Market Intelligence**: Monitor industry trends and news
+- **SEO Analysis**: Analyze competitor websites and content
+- **Content Curation**: Extract articles for content platforms
+- **Social Media Monitoring**: Track mentions and brand presence
+- **Data Enrichment**: Enhance existing data with web-sourced information
+- **Competitive Intelligence**: Analyze competitor strategies
 
 ---
 
-## Features
-
-- **Synchronous Processing**: Instant results, no polling
-- **JavaScript Execution**: Handle dynamic content
-- **Custom Headers**: Full control over requests
-- **Proxy Support**: Route requests through proxies
-- **Screenshot Capture**: Visual website representation
-- **Structured Data**: Extract JSON-LD and microdata
-- **Link Extraction**: Discover internal and external links
-- **Clean Text**: Remove HTML and extract readable content
+## Response Format
+
+The API returns a comprehensive JSON object with the following structure:
+
+\`\`\`json
+{
+  "url": "https://sharpapi.com/",
+  "title": "SharpAPI - AI-Powered Workflow Automation API",
+  "meta_description": "Automate workflows with AI-powered API...",
+  "meta_keywords": "AI API, automation, workflow",
+  "author": "SharpAPI Team",
+  "language": "en",
+  "charset": "UTF-8",
+  "canonical_url": "https://sharpapi.com/",
+  "viewport": "width=device-width, initial-scale=1",
+
+  "og_title": "SharpAPI - AI-Powered API",
+  "og_description": "Automate your workflows...",
+  "og_image": "https://sharpapi.com/og-image.jpg",
+  "og_type": "website",
+  "og_url": "https://sharpapi.com/",
+
+  "twitter_card": "summary_large_image",
+  "twitter_title": "SharpAPI",
+  "twitter_description": "AI-Powered API",
+  "twitter_image": "https://sharpapi.com/twitter-card.jpg",
+
+  "content": "Full page content as text...",
+  "text_content": "Clean text without HTML...",
+
+  "headings": [
+    { "level": "h1", "text": "Main Heading" },
+    { "level": "h2", "text": "Subheading" }
+  ],
+
+  "links": [
+    {
+      "url": "https://sharpapi.com/about",
+      "text": "About Us",
+      "type": "internal",
+      "rel": null
+    },
+    {
+      "url": "https://example.com",
+      "text": "External Link",
+      "type": "external",
+      "rel": "nofollow"
+    }
+  ],
+
+  "images": [
+    {
+      "src": "https://sharpapi.com/image.jpg",
+      "alt": "Image description"
+    }
+  ],
+
+  "extracted_at": "2026-01-10T15:30:00Z",
+  "processing_time_ms": 1250
+}
+\`\`\`
 
 ---
 
-## Best Practices
+## AI Integration
+
+The extracted data can be seamlessly integrated with **SharpAPI's AI-powered endpoints** for further analysis:
+
+### Text Processing
+- **[@sharpapi/sharpapi-node-summarize-text](https://www.npmjs.com/package/@sharpapi/sharpapi-node-summarize-text)** - Summarize extracted content
+- **[@sharpapi/sharpapi-node-paraphrase](https://www.npmjs.com/package/@sharpapi/sharpapi-node-paraphrase)** - Rewrite content
+- **[@sharpapi/sharpapi-node-translate](https://www.npmjs.com/package/@sharpapi/sharpapi-node-translate)** - Translate to other languages
+
+### SEO & Keywords
+- **[@sharpapi/sharpapi-node-generate-keywords](https://www.npmjs.com/package/@sharpapi/sharpapi-node-generate-keywords)** - Extract keywords
+- **[@sharpapi/sharpapi-node-seo-tags](https://www.npmjs.com/package/@sharpapi/sharpapi-node-seo-tags)** - Generate SEO tags
 
-1. **Respect robots.txt**: Check website policies before scraping
-2. **Rate Limiting**: Don't overwhelm target servers
-3. **Error Handling**: Implement robust error handling
-4. **Data Validation**: Validate extracted data
-5. **Legal Compliance**: Ensure scraping is legal for your use case
+### Content Analysis
+- **[@sharpapi/sharpapi-node-detect-spam](https://www.npmjs.com/package/@sharpapi/sharpapi-node-detect-spam)** - Detect spam content
+- **[@sharpapi/sharpapi-node-product-review-sentiment](https://www.npmjs.com/package/@sharpapi/sharpapi-node-product-review-sentiment)** - Analyze sentiment
+
+### Example Integration
+
+\`\`\`javascript
+const { SharpApiWebScrapingService } = require('@sharpapi/sharpapi-node-web-scraping');
+const { SharpApiSummarizeService } = require('@sharpapi/sharpapi-node-summarize-text');
+
+const scrapingService = new SharpApiWebScrapingService(process.env.SHARP_API_KEY);
+const summarizeService = new SharpApiSummarizeService(process.env.SHARP_API_KEY);
+
+async function scrapeAndSummarize(url) {
+  // 1. Scrape the webpage
+  const scraped = await scrapingService.scrapeUrl(url);
+
+  // 2. Summarize the content
+  const statusUrl = await summarizeService.summarize(scraped.content);
+  const summary = await summarizeService.fetchResults(statusUrl);
+
+  console.log('Original length:', scraped.content.length);
+  console.log('Summary:', summary.getResultJson());
+}
+
+scrapeAndSummarize('https://blog.example.com/long-article');
+\`\`\`
 
 ---
 
 ## API Endpoint
 
-**GET** `/utilities/scrape_url`
+**GET** `/utilities/scrape_url?url={url}`
+
+This endpoint is **synchronous** and returns 200 OK immediately.
 
 For detailed API specifications, refer to:
-- [Postman Documentation](https://documenter.getpostman.com/view/31106842/2sBXVeGsW6)
-- [Product Page](https://sharpapi.com/en/catalog/utility/web-scraping)
+- [Postman Documentation](https://documenter.getpostman.com/view/31106842/2s9Ye8faUp)
+- [Product Page](https://sharpapi.com/en/catalog/utility/web-scraping-api)
 
 ---
 
 ## Related Packages
 
-- [@sharpapi/sharpapi-node-detect-urls](https://www.npmjs.com/package/@sharpapi/sharpapi-node-detect-urls) - URL detection
+- [@sharpapi/sharpapi-node-detect-urls](https://www.npmjs.com/package/@sharpapi/sharpapi-node-detect-urls) - Extract URLs from text
+- [@sharpapi/sharpapi-node-detect-emails](https://www.npmjs.com/package/@sharpapi/sharpapi-node-detect-emails) - Extract emails from text
+- [@sharpapi/sharpapi-node-summarize-text](https://www.npmjs.com/package/@sharpapi/sharpapi-node-summarize-text) - Summarize content
+- [@sharpapi/sharpapi-node-seo-tags](https://www.npmjs.com/package/@sharpapi/sharpapi-node-seo-tags) - Generate SEO tags
 - [@sharpapi/sharpapi-node-client](https://www.npmjs.com/package/@sharpapi/sharpapi-node-client) - Full SharpAPI SDK
 
 ---

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sharpapi/sharpapi-node-web-scraping",
-  "version": "1.0.0",
+  "version": "1.0.2",
   "description": "SharpAPI.com Node.js SDK for Web Scraping API",
   "main": "src/index.js",
   "scripts": {
@@ -20,7 +20,7 @@
   "author": "Dawid Makowski <contact@sharpapi.com>",
   "license": "MIT",
   "dependencies": {
-    "@sharpapi/sharpapi-node-core": "file:../sharpapi-node-core"
+    "@sharpapi/sharpapi-node-core": "^1.0.0"
   },
   "devDependencies": {
     "jest": "^29.7.0"

package/sharpapi-node-airports/CLAUDE.md

@@ -0,0 +1,7 @@
+<claude-mem-context>
+# Recent Activity
+
+<!-- This section is auto-generated by claude-mem. Edit content outside the tags. -->
+
+*No recent activity*
+</claude-mem-context>

package/src/CLAUDE.md

@@ -0,0 +1,7 @@
+<claude-mem-context>
+# Recent Activity
+
+<!-- This section is auto-generated by claude-mem. Edit content outside the tags. -->
+
+*No recent activity*
+</claude-mem-context>

package/src/SharpApiWebScrapingService.js

@@ -2,111 +2,47 @@ const { SharpApiCoreService } = require('@sharpapi/sharpapi-node-core');
 
 /**
  * Service for accessing Web Scraping API using SharpAPI.com
+ *
+ * Fetches and extracts content from publicly accessible URLs, providing structured data
+ * including page metadata, content, links, and more in a machine-readable JSON format.
  */
 class SharpApiWebScrapingService extends SharpApiCoreService {
   /**
-   * Scrape a webpage URL and extract its content (synchronous endpoint)
-   *
-   * @param {string} url - The URL of the webpage to scrape
-   * @returns {Promise<object>} - The scraped content
+   * Creates a new SharpApiWebScrapingService instance
+   * @param {string} apiKey - Your SharpAPI API key
+   * @param {string} [apiBaseUrl='https://sharpapi.com/api/v1'] - API base URL
    */
-  async scrapeUrl(url) {
-    const response = await this.makeRequest('GET', '/utilities/scrape_url', { url });
-    return response.data;
+  constructor(apiKey, apiBaseUrl = 'https://sharpapi.com/api/v1') {
+    super(apiKey, apiBaseUrl, '@sharpapi/sharpapi-node-web-scraping/1.0.2');
   }
 
   /**
-   * Scrape a webpage and extract its HTML content
-   *
-   * @param {string} url - The URL of the webpage to scrape
-   * @param {object} [options] - Additional options for scraping
-   * @param {boolean} [options.javascript=false] - Whether to execute JavaScript on the page
-   * @param {number} [options.timeout=30000] - Timeout in milliseconds
-   * @param {string} [options.userAgent] - Custom User-Agent string
-   * @param {object} [options.headers] - Custom headers to send with the request
-   * @param {string} [options.proxy] - Proxy to use for the request
-   * @returns {Promise<object>} - The scraped HTML content and metadata
-   */
-  async scrapeHtml(url, options = {}) {
-    const data = { url, ...options };
-    const response = await this.makeRequest('POST', '/utility/web-scraping/html', data);
-    return response.data;
-  }
-
-  /**
-   * Extract structured data from a webpage
-   *
-   * @param {string} url - The URL of the webpage to scrape
-   * @param {object} [options] - Additional options for scraping
-   * @param {boolean} [options.javascript=false] - Whether to execute JavaScript on the page
-   * @param {number} [options.timeout=30000] - Timeout in milliseconds
-   * @param {string} [options.userAgent] - Custom User-Agent string
-   * @param {object} [options.headers] - Custom headers to send with the request
-   * @param {string} [options.proxy] - Proxy to use for the request
-   * @returns {Promise<object>} - The extracted structured data
-   */
-  async extractStructuredData(url, options = {}) {
-    const data = { url, ...options };
-    const response = await this.makeRequest('POST', '/utility/web-scraping/structured-data', data);
-    return response.data;
-  }
-
-  /**
-   * Take a screenshot of a webpage
+   * Scrape a webpage URL and extract its content (synchronous endpoint)
    *
-   * @param {string} url - The URL of the webpage to screenshot
-   * @param {object} [options] - Additional options for screenshot
-   * @param {boolean} [options.fullPage=false] - Whether to capture the full page or just the viewport
-   * @param {number} [options.width=1280] - Viewport width
-   * @param {number} [options.height=800] - Viewport height
-   * @param {boolean} [options.javascript=true] - Whether to execute JavaScript on the page
-   * @param {number} [options.timeout=30000] - Timeout in milliseconds
-   * @param {string} [options.userAgent] - Custom User-Agent string
-   * @param {object} [options.headers] - Custom headers to send with the request
-   * @param {string} [options.proxy] - Proxy to use for the request
-   * @returns {Promise<object>} - The screenshot data (base64 encoded)
-   */
-  async takeScreenshot(url, options = {}) {
-    const data = { url, ...options };
-    const response = await this.makeRequest('POST', '/utility/web-scraping/screenshot', data);
-    return response.data;
-  }
-
-  /**
-   * Extract text content from a webpage
+   * Fetches and extracts structured data from any publicly accessible URL, including:
+   * - Page title and metadata (description, keywords, author)
+   * - Open Graph and Twitter card details
+   * - Headers and meta tags (content type, viewport, canonical URL, charset)
+   * - Structured content extraction (headings, paragraphs, key text elements)
+   * - Internal and external links for site structure analysis
+   * - Language detection for localization
+   * - Timestamped results for tracking
    *
-   * @param {string} url - The URL of the webpage to scrape
-   * @param {object} [options] - Additional options for scraping
-   * @param {boolean} [options.javascript=false] - Whether to execute JavaScript on the page
-   * @param {number} [options.timeout=30000] - Timeout in milliseconds
-   * @param {string} [options.userAgent] - Custom User-Agent string
-   * @param {object} [options.headers] - Custom headers to send with the request
-   * @param {string} [options.proxy] - Proxy to use for the request
-   * @returns {Promise<object>} - The extracted text content
-   */
-  async extractText(url, options = {}) {
-    const data = { url, ...options };
-    const response = await this.makeRequest('POST', '/utility/web-scraping/text', data);
-    return response.data;
-  }
-
-  /**
-   * Extract links from a webpage
+   * @param {string} url - The URL of the webpage to scrape (e.g., 'https://example.com' or 'example.com')
+   * @returns {Promise<object>} - The scraped content with metadata, structured data, and links
    *
-   * @param {string} url - The URL of the webpage to scrape
-   * @param {object} [options] - Additional options for scraping
-   * @param {boolean} [options.javascript=false] - Whether to execute JavaScript on the page
-   * @param {number} [options.timeout=30000] - Timeout in milliseconds
-   * @param {string} [options.userAgent] - Custom User-Agent string
-   * @param {object} [options.headers] - Custom headers to send with the request
-   * @param {string} [options.proxy] - Proxy to use for the request
-   * @returns {Promise<object>} - The extracted links
+   * @example
+   * // Scrape a webpage
+   * const data = await service.scrapeUrl('https://sharpapi.com');
+   * console.log(data.title); // Page title
+   * console.log(data.description); // Meta description
+   * console.log(data.content); // Main content
+   * console.log(data.links); // Array of links found
    */
-  async extractLinks(url, options = {}) {
-    const data = { url, ...options };
-    const response = await this.makeRequest('POST', '/utility/web-scraping/links', data);
-    return response.data;
+  async scrapeUrl(url) {
+    const response = await this.makeRequest('GET', '/utilities/scrape_url', { url });
+    return response;
   }
 }
 
-module.exports = { SharpApiWebScrapingService };
+module.exports = { SharpApiWebScrapingService };