xscrape 1.0.0

package/LICENSE.md

@@ -0,0 +1,9 @@
+MIT License
+
+Copyright (c) 2024 Johnie Hjelm
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/README.md

@@ -0,0 +1,139 @@
+# xscrape
+
+`xscrape` is a powerful and flexible library designed for extracting and
+transforming data from HTML documents using user-defined schemas. It integrates
+seamlessly with various schema validation libraries such as Zod, Yup, Joi, and
+Effect Schema, allowing you to use your preferred validation tool.
+
+## Features
+
+- HTML Parsing: Extract data from HTML using CSS selectors with the help of
+  [cheerio](https://github.com/cheeriojs/cheerio).
+- Schema Validation: Validate and transform extracted data with schema validation libraries like [Zod](https://github.com/colinhacks/zod).
+- Custom Transformations: Provide custom transformations for extractedattributes.
+- Default Values: Define default values for missing data fields.
+
+### Schema Support
+
+| Schema Library                                       | Status              | Notes                                                              |
+| ---------------------------------------------------- | ------------------- | ------------------------------------------------------------------ |
+| [Zod](https://github.com/colinhacks/zod)             | ✅ Supported        | Default schema tool for `xscrape`                                  |
+| [Effect/Schema](https://github.com/Effect-TS/schema) | 🚧 Planned          | Planned support for Effect/Schema for additional flexibility       |
+| [Joi](https://github.com/sideway/joi)                | 🚧 Planned          | Support for Joi for those familiar with server-side validation     |
+| [Yup](https://github.com/jquense/yup)                | 🚧 Planned          | Adding Yup support for schema validation in front-end applications |
+| Others...                                            | 🔄 In Consideration | Potential support for other schema tools as per user feedback      |
+
+## Installation
+
+To install this library, use npm or yarn:
+
+```bash
+pnpm add xscrape
+# or
+npm install xscrape
+```
+
+## Usage
+
+Below is an example of how to use xscrape for extracting and transforming data
+from an HTML document:
+
+1. Define Your Schema
+
+```ts
+import { z } from 'zod';
+
+const schema = z.object({
+  title: z.string().default('No title'),
+  description: z.string(),
+  keywords: z.array(z.string()),
+  views: z.number(),
+});
+```
+
+2. Define Field Definitions
+
+```ts
+import { type SchemaFieldDefinitions } from 'xscrape';
+
+type FieldDefinitions = SchemaFieldDefinitions<z.infer<typeof schema>>;
+
+const fields: FieldDefinitions = {
+  title: { selector: 'title' },
+  description: {
+    selector: 'meta[name="description"]',
+    attribute: 'content',
+
+    defaultValue: 'No description',
+  },
+  keywords: {
+    selector: 'meta[name="keywords"]',
+    attribute: 'content',
+    transform: (value) => value.split(','),
+    defaultValue: [],
+  },
+  views: {
+    selector: 'meta[name="views"]',
+    attribute: 'content',
+    transform: (value) => parseInt(value, 10),
+    defaultValue: 0,
+  },
+};
+```
+
+3. Create a Scraper and Extract Data
+
+```ts
+import { createScraper, ZodValidator } from 'xscrape';
+
+const validator = new ZodValidator(schema);
+const scraper = createScraper({ fields, validator });
+
+const html = `
+   <!DOCTYPE html>
+   <html>
+   <head>
+     <meta name="description" content="An example description.">
+     <meta name="keywords" content="typescript,html,parsing">
+     <meta name="views" content="1234">
+     <title>Example Title</title>
+   </head>
+   <body></body>
+   </html>
+   `;
+
+const data = scraper(html);
+console.log(data);
+
+// Outputs:
+// {
+// title: 'Example Title',
+// description: 'An example description.',
+// keywords: ['typescript', 'html', 'parsing'],
+// views: 1234
+// }
+```
+
+## Configuration
+
+xscrape offers a range of configuration options through the types provided,
+allowing for detailed customization and robust data extraction and validation:
+
+- `SchemaFieldDefinitions`: Determines how fields are extracted from the HTML.
+- `SchemaValidator`: Validates the extracted data according to defined schemas.
+
+## API Reference
+
+- `createScraper(config: ScrapeConfig): (html: string) => T` Creates a scraping function based on the specified fields and validator.
+- `ZodValidator` A built-in validator using Zod, allowing you to define schemas andvalidate data effortlessly.
+
+For a complete list of API methods and more advanced configuration options,refer to the documentation on the project homepage https://github.com/johnie/xscrape.
+
+## Contributing
+
+Contributions are welcome! Please see the Contributing Guide https://github.com/johnie/xscrape/blob/main/CONTRIBUTING.md for more information.
+
+## License
+
+This project is licensed under the MIT License. See the LICENSE
+https://github.com/johnie/xscrape/blob/main/LICENSE file for details.

package/dist/index.cjs

@@ -0,0 +1,86 @@
+"use strict";
+var __create = Object.create;
+var __defProp = Object.defineProperty;
+var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __getOwnPropNames = Object.getOwnPropertyNames;
+var __getProtoOf = Object.getPrototypeOf;
+var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __export = (target, all) => {
+  for (var name in all)
+    __defProp(target, name, { get: all[name], enumerable: true });
+};
+var __copyProps = (to, from, except, desc) => {
+  if (from && typeof from === "object" || typeof from === "function") {
+    for (let key of __getOwnPropNames(from))
+      if (!__hasOwnProp.call(to, key) && key !== except)
+        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
+  }
+  return to;
+};
+var __toESM = (mod, isNodeMode, target) => (target = mod != null ? __create(__getProtoOf(mod)) : {}, __copyProps(
+  // If the importer is in node compatibility mode or this is not an ESM
+  // file that has been converted to a CommonJS file using a Babel-
+  // compatible transform (i.e. "__esModule" has not been set), then set
+  // "default" to the CommonJS "module.exports" for node compatibility.
+  isNodeMode || !mod || !mod.__esModule ? __defProp(target, "default", { value: mod, enumerable: true }) : target,
+  mod
+));
+var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
+
+// src/index.ts
+var src_exports = {};
+__export(src_exports, {
+  ZodValidator: () => ZodValidator,
+  createScraper: () => createScraper
+});
+module.exports = __toCommonJS(src_exports);
+
+// src/createScraper.ts
+var cheerio = __toESM(require("cheerio"), 1);
+var createScraper = ({
+  fields,
+  validator
+}) => {
+  return (html) => {
+    const $ = cheerio.load(html);
+    const data = {};
+    for (const key in fields) {
+      const fieldDef = fields[key];
+      const elements = $(fieldDef.selector);
+      let values = [];
+      elements.each((_, element) => {
+        const value = fieldDef.attribute ? $(element).attr(fieldDef.attribute) : $(element).text();
+        if (value !== void 0) {
+          values.push(value);
+        }
+      });
+      if (values.length === 0 && fieldDef.defaultValue !== void 0) {
+        data[key] = fieldDef.defaultValue;
+      } else if (fieldDef.multiple) {
+        data[key] = values.map(
+          (value) => fieldDef.transform ? fieldDef.transform(value) : value
+        );
+      } else {
+        const value = values[0];
+        data[key] = fieldDef.transform && value ? fieldDef.transform(value) : value;
+      }
+    }
+    return validator.validate(data);
+  };
+};
+
+// src/validators/zod.ts
+var import_zod = require("zod");
+var ZodValidator = class {
+  constructor(schema) {
+    this.schema = schema;
+  }
+  validate(data) {
+    return this.schema.parse(data);
+  }
+};
+// Annotate the CommonJS export names for ESM import in node:
+0 && (module.exports = {
+  ZodValidator,
+  createScraper
+});

package/dist/index.d.cts

@@ -0,0 +1,29 @@
+import { ZodSchema } from 'zod';
+
+type ScrapeConfig<T> = {
+    fields: SchemaFieldDefinitions<T>;
+    validator: SchemaValidator<T>;
+};
+type FieldDefinition<T> = {
+    selector: string;
+    attribute?: string;
+    transform?: (value: string) => T;
+    defaultValue?: T;
+    multiple?: boolean;
+};
+type SchemaFieldDefinitions<T> = {
+    [K in keyof T]: FieldDefinition<T[K]>;
+};
+interface SchemaValidator<T> {
+    validate(data: unknown): T;
+}
+
+declare const createScraper: <T>({ fields, validator, }: ScrapeConfig<T>) => ((html: string) => T);
+
+declare class ZodValidator<T> implements SchemaValidator<T> {
+    private schema;
+    constructor(schema: ZodSchema<T>);
+    validate(data: unknown): T;
+}
+
+export { type FieldDefinition, type SchemaFieldDefinitions, type SchemaValidator, type ScrapeConfig, ZodValidator, createScraper };

package/dist/index.d.ts

@@ -0,0 +1,29 @@
+import { ZodSchema } from 'zod';
+
+type ScrapeConfig<T> = {
+    fields: SchemaFieldDefinitions<T>;
+    validator: SchemaValidator<T>;
+};
+type FieldDefinition<T> = {
+    selector: string;
+    attribute?: string;
+    transform?: (value: string) => T;
+    defaultValue?: T;
+    multiple?: boolean;
+};
+type SchemaFieldDefinitions<T> = {
+    [K in keyof T]: FieldDefinition<T[K]>;
+};
+interface SchemaValidator<T> {
+    validate(data: unknown): T;
+}
+
+declare const createScraper: <T>({ fields, validator, }: ScrapeConfig<T>) => ((html: string) => T);
+
+declare class ZodValidator<T> implements SchemaValidator<T> {
+    private schema;
+    constructor(schema: ZodSchema<T>);
+    validate(data: unknown): T;
+}
+
+export { type FieldDefinition, type SchemaFieldDefinitions, type SchemaValidator, type ScrapeConfig, ZodValidator, createScraper };

package/dist/index.js

@@ -0,0 +1,48 @@
+// src/createScraper.ts
+import * as cheerio from "cheerio";
+var createScraper = ({
+  fields,
+  validator
+}) => {
+  return (html) => {
+    const $ = cheerio.load(html);
+    const data = {};
+    for (const key in fields) {
+      const fieldDef = fields[key];
+      const elements = $(fieldDef.selector);
+      let values = [];
+      elements.each((_, element) => {
+        const value = fieldDef.attribute ? $(element).attr(fieldDef.attribute) : $(element).text();
+        if (value !== void 0) {
+          values.push(value);
+        }
+      });
+      if (values.length === 0 && fieldDef.defaultValue !== void 0) {
+        data[key] = fieldDef.defaultValue;
+      } else if (fieldDef.multiple) {
+        data[key] = values.map(
+          (value) => fieldDef.transform ? fieldDef.transform(value) : value
+        );
+      } else {
+        const value = values[0];
+        data[key] = fieldDef.transform && value ? fieldDef.transform(value) : value;
+      }
+    }
+    return validator.validate(data);
+  };
+};
+
+// src/validators/zod.ts
+import "zod";
+var ZodValidator = class {
+  constructor(schema) {
+    this.schema = schema;
+  }
+  validate(data) {
+    return this.schema.parse(data);
+  }
+};
+export {
+  ZodValidator,
+  createScraper
+};

package/package.json

@@ -0,0 +1,63 @@
+{
+  "name": "xscrape",
+  "version": "1.0.0",
+  "description": "A flexible and powerful library designed to extract and transform data from HTML documents using user-defined schemas",
+  "main": "dist/index.js",
+  "exports": {
+    ".": {
+      "import": "./dist/index.js",
+      "require": "./dist/index.cjs"
+    }
+  },
+  "files": [
+    "dist"
+  ],
+  "type": "module",
+  "scripts": {
+    "build": "tsup",
+    "ci": "npm run build && npm run check-format && npm run check-exports && npm run lint && npm run test",
+    "lint": "tsc",
+    "test": "vitest run",
+    "format": "prettier --write .",
+    "check-format": "prettier --check .",
+    "check-exports": "attw --pack .",
+    "local-release": "npm run ci && changeset version && changeset publish"
+  },
+  "keywords": [
+    "web-scraping",
+    "data-extraction",
+    "automation",
+    "html-parsing",
+    "data-transformation",
+    "user-defined-schemas",
+    "crawler",
+    "scraper",
+    "zod",
+    "yup",
+    "joi",
+    "effect-schema"
+  ],
+  "author": "Johnie Hjelm <johnie@hjelm.im>",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/johnie/xscrape.git"
+  },
+  "bugs": {
+    "url": "https://github.com/johnie/xscrape/issues"
+  },
+  "homepage": "https://github.com/johnie/xscrape#readme",
+  "devDependencies": {
+    "@arethetypeswrong/cli": "^0.16.4",
+    "@changesets/cli": "^2.27.9",
+    "prettier": "^3.3.3",
+    "tsup": "^8.3.5",
+    "typescript": "^5.6.3",
+    "vite": "^5.4.10",
+    "vitest": "^2.1.3"
+  },
+  "dependencies": {
+    "cheerio": "^1.0.0",
+    "zod": "^3.23.8"
+  }
+}