ohmyscrapper 0.2.1__tar.gz

ohmyscrapper-0.2.1/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.3
+Name: ohmyscrapper
+Version: 0.2.1
+Summary: This project aims to create a text-based scraper containing links to create a final PDF with general information about job openings.
+Author: Cesar Cardoso gh@bouli
+Author-email: Cesar Cardoso gh@bouli <hello@cesarcardoso.cc>
+Requires-Dist: beautifulsoup4>=4.14.3
+Requires-Dist: google-genai>=1.55.0
+Requires-Dist: markdown>=3.10
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: python-dotenv>=1.2.1
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: requests>=2.32.5
+Requires-Dist: rich>=14.2.0
+Requires-Dist: urlextract>=1.9.0
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+# OhMyScrapper - v0.2.1
+
+This project aims to create a text-based scraper containing links to create a
+final PDF with general information about job openings.
+
+> This project is using [uv](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer) by default.
+
+## Scope
+
+- Read texts;
+- Extract links;
+- Use meta og:tags to extract information;
+
+## Installation
+
+I recomend to use the [uv](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer), so you can just use the command bellow and everything is installed:
+```shell
+uv sync
+```
+
+## How to use and test (development only)
+
+OhMyScrapper works in 3 stages:
+
+1. It collects and loads urls from a text (by default `input/_chat.txt`) in a database;
+2. It scraps/access the collected urls and read what is relevant. If it finds new urls, they are collected as well;
+3. Export a list of urls in CSV files;
+
+You can do 3 stages with the command:
+```shell
+make start
+```
+> Remember to add your text file in the folder `/input` with the name `_chat.txt`!
+
+You will find the exported files in the folder `/output` like this:
+- `/output/report.csv`
+- `/output/report.csv-preview.html`
+- `/output/urls-simplified.csv`
+- `/output/urls-simplified.csv-preview.html`
+- `/output/urls.csv`
+- `/output/urls.csv-preview.html`
+
+### BUT: if you want to do step by step, here it is:
+
+First we load a text file you would like to look for urls, the idea here is to
+use the whatsapp history, but it works with any txt file.
+
+The default file is `input/_chat.txt`. If you have the default file you just use
+the command `load`:
+```shell
+make load
+```
+or, if you have another file, just use the argument `-file` like this:
+```shell
+uv run main.py load -file=my-text-file.txt
+```
+That will create a database if it doesn't exist and store every url the oh-my-scrapper
+find. After that, let's scrap the urls with the command `scrap-urls`:
+
+```shell
+make scrap-urls
+```
+
+That will scrap only the linkedin urls we are interested in. For now they are:
+- linkedin_post: https://%.linkedin.com/posts/%
+- linkedin_redirect: https://lnkd.in/%
+- linkedin_job: https://%.linkedin.com/jobs/view/%
+- linkedin_feed" https://%.linkedin.com/feed/%
+- linkedin_company: https://%.linkedin.com/company/%
+
+But we can use every other one generically using the argument `--ignore-type`:
+```shell
+uv run main.py scrap-urls --ignore-type
+```
+
+And we can ask to make it recursively adding the argument `--recursive`:
+```shell
+uv run main.py scrap-urls --recursive
+```
+> !!! important: we are not sure about blocks we can have for excess of requests
+
+And we can finally export with the command:
+```shell
+make export
+```
+
+
+That's the basic usage!
+But you can understand more using the help:
+```shell
+uv run main.py --help
+```

ohmyscrapper-0.2.1/README.md

@@ -0,0 +1,92 @@
+# OhMyScrapper - v0.2.1
+
+This project aims to create a text-based scraper containing links to create a
+final PDF with general information about job openings.
+
+> This project is using [uv](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer) by default.
+
+## Scope
+
+- Read texts;
+- Extract links;
+- Use meta og:tags to extract information;
+
+## Installation
+
+I recomend to use the [uv](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer), so you can just use the command bellow and everything is installed:
+```shell
+uv sync
+```
+
+## How to use and test (development only)
+
+OhMyScrapper works in 3 stages:
+
+1. It collects and loads urls from a text (by default `input/_chat.txt`) in a database;
+2. It scraps/access the collected urls and read what is relevant. If it finds new urls, they are collected as well;
+3. Export a list of urls in CSV files;
+
+You can do 3 stages with the command:
+```shell
+make start
+```
+> Remember to add your text file in the folder `/input` with the name `_chat.txt`!
+
+You will find the exported files in the folder `/output` like this:
+- `/output/report.csv`
+- `/output/report.csv-preview.html`
+- `/output/urls-simplified.csv`
+- `/output/urls-simplified.csv-preview.html`
+- `/output/urls.csv`
+- `/output/urls.csv-preview.html`
+
+### BUT: if you want to do step by step, here it is:
+
+First we load a text file you would like to look for urls, the idea here is to
+use the whatsapp history, but it works with any txt file.
+
+The default file is `input/_chat.txt`. If you have the default file you just use
+the command `load`:
+```shell
+make load
+```
+or, if you have another file, just use the argument `-file` like this:
+```shell
+uv run main.py load -file=my-text-file.txt
+```
+That will create a database if it doesn't exist and store every url the oh-my-scrapper
+find. After that, let's scrap the urls with the command `scrap-urls`:
+
+```shell
+make scrap-urls
+```
+
+That will scrap only the linkedin urls we are interested in. For now they are:
+- linkedin_post: https://%.linkedin.com/posts/%
+- linkedin_redirect: https://lnkd.in/%
+- linkedin_job: https://%.linkedin.com/jobs/view/%
+- linkedin_feed" https://%.linkedin.com/feed/%
+- linkedin_company: https://%.linkedin.com/company/%
+
+But we can use every other one generically using the argument `--ignore-type`:
+```shell
+uv run main.py scrap-urls --ignore-type
+```
+
+And we can ask to make it recursively adding the argument `--recursive`:
+```shell
+uv run main.py scrap-urls --recursive
+```
+> !!! important: we are not sure about blocks we can have for excess of requests
+
+And we can finally export with the command:
+```shell
+make export
+```
+
+
+That's the basic usage!
+But you can understand more using the help:
+```shell
+uv run main.py --help
+```

ohmyscrapper-0.2.1/pyproject.toml

@@ -0,0 +1,42 @@
+[project]
+name = "ohmyscrapper"
+version = "0.2.1"
+description = "This project aims to create a text-based scraper containing links to create a final PDF with general information about job openings."
+readme = "README.md"
+authors = [
+    { name = "Cesar Cardoso gh@bouli", email = "hello@cesarcardoso.cc" }
+]
+requires-python = ">=3.11"
+dependencies = [
+    "beautifulsoup4>=4.14.3",
+    "google-genai>=1.55.0",
+    "markdown>=3.10",
+    "pandas>=2.3.3",
+    "python-dotenv>=1.2.1",
+    "pyyaml>=6.0.3",
+    "requests>=2.32.5",
+    "rich>=14.2.0",
+    "urlextract>=1.9.0",
+]
+
+[project.scripts]
+ohmyscrapper = "ohmyscrapper:main"
+
+[build-system]
+requires = ["uv_build>=0.9.17,<0.10.0"]
+build-backend = "uv_build"
+
+[tool.bumpversion]
+tag = true
+tag_name = "v{new_version}"
+pre_commit_hooks = ["uv sync --upgrade", "git add uv.lock"]
+commit = true
+
+[[tool.bumpversion.files]]
+filename = "pyproject.toml"
+
+[[tool.bumpversion.files]]
+filename = "README.md"
+
+[[tool.bumpversion.files]]
+filename = "src/ohmyscrapper/__init__.py"

ohmyscrapper-0.2.1/src/ohmyscrapper/__init__.py

@@ -0,0 +1,162 @@
+import argparse
+
+from ohmyscrapper.modules.classify_urls import classify_urls
+from ohmyscrapper.modules.sniff_url import sniff_url
+from ohmyscrapper.modules.load_txt import load_txt
+from ohmyscrapper.modules.seed import seed
+from ohmyscrapper.modules.scrap_urls import scrap_urls
+from ohmyscrapper.modules.show import (
+    show_url,
+    show_urls,
+    show_urls_valid_prefix,
+    export_urls,
+    export_report,
+)
+from ohmyscrapper.modules.untouch_all import untouch_all
+from ohmyscrapper.modules.process_with_ai import process_with_ai, reprocess_ai_history
+from ohmyscrapper.modules.merge_dbs import merge_dbs
+
+
+def main():
+    parser = argparse.ArgumentParser(prog="ohmyscrapper")
+    parser.add_argument("--version", action="version", version="%(prog)s v0.2.1")
+
+    subparsers = parser.add_subparsers(dest="command", help="Available commands")
+
+    ai_process_parser = subparsers.add_parser(
+        "process-with-ai", help="Process with AI."
+    )
+    ai_process_parser.add_argument(
+        "--history", default=False, help="Reprocess ai history", action="store_true"
+    )
+
+    seed_parser = subparsers.add_parser(
+        "seed", help="Seed database. Necessary to classify urls."
+    )
+    untouch_parser = subparsers.add_parser(
+        "untouch-all", help="Untouch all urls. That resets classification"
+    )
+
+    classify_urls_parser = subparsers.add_parser(
+        "classify-urls", help="Classify loaded urls"
+    )
+    classify_urls_parser.add_argument(
+        "--recursive", default=False, help="Run in recursive mode", action="store_true"
+    )
+
+    load_txt_parser = subparsers.add_parser("load", help="Load txt file")
+    load_txt_parser.add_argument(
+        "-file", default="input/_chat.txt", help="File path. Default is input/_chat.txt"
+    )
+
+    scrap_urls_parser = subparsers.add_parser("scrap-urls", help="Scrap urls")
+    scrap_urls_parser.add_argument(
+        "--recursive", default=False, help="Run in recursive mode", action="store_true"
+    )
+    scrap_urls_parser.add_argument(
+        "--ignore-type", default=False, help="Ignore urls types", action="store_true"
+    )
+    scrap_urls_parser.add_argument(
+        "--randomize", default=False, help="Random order", action="store_true"
+    )
+    scrap_urls_parser.add_argument(
+        "--only-parents", default=False, help="Only parents urls", action="store_true"
+    )
+
+    sniff_url_parser = subparsers.add_parser("sniff-url", help="Check url")
+    sniff_url_parser.add_argument(
+        "url", default="https://cesarcardoso.cc/", help="Url to sniff"
+    )
+
+    show_urls_parser = subparsers.add_parser("show", help="Show urls and prefixes")
+    show_urls_parser.add_argument(
+        "--prefixes", default=False, help="Show urls valid prefix", action="store_true"
+    )
+    show_urls_parser.add_argument("--limit", default=0, help="Limit of lines to show")
+    show_urls_parser.add_argument("-url", default="", help="Url to show")
+
+    export_parser = subparsers.add_parser("export", help="Export urls to csv.")
+    export_parser.add_argument("--limit", default=0, help="Limit of lines to export")
+    export_parser.add_argument(
+        "--file",
+        default="output/urls.csv",
+        help="File path. Default is output/urls.csv",
+    )
+    export_parser.add_argument(
+        "--simplify",
+        default=False,
+        help="Ignore json and descriptions",
+        action="store_true",
+    )
+
+    report_parser = subparsers.add_parser("report", help="Export urls report to csv.")
+    merge_parser = subparsers.add_parser("merge_dbs", help="Merge databases.")
+
+    # TODO: What is that?
+    # seed_parser.set_defaults(func=seed)
+    # classify_urls_parser.set_defaults(func=classify_urls)
+    # load_txt_parser.set_defaults(func=load_txt)
+
+    args = parser.parse_args()
+
+    if args.command == "classify-urls":
+        classify_urls(args.recursive)
+        return
+
+    if args.command == "load":
+        load_txt(args.file)
+        return
+
+    if args.command == "seed":
+        seed()
+        return
+
+    if args.command == "untouch-all":
+        untouch_all()
+        return
+
+    if args.command == "sniff-url":
+        sniff_url(args.url)
+        return
+
+    if args.command == "scrap-urls":
+        scrap_urls(
+            recursive=args.recursive,
+            ignore_valid_prefix=args.ignore_type,
+            randomize=args.randomize,
+            only_parents=args.only_parents,
+        )
+        return
+
+    if args.command == "show":
+        if args.prefixes:
+            show_urls_valid_prefix(int(args.limit))
+            return
+        if args.url != "":
+            show_url(args.url)
+            return
+        show_urls(int(args.limit))
+        return
+
+    if args.command == "export":
+        export_urls(limit=int(args.limit), csv_file=args.file, simplify=args.simplify)
+        return
+
+    if args.command == "process-with-ai":
+        if args.history:
+            reprocess_ai_history()
+        else:
+            process_with_ai()
+        return
+
+    if args.command == "report":
+        export_report()
+        return
+
+    if args.command == "merge_dbs":
+        merge_dbs()
+        return
+
+
+if __name__ == "__main__":
+    main()

ohmyscrapper-0.2.1/src/ohmyscrapper/__main__.py

@@ -0,0 +1,4 @@
+from . import main
+
+if __name__ == "__main__":
+    main()