ohmyscrapper 0.2.1__tar.gz → 0.3.4__tar.gz

{ohmyscrapper-0.2.1 → ohmyscrapper-0.3.4}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.3
 Name: ohmyscrapper
-Version: 0.2.1
-Summary: This project aims to create a text-based scraper containing links to create a final PDF with general information about job openings.
-Author: Cesar Cardoso gh@bouli
-Author-email: Cesar Cardoso gh@bouli <hello@cesarcardoso.cc>
+Version: 0.3.4
+Summary: OhMyScrapper scrapes texts and urls looking for links and jobs-data to create a final report with general information about job positions.
+Author: Cesar Cardoso
+Author-email: Cesar Cardoso <hello@cesarcardoso.cc>
 Requires-Dist: beautifulsoup4>=4.14.3
 Requires-Dist: google-genai>=1.55.0
 Requires-Dist: markdown>=3.10
@@ -16,39 +16,50 @@ Requires-Dist: urlextract>=1.9.0
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 
-# OhMyScrapper - v0.2.1
+# 🐶 OhMyScrapper - v0.3.4
 
-This project aims to create a text-based scraper containing links to create a
-final PDF with general information about job openings.
-
-> This project is using [uv](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer) by default.
+OhMyScrapper scrapes texts and urls looking for links and jobs-data to create a
+final report with general information about job positions.
 
 ## Scope
 
 - Read texts;
-- Extract links;
-- Use meta og:tags to extract information;
+- Extract and load urls;
+- Scrapes the urls looking for og:tags and titles;
+- Export a list of links with relevant information;
 
 ## Installation
 
+You can install directly in your `pip`:
+```shell
+pip install ohmyscrapper
+```
+
 I recomend to use the [uv](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer), so you can just use the command bellow and everything is installed:
 ```shell
-uv sync
+uv add ohmyscrapper
+uv run ohmyscrapper --version
 ```
 
+But you can use everything as a tool, for example:
+```shell
+uvx ohmyscrapper --version
+```
+
+
 ## How to use and test (development only)
 
 OhMyScrapper works in 3 stages:
 
-1. It collects and loads urls from a text (by default `input/_chat.txt`) in a database;
+1. It collects and loads urls from a text in a database;
 2. It scraps/access the collected urls and read what is relevant. If it finds new urls, they are collected as well;
 3. Export a list of urls in CSV files;
 
 You can do 3 stages with the command:
 ```shell
-make start
+ohmyscrapper start
 ```
-> Remember to add your text file in the folder `/input` with the name `_chat.txt`!
+> Remember to add your text file in the folder `/input` with the name that finishes with `.txt`!
 
 You will find the exported files in the folder `/output` like this:
 - `/output/report.csv`
@@ -60,23 +71,22 @@ You will find the exported files in the folder `/output` like this:
 
 ### BUT: if you want to do step by step, here it is:
 
-First we load a text file you would like to look for urls, the idea here is to
-use the whatsapp history, but it works with any txt file.
+First we load a text file you would like to look for urls. It it works with any txt file.
 
-The default file is `input/_chat.txt`. If you have the default file you just use
-the command `load`:
+The default folder is `/input`. Put one or more text (finished with `.txt`) files
+in this folder and use the command `load`:
 ```shell
-make load
+ohmyscrapper load
 ```
-or, if you have another file, just use the argument `-file` like this:
+or, if you have another file in a different folder, just use the argument `-file` like this:
 ```shell
-uv run main.py load -file=my-text-file.txt
+ohmyscrapper load -file=my-text-file.txt
 ```
 That will create a database if it doesn't exist and store every url the oh-my-scrapper
 find. After that, let's scrap the urls with the command `scrap-urls`:
 
 ```shell
-make scrap-urls
+ohmyscrapper scrap-urls --recursive --ignore-type
 ```
 
 That will scrap only the linkedin urls we are interested in. For now they are:
@@ -88,23 +98,33 @@ That will scrap only the linkedin urls we are interested in. For now they are:
 
 But we can use every other one generically using the argument `--ignore-type`:
 ```shell
-uv run main.py scrap-urls --ignore-type
+ohmyscrapper scrap-urls --ignore-type
 ```
 
 And we can ask to make it recursively adding the argument `--recursive`:
 ```shell
-uv run main.py scrap-urls --recursive
+ohmyscrapper scrap-urls --recursive
 ```
 > !!! important: we are not sure about blocks we can have for excess of requests
 
 And we can finally export with the command:
 ```shell
-make export
+ohmyscrapper export
+ohmyscrapper export --file=output/urls-simplified.csv --simplify
+ohmyscrapper report
 ```
 
 
 That's the basic usage!
 But you can understand more using the help:
 ```shell
-uv run main.py --help
+ohmyscrapper --help
 ```
+
+## See Also
+
+- Github: https://github.com/bouli/ohmyscrapper
+- PyPI: https://pypi.org/project/ohmyscrapper/
+
+## License
+This package is distributed under the [MIT license](https://opensource.org/license/MIT).

{ohmyscrapper-0.2.1 → ohmyscrapper-0.3.4}/README.md

@@ -1,36 +1,47 @@
-# OhMyScrapper - v0.2.1
+# 🐶 OhMyScrapper - v0.3.4
 
-This project aims to create a text-based scraper containing links to create a
-final PDF with general information about job openings.
-
-> This project is using [uv](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer) by default.
+OhMyScrapper scrapes texts and urls looking for links and jobs-data to create a
+final report with general information about job positions.
 
 ## Scope
 
 - Read texts;
-- Extract links;
-- Use meta og:tags to extract information;
+- Extract and load urls;
+- Scrapes the urls looking for og:tags and titles;
+- Export a list of links with relevant information;
 
 ## Installation
 
+You can install directly in your `pip`:
+```shell
+pip install ohmyscrapper
+```
+
 I recomend to use the [uv](https://docs.astral.sh/uv/getting-started/installation/#standalone-installer), so you can just use the command bellow and everything is installed:
 ```shell
-uv sync
+uv add ohmyscrapper
+uv run ohmyscrapper --version
 ```
 
+But you can use everything as a tool, for example:
+```shell
+uvx ohmyscrapper --version
+```
+
+
 ## How to use and test (development only)
 
 OhMyScrapper works in 3 stages:
 
-1. It collects and loads urls from a text (by default `input/_chat.txt`) in a database;
+1. It collects and loads urls from a text in a database;
 2. It scraps/access the collected urls and read what is relevant. If it finds new urls, they are collected as well;
 3. Export a list of urls in CSV files;
 
 You can do 3 stages with the command:
 ```shell
-make start
+ohmyscrapper start
 ```
-> Remember to add your text file in the folder `/input` with the name `_chat.txt`!
+> Remember to add your text file in the folder `/input` with the name that finishes with `.txt`!
 
 You will find the exported files in the folder `/output` like this:
 - `/output/report.csv`
@@ -42,23 +53,22 @@ You will find the exported files in the folder `/output` like this:
 
 ### BUT: if you want to do step by step, here it is:
 
-First we load a text file you would like to look for urls, the idea here is to
-use the whatsapp history, but it works with any txt file.
+First we load a text file you would like to look for urls. It it works with any txt file.
 
-The default file is `input/_chat.txt`. If you have the default file you just use
-the command `load`:
+The default folder is `/input`. Put one or more text (finished with `.txt`) files
+in this folder and use the command `load`:
 ```shell
-make load
+ohmyscrapper load
 ```
-or, if you have another file, just use the argument `-file` like this:
+or, if you have another file in a different folder, just use the argument `-file` like this:
 ```shell
-uv run main.py load -file=my-text-file.txt
+ohmyscrapper load -file=my-text-file.txt
 ```
 That will create a database if it doesn't exist and store every url the oh-my-scrapper
 find. After that, let's scrap the urls with the command `scrap-urls`:
 
 ```shell
-make scrap-urls
+ohmyscrapper scrap-urls --recursive --ignore-type
 ```
 
 That will scrap only the linkedin urls we are interested in. For now they are:
@@ -70,23 +80,33 @@ That will scrap only the linkedin urls we are interested in. For now they are:
 
 But we can use every other one generically using the argument `--ignore-type`:
 ```shell
-uv run main.py scrap-urls --ignore-type
+ohmyscrapper scrap-urls --ignore-type
 ```
 
 And we can ask to make it recursively adding the argument `--recursive`:
 ```shell
-uv run main.py scrap-urls --recursive
+ohmyscrapper scrap-urls --recursive
 ```
 > !!! important: we are not sure about blocks we can have for excess of requests
 
 And we can finally export with the command:
 ```shell
-make export
+ohmyscrapper export
+ohmyscrapper export --file=output/urls-simplified.csv --simplify
+ohmyscrapper report
 ```
 
 
 That's the basic usage!
 But you can understand more using the help:
 ```shell
-uv run main.py --help
+ohmyscrapper --help
 ```
+
+## See Also
+
+- Github: https://github.com/bouli/ohmyscrapper
+- PyPI: https://pypi.org/project/ohmyscrapper/
+
+## License
+This package is distributed under the [MIT license](https://opensource.org/license/MIT).

{ohmyscrapper-0.2.1 → ohmyscrapper-0.3.4}/pyproject.toml

@@ -1,10 +1,10 @@
 [project]
 name = "ohmyscrapper"
-version = "0.2.1"
-description = "This project aims to create a text-based scraper containing links to create a final PDF with general information about job openings."
+version = "0.3.4"
+description = "OhMyScrapper scrapes texts and urls looking for links and jobs-data to create a final report with general information about job positions."
 readme = "README.md"
 authors = [
-    { name = "Cesar Cardoso gh@bouli", email = "hello@cesarcardoso.cc" }
+    { name = "Cesar Cardoso", email = "hello@cesarcardoso.cc" }
 ]
 requires-python = ">=3.11"
 dependencies = [

{ohmyscrapper-0.2.1 → ohmyscrapper-0.3.4}/src/ohmyscrapper/__init__.py

@@ -19,13 +19,22 @@ from ohmyscrapper.modules.merge_dbs import merge_dbs
 
 def main():
     parser = argparse.ArgumentParser(prog="ohmyscrapper")
-    parser.add_argument("--version", action="version", version="%(prog)s v0.2.1")
+    parser.add_argument("--version", action="version", version="%(prog)s v0.3.4")
 
     subparsers = parser.add_subparsers(dest="command", help="Available commands")
+    start_parser = subparsers.add_parser(
+        "start",
+        help="Make the entire process of 📦 loading, 🐶 scraping and 📜🖋️ exporting with the default configuration.",
+    )
 
-    ai_process_parser = subparsers.add_parser(
-        "process-with-ai", help="Process with AI."
+    start_parser.add_argument(
+        "--ai",
+        default=False,
+        help="Make the entire process of loading, processing, reprocessing with AI and exporting with the default configuration.",
+        action="store_true",
     )
+
+    ai_process_parser = subparsers.add_parser("ai", help="Process with AI.")
     ai_process_parser.add_argument(
         "--history", default=False, help="Reprocess ai history", action="store_true"
     )
@@ -44,12 +53,13 @@ def main():
         "--recursive", default=False, help="Run in recursive mode", action="store_true"
     )
 
-    load_txt_parser = subparsers.add_parser("load", help="Load txt file")
+    load_txt_parser = subparsers.add_parser("load", help="📦 Load txt file")
+    load_txt_parser.add_argument("-file", default=None, help="File path.")
     load_txt_parser.add_argument(
-        "-file", default="input/_chat.txt", help="File path. Default is input/_chat.txt"
+        "--verbose", default=False, help="Run in verbose mode", action="store_true"
     )
 
-    scrap_urls_parser = subparsers.add_parser("scrap-urls", help="Scrap urls")
+    scrap_urls_parser = subparsers.add_parser("scrap-urls", help="🐶 Scrap urls")
     scrap_urls_parser.add_argument(
         "--recursive", default=False, help="Run in recursive mode", action="store_true"
     )
@@ -62,8 +72,11 @@ def main():
     scrap_urls_parser.add_argument(
         "--only-parents", default=False, help="Only parents urls", action="store_true"
     )
+    scrap_urls_parser.add_argument(
+        "--verbose", default=False, help="Run in verbose mode", action="store_true"
+    )
 
-    sniff_url_parser = subparsers.add_parser("sniff-url", help="Check url")
+    sniff_url_parser = subparsers.add_parser("sniff-url", help="🐕 Sniff/Check url")
     sniff_url_parser.add_argument(
         "url", default="https://cesarcardoso.cc/", help="Url to sniff"
     )
@@ -75,7 +88,7 @@ def main():
     show_urls_parser.add_argument("--limit", default=0, help="Limit of lines to show")
     show_urls_parser.add_argument("-url", default="", help="Url to show")
 
-    export_parser = subparsers.add_parser("export", help="Export urls to csv.")
+    export_parser = subparsers.add_parser("export", help="📊🖋️ Export urls to csv.")
     export_parser.add_argument("--limit", default=0, help="Limit of lines to export")
     export_parser.add_argument(
         "--file",
@@ -89,14 +102,11 @@ def main():
         action="store_true",
     )
 
-    report_parser = subparsers.add_parser("report", help="Export urls report to csv.")
+    report_parser = subparsers.add_parser(
+        "report", help="📜🖋️ Export urls report to csv."
+    )
     merge_parser = subparsers.add_parser("merge_dbs", help="Merge databases.")
 
-    # TODO: What is that?
-    # seed_parser.set_defaults(func=seed)
-    # classify_urls_parser.set_defaults(func=classify_urls)
-    # load_txt_parser.set_defaults(func=load_txt)
-
     args = parser.parse_args()
 
     if args.command == "classify-urls":
@@ -104,7 +114,7 @@ def main():
         return
 
     if args.command == "load":
-        load_txt(args.file)
+        load_txt(file_name=args.file, verbose=args.verbose)
         return
 
     if args.command == "seed":
@@ -125,6 +135,7 @@ def main():
             ignore_valid_prefix=args.ignore_type,
             randomize=args.randomize,
             only_parents=args.only_parents,
+            verbose=args.verbose,
         )
         return
 
@@ -157,6 +168,21 @@ def main():
         merge_dbs()
         return
 
+    if args.command == "start":
+        load_txt()
+        scrap_urls(
+            recursive=True,
+            ignore_valid_prefix=True,
+            randomize=False,
+            only_parents=False,
+        )
+        if args.ai:
+            process_with_ai()
+        export_urls()
+        export_urls(csv_file="output/urls-simplified.csv", simplify=True)
+        export_report()
+        return
+
 
 if __name__ == "__main__":
     main()