mdfb 0.1.0__tar.gz

mdfb-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Ibrahim Haji Abdi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mdfb-0.1.0/PKG-INFO

@@ -0,0 +1,121 @@
+Metadata-Version: 2.1
+Name: mdfb
+Version: 0.1.0
+Summary: 
+Author: Ibrahim Haji Abdi
+Author-email: ibrahim.hajiabdi09@gmail.com
+Requires-Python: >=3.8,<3.14
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: argparse (>=1.4.0,<2.0.0)
+Requires-Dist: atproto (>=0.0.55,<0.0.56)
+Requires-Dist: pathvalidate (>=3.2.1,<4.0.0)
+Requires-Dist: pytest (>=8.3.4,<9.0.0)
+Requires-Dist: pytest-mock (>=3.14.0,<4.0.0)
+Requires-Dist: requests-mock (>=1.12.1,<2.0.0)
+Requires-Dist: tdqm (>=0.0.1,<0.0.2)
+Requires-Dist: tenacity (>=9.0.0,<10.0.0)
+Requires-Dist: tqdm (>=4.67.1,<5.0.0)
+Description-Content-Type: text/markdown
+
+# mdfb-downloader-for-bluesky
+
+mass-downloader-for-bluesky (mdfb) is a Python cli application that can download large amounts of posts from bluesky from any given account.
+
+## Navigation
+- [mdfb-downloader-for-bluesky](#mdfb-downloader-for-bluesky)
+  - [Navigation](#navigation)
+  - [Installation](#installation)
+    - [Manual](#manual)
+  - [Usage](#usage)
+    - [Examples](#examples)
+      - [Linux](#linux)
+      - [Windows](#windows)
+    - [Naming Convention](#naming-convention)
+    - [Download Amount](#download-amount)
+    - [Note](#note)
+  - [Options](#options)
+    - [Note](#note-1)
+
+## Installation
+
+You will need [Python](https://www.python.org/downloads/) to be installed to use this CLI.
+
+You can install via pip by:
+```bash
+pip install mdfb
+```
+
+### Manual
+
+Have [Poetry](https://python-poetry.org/) installed. 
+
+Then clone the project, open a poetry shell and then install all dependencies.
+
+
+```bash
+git clone git@github.com:IbrahimHajiAbdi/mdfb-downloader-for-bluesky.git
+cd mdfb-downloader-for-bluesky
+poetry shell
+poetry install
+```
+
+## Usage
+``mdfb`` works by using the public API offered by bluesky to retrieve posts liked, reposted or posted by the desired account. 
+
+``mdfb`` will download the information for a post and the accompanying media, video or image(s). If there is no image(s) or video, it will just download the information of the post. The information of the post will be a JSON file and have lots of accompanying data, such as the text in the post, creation time of the post and author details. Currently, the retrieved posts start from the latest post to the oldest.
+
+You will need to be inside a poetry shell to use ``mdfb`` if installed manually
+
+### Examples
+
+Some example commands would be:
+
+#### Linux
+```bash
+mdfb --handle bsky.app -l 10 --like ./media/
+```
+
+#### Windows
+```bash
+mdfb --handle bsky.app -l 100 --like --repost --post ./media/
+```
+
+### Naming Convention
+``mdfb``'s naming convention is: ``"{rkey}_{handle}_{text}"``, if it is downloading a post with multiple images then the naming will be: ``"{rkey}_{handle}_{text}_{i}"``, where "i" represents the order of the images in the post ranging from 1 - 4. In addition, the filenames are limited to 256 bytes and will be truncated down to that size. 
+
+### Download Amount
+When specifying the limit, this will be true for all types of post downloaded. For example: 
+```bash
+mdfb --handle bsky.app -l 100 --like --repost --post ./media/
+```
+This would download 100 likes, reposts and post, totalling 300 posts downloaded.
+
+### Note
+The maximum number of threads is currently 3, that can be changed in the ``mdfb/utils/constants.py`` file. Furthermore, there are more constants that can be changed in that file, such as delay between each request and the number of retires before marking that post as a failure and continuing.
+
+## Options
+- ``--handle``
+  - The handle of the target account.
+- ``--did, -d``
+  - The DID of the target account. 
+- ``--limit, -l``
+  - The amount of posts that want to be downloaded. **Required**.
+- ``directory``
+  - Positional argument, where all the downloaded files are to be located. **Required**.
+- ``--threads``
+  - The amount of threads wanted to download posts more efficiently, maximum number of threads is 3.
+- ``--like``
+  - To retrieved liked posts
+- ``--repost``
+  - To retrieved reposts
+- ``--post``
+  - To retrieved posts
+### Note
+At least one of the flags: ``--like``, ``--repost``, ``--post`` is **required**.
+In addition, ``--did, -d`` and ``--handle`` are mutually exclusive, and at least one of them is **required** as well. 

mdfb-0.1.0/README.md

@@ -0,0 +1,96 @@
+# mdfb-downloader-for-bluesky
+
+mass-downloader-for-bluesky (mdfb) is a Python cli application that can download large amounts of posts from bluesky from any given account.
+
+## Navigation
+- [mdfb-downloader-for-bluesky](#mdfb-downloader-for-bluesky)
+  - [Navigation](#navigation)
+  - [Installation](#installation)
+    - [Manual](#manual)
+  - [Usage](#usage)
+    - [Examples](#examples)
+      - [Linux](#linux)
+      - [Windows](#windows)
+    - [Naming Convention](#naming-convention)
+    - [Download Amount](#download-amount)
+    - [Note](#note)
+  - [Options](#options)
+    - [Note](#note-1)
+
+## Installation
+
+You will need [Python](https://www.python.org/downloads/) to be installed to use this CLI.
+
+You can install via pip by:
+```bash
+pip install mdfb
+```
+
+### Manual
+
+Have [Poetry](https://python-poetry.org/) installed. 
+
+Then clone the project, open a poetry shell and then install all dependencies.
+
+
+```bash
+git clone git@github.com:IbrahimHajiAbdi/mdfb-downloader-for-bluesky.git
+cd mdfb-downloader-for-bluesky
+poetry shell
+poetry install
+```
+
+## Usage
+``mdfb`` works by using the public API offered by bluesky to retrieve posts liked, reposted or posted by the desired account. 
+
+``mdfb`` will download the information for a post and the accompanying media, video or image(s). If there is no image(s) or video, it will just download the information of the post. The information of the post will be a JSON file and have lots of accompanying data, such as the text in the post, creation time of the post and author details. Currently, the retrieved posts start from the latest post to the oldest.
+
+You will need to be inside a poetry shell to use ``mdfb`` if installed manually
+
+### Examples
+
+Some example commands would be:
+
+#### Linux
+```bash
+mdfb --handle bsky.app -l 10 --like ./media/
+```
+
+#### Windows
+```bash
+mdfb --handle bsky.app -l 100 --like --repost --post ./media/
+```
+
+### Naming Convention
+``mdfb``'s naming convention is: ``"{rkey}_{handle}_{text}"``, if it is downloading a post with multiple images then the naming will be: ``"{rkey}_{handle}_{text}_{i}"``, where "i" represents the order of the images in the post ranging from 1 - 4. In addition, the filenames are limited to 256 bytes and will be truncated down to that size. 
+
+### Download Amount
+When specifying the limit, this will be true for all types of post downloaded. For example: 
+```bash
+mdfb --handle bsky.app -l 100 --like --repost --post ./media/
+```
+This would download 100 likes, reposts and post, totalling 300 posts downloaded.
+
+### Note
+The maximum number of threads is currently 3, that can be changed in the ``mdfb/utils/constants.py`` file. Furthermore, there are more constants that can be changed in that file, such as delay between each request and the number of retires before marking that post as a failure and continuing.
+
+## Options
+- ``--handle``
+  - The handle of the target account.
+- ``--did, -d``
+  - The DID of the target account. 
+- ``--limit, -l``
+  - The amount of posts that want to be downloaded. **Required**.
+- ``directory``
+  - Positional argument, where all the downloaded files are to be located. **Required**.
+- ``--threads``
+  - The amount of threads wanted to download posts more efficiently, maximum number of threads is 3.
+- ``--like``
+  - To retrieved liked posts
+- ``--repost``
+  - To retrieved reposts
+- ``--post``
+  - To retrieved posts
+### Note
+At least one of the flags: ``--like``, ``--repost``, ``--post`` is **required**.
+In addition, ``--did, -d`` and ``--handle`` are mutually exclusive, and at least one of them is **required** as well. 

mdfb-0.1.0/mdfb/core/download_blobs.py

@@ -0,0 +1,106 @@
+import json
+import os
+from atproto_client.namespaces.sync_ns import ComAtprotoSyncNamespace
+from atproto_client.models.com.atproto.repo.list_records import ParamsDict
+from atproto import Client
+import re, time
+from pathvalidate import sanitize_filename
+import encodings
+import logging
+from mdfb.utils.constants import DELAY, RETRIES, EXP_WAIT_MAX, EXP_WAIT_MIN, EXP_WAIT_MULTIPLIER
+from tqdm import tqdm
+
+from tenacity import retry, stop_after_attempt, wait_exponential
+
+def download_blobs(posts: list[dict], file_path: str, progress_bar: tqdm) -> None:
+    """
+    download_blobs: for the given posts, returned from fetch_post_details(), and filepath, downloads the associated blobs for each post.
+
+    Args:
+        posts (list[dict]): post details returned from fetch_post_details()
+        file_path (str): filepath for where the files will be stored
+        progress_bar (tqdm): progress bar
+    """
+    logger = logging.getLogger(__name__)
+    for post in posts:
+        did = post["did"]
+        filename = _make_base_filename(post["rkey"], post["text"], post["handle"])
+        if "video_cid" in post:
+            video_filename = _append_extension(filename, post["mime_type"])
+            success = _get_blob_with_retries(did, post["video_cid"], video_filename, file_path, logger)
+            if success:
+                logger.info(f"Successful downloaded video: {video_filename}")
+            time.sleep(DELAY)
+
+        if "images_cid" in post:
+            for index ,image_cid in enumerate(post["images_cid"]):
+                if len(post["images_cid"]) > 1:
+                    image_filename = _append_extension(filename, post["mime_type"], index + 1)
+                else: image_filename = _append_extension(filename, post["mime_type"])
+                success = _get_blob_with_retries(did, image_cid, image_filename, file_path, logger)
+                if success:
+                    logger.info(f"Successful downloaded image: {image_filename}")
+                time.sleep(DELAY)
+                
+        with open(f"{os.path.join(file_path, filename)}.json", "wt") as json_file:
+            json.dump(post["response"], json_file, indent=4)
+        logger.info(f"Sucessful wrote file: {filename + ".json"}")
+        progress_bar.update(1)
+
+def _get_blob_with_retries(did: str, cid: str, filename: str, file_path: str, logger: logging.Logger):
+    try:
+        _get_blob(did, cid, filename, file_path, logger)
+        return True
+    except Exception:
+        logger.error(f"Error occured for downloading this file, DID: {did}, CID: {cid}, after {RETRIES} retires", exc_info=True)
+        return False
+
+@retry(
+    wait=wait_exponential(multiplier=EXP_WAIT_MULTIPLIER, min=EXP_WAIT_MIN, max=EXP_WAIT_MAX), 
+    stop=stop_after_attempt(RETRIES)
+)
+def _get_blob(did: str, cid: str, filename: str, file_path: str, logger: logging.Logger) -> bool:
+    try:
+        res = ComAtprotoSyncNamespace(Client()).get_blob(ParamsDict(
+                did=did,
+                cid=cid
+        ))
+        with open(os.path.join(file_path, filename), "wb") as file:
+            file.write(res)
+    except Exception:
+        logger.error(f"Error occured for downloading this file, DID: {did}, CID: {cid}")
+        raise 
+    
+def _make_base_filename(rkey: str, text: str, handle: str) -> str:
+    filename = f"{rkey}_{handle}_{text}"
+    filename = _truncate_filename(filename, 245)
+    return sanitize_filename(filename)
+
+def _append_extension(base_filename: str, mime_type: str = None, i: int = None) -> str:
+    filename = base_filename
+    if i:
+        filename += f"_{i}"
+    if mime_type:
+            file_type = re.search(r"\w+$", mime_type).group()
+            filename += f".{file_type}"
+    return filename
+    
+def _truncate_filename(filename: str, MAX_BYTE: int) -> str:
+    """
+    _truncate_filename: truncates the given filename to the maximum number of bytes given, or less. This is only for utf-8 encoded strings and 
+    if the filename at the maximum number of bytes is an invalid utf-8 string, then it removes one byte from the end so the string is valid.
+
+    Args:
+        filename (str): string of the filename
+        MAX_BYTE (int): maximum bytes allowed
+
+    Returns:
+        str: truncated filename such that it is within the maximum number of bytes
+    """
+    byte_len = 0
+    iter_encoder = encodings.search_function("utf-8").incrementalencoder()
+    for i, char in enumerate(filename):
+        byte_len += len(iter_encoder.encode(char))
+        if byte_len > MAX_BYTE:
+            return filename[:i]
+    return filename

mdfb-0.1.0/mdfb/core/fetch_post_details.py

@@ -0,0 +1,113 @@
+import re
+from atproto_client.namespaces.sync_ns import AppBskyFeedNamespace
+from atproto_client.models.com.atproto.repo.list_records import ParamsDict
+from atproto import Client
+from atproto.exceptions import AtProtocolError
+import time, json, logging
+
+from tenacity import RetryError, retry, stop_after_attempt, wait_exponential
+
+from mdfb.utils.helpers import get_chunk
+from mdfb.utils.constants import DELAY, EXP_WAIT_MAX, EXP_WAIT_MIN, EXP_WAIT_MULTIPLIER, RETRIES
+
+def fetch_post_details(uris: list[str]) -> list[dict]:
+    """
+    fetch_post_details: Fetches post details from the given AT-URIs
+
+    Args:
+        uris (list[str]): A list of AT-URIs
+
+    Returns:
+        list[dict]: A list of dictionaries that contain post details
+    """
+    all_post_details = []
+    logger = logging.getLogger(__name__)
+    seen_uris = set()
+    client = Client("https://public.api.bsky.app/")
+    
+    for uri_chunk in get_chunk(uris, 25):
+        logger.info(f"Fetching details from {len(uri_chunk)} URIs")
+        res = _get_post_details_with_retries(uri_chunk, client, logger)
+        if not res:
+            continue
+        records = json.loads(res.model_dump_json())
+                
+        for post in records["posts"]:
+            seen_uris.add(post["uri"])
+            post_details = {
+                "rkey": _get_rkey(post["uri"]),
+                "text": post["record"].get("text", ""),
+                "response": post,
+                **_get_author_details(post["author"])
+            }
+        
+            embed_media = post["record"].get("embed", None)
+            if not embed_media:
+                all_post_details.append(post_details)
+                continue
+
+            embed_media = embed_media.get("media", embed_media)
+            post_details.update(_extract_media(embed_media))
+            
+            logger.info("Post details retrieved for URI: %s", post["uri"])
+            all_post_details.append(post_details)
+        for uri in uri_chunk:
+            if uri not in seen_uris:
+                logger.info(f"The post associated with this URI is missing/deleted: {uri}")
+        time.sleep(DELAY)
+    return all_post_details
+
+def _extract_media(embed: dict) -> dict:
+    """
+    _extract_media: Extracts information from the media, or embede, key in the post details JSON response from the atproto API: app.bsky.feed.getPosts
+
+    Args:
+        embed (dict): The embed key from the API response of atproto API: app.bsky.feed.getPosts
+
+    Returns:
+        dict: The associated information from embed
+    """
+    media_links = {}
+    if embed.get("images"):
+        for image_obj in embed["images"]:
+            image = image_obj["image"]["ref"]["link"]
+            if "images_cid" not in media_links:
+                media_links["images_cid"] = [image]
+            else: media_links["images_cid"].append(image)
+            media_links["mime_type"] = image_obj["image"]["mime_type"]
+    if embed.get("video"):
+        media_links["video_cid"] = embed["video"]["ref"]["link"]
+        media_links["mime_type"] = embed["video"]["mime_type"]   
+    return media_links
+
+def _get_post_details_with_retries(uri_chunk: list, client: Client, logger: logging.Logger):
+    try:
+        return _get_post_details(uri_chunk, client, logger)
+    except (RetryError, AtProtocolError):
+        logger.error(f"Failure to fetch records from the URIs: {uri_chunk}", exc_info=True)
+        pass
+
+@retry(
+    wait=wait_exponential(multiplier=EXP_WAIT_MULTIPLIER, min=EXP_WAIT_MIN, max=EXP_WAIT_MAX), 
+    stop=stop_after_attempt(RETRIES)
+)
+def _get_post_details(uri_chunk: list, client: Client, logger: logging.Logger):
+    try:
+        res = AppBskyFeedNamespace(client).get_posts(ParamsDict(
+            uris=uri_chunk
+        ))
+        return res
+    except (AtProtocolError, RetryError):
+        logger.error(f"Error occurred fetching records from URIs: {uri_chunk}")
+        raise
+    
+def _get_rkey(at_uri: str) -> str:
+    match = re.search(r"\w+$", at_uri)
+    return match.group()
+
+def _get_author_details(author: dict) -> dict:
+    author_details = {}
+    author_details["did"] = author["did"]
+    author_details["handle"] = author["handle"]
+    author_details["display_name"] = author["display_name"]
+    return author_details

mdfb-0.1.0/mdfb/core/get_post_identifiers.py

@@ -0,0 +1,61 @@
+import json
+from atproto_client.namespaces.sync_ns import ComAtprotoRepoNamespace
+from atproto_client.models.com.atproto.repo.list_records import ParamsDict
+from atproto import Client
+from atproto.exceptions import AtProtocolError
+import re, time, logging
+
+from mdfb.utils.constants import DELAY
+
+
+def get_post_identifiers(did: str, limit: int, feed_type: str) -> list[str]:
+    """
+    get_post_identifiers: Gets the given amount AT-URIs of the posts wanted from the desired account 
+
+    Args:
+        did (str): DID of the target account
+        limit (int): The amount wanted to get
+        feed_type (str): The type of post wanted from the account: like, repost and post
+
+    Raises:
+        SystemExit: If there is a failure to retreive posts
+
+    Returns:
+        list[str]: A list of the desired AT-URIs
+    """
+    cursor = ""
+    post_uris = []
+    logger = logging.getLogger(__name__)
+    client = Client()
+    while limit > 0:
+        fetch_amount = min(100, limit)
+        try:
+            logger.info(f"Fetching up to {fetch_amount} posts for DID: {did}, feed_type: {feed_type}")
+            res = ComAtprotoRepoNamespace(client).list_records(ParamsDict(
+                collection=f"app.bsky.feed.{feed_type}",
+                repo=did,
+                limit=fetch_amount,
+                cursor=cursor,
+            ))  
+            res = json.loads(res.model_dump_json())
+        except AtProtocolError as e:
+            logger.error(f"Failure to fetch posts: {e}", exc_info=True) 
+            print("Failure to get fetch posts. See logs for details.")
+            raise SystemExit(1) from e
+        
+        limit -= fetch_amount
+        logger.info("Successful retrieved: %d posts, %d remaining", fetch_amount, limit)
+        records = res.get("records", {})
+        if not records:
+            logger.info(f"No more records to fetch for DID: {did}, feed_type: {feed_type}")
+            break
+        last_record_cid = re.search(r"\w+$", records[-1]["uri"])[0]
+        cursor = last_record_cid
+        for record in records:
+            if feed_type == "post":
+                uri = record["uri"]
+            else:
+                uri = record["value"]["subject"]["uri"]
+            post_uris.append(uri)
+        time.sleep(DELAY)
+    return post_uris

mdfb-0.1.0/mdfb/core/resolve_handle.py

@@ -0,0 +1,24 @@
+import logging
+from atproto_identity.handle.resolver import HandleResolver
+from atproto_identity.exceptions import DidNotFoundError
+
+def resolve_handle(handle: str) -> str:
+    """
+    resolve_handle: for a given handle, uses atproto API: com.atproto.identity.resolveHandle, to resolve the handle to a DID
+
+    Args:
+        handle (str): handle of the target account
+
+    Raises:
+        DidNotFoundError: if the handle is able to be resolved
+
+    Returns:
+        str: resolved DID
+    """
+    logger = logging.getLogger(__name__)
+    try:
+        did = HandleResolver().ensure_resolve(handle)    
+    except DidNotFoundError:
+        logger.error(f"Unable to resolve handle: {handle}")
+        raise DidNotFoundError(f"Unable to resolve handle: {handle}")
+    return did  

mdfb-0.1.0/mdfb/mdfb.py

@@ -0,0 +1,100 @@
+from argparse import ArgumentParser
+from tqdm import tqdm
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+from mdfb.core.get_post_identifiers import get_post_identifiers
+from mdfb.core.fetch_post_details import fetch_post_details
+from mdfb.core.download_blobs import download_blobs
+from mdfb.core.resolve_handle import resolve_handle
+from mdfb.utils.validation import *
+from mdfb.utils.helpers import split_list
+from mdfb.utils.logging import setup_logging
+from mdfb.utils.constants import DEFAULT_THREADS 
+
+def fetch_posts(did: str, limit: int, post_types: dict) -> list[str]:
+    post_uris = []
+    for post_type, wanted in post_types.items():
+        if wanted:
+            post_uris.extend(get_post_identifiers(did, limit, post_type))
+    return post_uris
+
+def process_posts(posts: list, num_threads: int) -> list[dict]:
+    """
+    process_posts: processes the given list of post URIs to get the post details required for downloading, can be threaded 
+
+    Args:
+        posts (list): list of URIs of the post wanted
+        num_threads (int): number of threads 
+
+    Returns:
+        list[dict]: list of dictionaries that contain post details for each post
+    """
+    posts = split_list(posts, num_threads)
+    post_details = []
+    
+    with ThreadPoolExecutor(max_workers=num_threads) as executor:
+        futures = []
+        for post_batch in posts:
+            futures.append(executor.submit(fetch_post_details, post_batch))
+        for future in as_completed(futures):
+            post_details.extend(future.result())
+    return post_details
+
+def main():
+    parser = ArgumentParser()
+
+    parser.add_argument("directory", action="store", help="Directory for where all downloaded post will be stored")
+    parser.add_argument("-l", "--limit", action="store", required=True, help="The number of posts to be downloaded")  
+    parser.add_argument("--like", action="store_true", help="To retreive liked posts")
+    parser.add_argument("--post", action="store_true", help="To retreive posts")
+    parser.add_argument("--repost", action="store_true", help="To retreive reposts")
+    parser.add_argument("--threads", action="store", help="Number of threads, maximum of 3 threads")
+
+    group = parser.add_mutually_exclusive_group(required=True)
+    group.add_argument("--did", action="store", help="The DID associated with the account")
+    group.add_argument("--handle", action="store", help="The handle for the account e.g. johnny.bsky.social")
+    
+    args = parser.parse_args()
+    try:
+        did = validate_did(args.did) if args.did else resolve_handle(args.handle)
+        directory = validate_directory(args.directory)
+        limit = validate_limit(args.limit)
+
+        setup_logging(directory)
+
+        num_threads = validate_threads(args.threads) if args.threads else DEFAULT_THREADS
+        
+        if not any([args.like, args.post, args.repost]):
+            raise ValueError("At least one flag (--like, --post, --repost) must be set.")
+        
+        post_types = {
+            "like": args.like,
+            "repost": args.repost,
+            "post": args.post
+        }
+
+
+        print("Fetching post identifiers...")
+        posts = fetch_posts(did, limit, post_types)
+
+        wanted_post_types = [post_type for post_type, wanted in post_types.items() if wanted]
+        account = args.handle if args.handle else did
+        validate_no_posts(posts, account, wanted_post_types)
+
+        print("Getting post details...")
+        post_details = process_posts(posts, num_threads)
+
+        num_of_posts = len(post_details)
+        post_links = split_list(post_details, num_threads)
+
+        with tqdm(total=num_of_posts, desc="Downloading files") as progress_bar:
+            with ThreadPoolExecutor(max_workers=num_threads) as executor:
+                futures = []
+                for batch_post_link in post_links:
+                    futures.append(executor.submit(download_blobs, batch_post_link, directory, progress_bar))
+                    
+    except Exception as e:
+        print(f"Error: {e}")
+        
+if __name__ == "__main__":
+    main()  

mdfb-0.1.0/mdfb/utils/constants.py

@@ -0,0 +1,7 @@
+MAX_THREADS = 3
+DELAY = 0.25       # in seconds
+DEFAULT_THREADS = 1
+RETRIES = 5
+EXP_WAIT_MULTIPLIER = 1
+EXP_WAIT_MAX = 16
+EXP_WAIT_MIN = 0.5 

mdfb-0.1.0/mdfb/utils/helpers.py

@@ -0,0 +1,45 @@
+def split_list(input_list: list, split_by: int) -> list[list[str]]:
+    """
+    split_list: splits the list into the given number of equal sized chunks, used for distributing data so that it can be used for threads
+
+    Args:
+        input_list (list): input list, length must be >= `split_by`
+        split_by (int): number of chunks wanted, must be >= 1
+
+    Returns:
+        list[list[str]]: a 2d array of list split into the desired number of chunks, given by `split_by`
+    """
+    if split_by < 1:
+        raise ValueError("Please enter split_by to be greater than 0")
+    part_size, remainder = divmod(len(input_list), split_by)
+
+    res = []
+    start = 0
+    for _ in range(split_by):
+        end = start + part_size
+        if remainder > 0:
+            end += 1
+            remainder -= 1
+        res.append(input_list[start:end])
+        start = end
+    return res
+
+def get_chunk(posts: list, chunk_size: int):
+    """
+    get_chunk: splits a list into smaller chunks of a specified size.
+
+    Args:
+        posts (list): the list to be divided into chunks.
+        chunk_size (int): the size of each chunk. Must be >= 1.
+
+    Yields:
+        list: a sublist containing up to `chunk_size` elements from the original list.
+
+    Raises:
+        ValueError: If `chunk_size` is less than 1.
+    """   
+    if chunk_size < 1:
+        raise ValueError("Please enter a chunk size >= 1")
+    for i in range(0, len(posts), chunk_size):
+        chunk = posts[i:i+chunk_size]
+        yield chunk

mdfb-0.1.0/mdfb/utils/logging.py

@@ -0,0 +1,13 @@
+import datetime
+import logging
+import os
+
+def setup_logging(directory: str):
+    log_name = datetime.datetime.now().strftime("mdfb_%d%m%Y_%H%M%S.log")
+    logging.basicConfig(
+        filename=os.path.join(directory, log_name), 
+        encoding='utf-8', 
+        level=logging.INFO,
+        format='[%(asctime)s] %(message)s', 
+        datefmt='%m/%d/%Y %I:%M:%S %p',
+    )

mdfb-0.1.0/mdfb/utils/validation.py

@@ -0,0 +1,37 @@
+import logging
+import os
+import re
+from mdfb.utils.constants import MAX_THREADS
+
+def validate_directory(directory: str) -> str:
+    if not os.path.exists(directory) or not os.path.isdir(directory):
+        raise ValueError("The given filepath is either not valid or does not exist")
+    return directory.rstrip("/")
+
+def validate_limit(limit: str) -> int:
+    if not limit.isdigit():
+        raise ValueError("The given limit is not a integer")
+    elif int(limit) < 1:
+        raise ValueError("The given limit is 0 or less")
+    return int(limit)
+
+def validate_did(did: str) -> str:
+    if not re.search(r"^did:[a-z]+:[a-zA-Z0-9._:%-]*[a-zA-Z0-9._-]$", did):
+        raise ValueError("The given DID is not valid")
+    return did
+
+def validate_threads(threads: str) -> int:
+    if not threads.isdigit():
+        raise ValueError("Please enter an integer")
+    threads = int(threads)
+    if threads > MAX_THREADS:
+        logging.info(f"Entered {threads} threads, but the maximum is {MAX_THREADS}. Setting to {MAX_THREADS} threads")
+        print(f"Entered {threads} threads, but the maximum is {MAX_THREADS}. Setting to {MAX_THREADS} threads.")
+        threads = MAX_THREADS
+    if threads < 1:
+        raise ValueError("Please set threads to 1 or more")
+    return threads
+
+def validate_no_posts(posts: list, account: str, post_types: list):
+    if not posts:
+        raise ValueError(f"There are no posts associated with account: {account}, for post_type(s): {post_types}")

mdfb-0.1.0/pyproject.toml

@@ -0,0 +1,45 @@
+[tool.poetry]
+name = "mdfb"
+version = "0.1.0"
+description = ""
+authors = ["Ibrahim Haji Abdi <ibrahim.hajiabdi09@gmail.com>"]
+readme = "README.md"
+
+[tool.poetry.scripts]
+mdfb = "mdfb.mdfb:main"
+
+[tool.poetry.dependencies]
+python = "<3.14,>=3.8"
+atproto = "^0.0.55"
+tdqm = "^0.0.1"
+argparse = "^1.4.0"
+pathvalidate = "^3.2.1"
+pytest = "^8.3.4"
+tqdm = "^4.67.1"
+requests-mock = "^1.12.1"
+pytest-mock = "^3.14.0"
+tenacity = "^9.0.0"
+
+[project]
+name = "mdfb"
+version = "0.1.0"
+authors = [
+  { name="Ibrahim Haji Abdi", email="ibrahim.hajiabdi09@gmail.com" },
+]
+description = "A CLI for downloading posts in bulk from Bluesky from specified a account"
+readme = "README.md"
+requires-python = "<3.14,>=3.8"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Environment :: Console"
+]
+
+[project.urls]
+Homepage = "https://github.com/IbrahimHajiAbdi/mass-downloader-for-bluesky"
+Issues = "https://github.com/IbrahimHajiAbdi/mass-downloader-for-bluesky/issues"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"