tumblrbot 1.6.0__tar.gz → 1.8.0__tar.gz

{tumblrbot-1.6.0 → tumblrbot-1.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tumblrbot
-Version: 1.6.0
+Version: 1.8.0
 Summary: An updated bot that posts to Tumblr, based on your very own blog!
 Requires-Python: >= 3.13
 Description-Content-Type: text/markdown
@@ -36,6 +36,8 @@ Project-URL: Source, https://github.com/MaidScientistIzutsumiMarin/tumblrbot
 
 [Tumblr]: https://tumblr.com
 [Tumblr Tokens]: https://tumblr.com/oauth/apps
+[Tumblr API Documentation on Blog Identifiers]: https://tumblr.com/docs/en/api/v2#blog-identifiers
+[Tumblr API Documentation on Rate Limits]: https://tumblr.com/docs/en/api/v2#rate-limits
 
 [Download]: src/tumblrbot/flow/download.py
 [Examples]: src/tumblrbot/flow/examples.py
@@ -58,11 +60,12 @@ Features:
    1. Asks for [OpenAI] and [Tumblr] tokens.
       - Stores API tokens using [keyring].
    1. Retrieves [Tumblr] [OAuth] tokens.
-   1. [Downloads posts][Download] from the [configured][config] [Tumblr] blogs.
+   1. [Downloads posts][Download] from the [configured][config] blogs.
       - Skips redownloading already downloaded posts.
       - Shows progress and previews the current post.
    1. [Creates examples][Examples] to fine-tune the model from your posts.
       - Filters out posts that contain more than just text data.
+      - Filters out posts that contain [configured][config] regular expressions.
       - Adds custom user messages and assistant responses to the dataset from the [configured][config] file.
    1. Filters out any posts flagged by the [OpenAI Moderation API].
    1. [Uploads examples][Fine-Tune] to [OpenAI] and begins the fine-tuning process.
@@ -70,22 +73,23 @@ Features:
       - Resumes monitoring the same fine-tuning process when restarted.
       - Deletes the uploaded examples file if fine-tuning does not succeed (optional).
       - Stores the output model automatically when fine-tuning is completed.
-   1. [Generates and uploads posts][Generate] to the [configured][config] [Tumblr] blog using the [configured][config] fine-tuned model.
+   1. [Generates and uploads posts][Generate] to the [configured][config] blog using the [configured][config] fine-tuned model.
       - Creates tags by extracting keywords at the [configured][config] frequency using the [configured][config] model.
-      - Uploads posts as drafts to the [configured][config] [Tumblr] blog.
-      - Reblog posts at the [configured][config] frequency.
+      - Uploads posts as drafts to the [configured][config] blog.
+      - Reblogs posts from the [configured][config] blogs at the [configured][config] frequency.
       - Shows progress and previews the current post.
 - Colorful output, progress bars, and post previews using [rich].
 - Automatically keeps the [config] file up-to-date and recreates it if missing.
 
 **To-Do:**
 
-- ...
+- Create training data from a sample of posts (possible).
 
 **Known Issues:**
 
 - Sometimes, you will get an error about the training file not being found when starting fine-tuning. We do not currently have a fix or workaround for this. You should instead use the online portal for fine-tuning if this continues to happen. Read more in [fine-tuning].
 - Post counts are incorrect when downloading posts. We are not certain what the cause of this is, but our tests suggest this is a [Tumblr] API problem that is giving inaccurate numbers.
+- During post downloading or post generation, you may receive a "Limit Exceeded" error message from the [Tumblr] API. This is caused by server-side rate-limiting by [Tumblr]. The only workaround is trying again or waiting for a period of time before retrying. In most cases, you either have to wait for a minute or an hour for the limits to reset. You can read more about the limits in the [Tumblr API documentation on rate limits].
 
 **Please submit an issue or contact us for features you want added/reimplemented.**
 
@@ -127,7 +131,7 @@ API tokens can be created here: [Tumblr Tokens].
    1. You now have access to your `consumer key` next to `Oauth Consumer Key`.
    1. Press `Show secret key` to see your `Consumer Secret`.
 
-When running this program, you will be prompted to enter all of these tokens. **The fields are password-protected, so there will be no output to the console.** If something goes wrong while entering the tokens, you can always reset them by running the program again and answering `y` to the relevant prompt.
+When running this program, you will be prompted to enter all of these tokens. If something goes wrong while entering the tokens, you can always reset them by running the program again and answering `y` to the relevant prompt.
 
 After inputting the [Tumblr] tokens, you will be given a URL that you need to open in your browser. Press `Allow`, then copy and paste the URL of the page you are redirected to into the console.
 
@@ -137,6 +141,10 @@ All config options can be found in `config.toml` after running the program once.
 
 All file options can include directories that will be created when the program is run.
 
+All config options that involve *blog identifiers* expect any version of a blog URL, which is explained in more detail in the [Tumblr API documentation on blog identifiers].
+
+Specific Options:
+
 - `custom_prompts_file` This file should follow the following file format:
 
    ```json
@@ -147,14 +155,18 @@ All file options can include directories that will be created when the program i
 
    To be specific, it should follow the [JSON Lines] file format with one collection of name/value pairs (a dictionary) per line. You can validate your file using the [JSON Lines Validator].
 
+- **`filtered_words`** - During training data generation, any posts with the specified words will be removed. Word boundaries are not checked by default, so "the" will also filter out posts with "them" or "thematic". This setting supports regular expressions, so you can explicitly look for word boundaries by surrounding an entry with "\\\b", i.e. "\\\bthe\\\b". Regular expressions have to be escaped like so due to how JSON data is read in. If you are familiar with regular expressions, it could be useful for you to know that every entry is joined with a "|" which is then used to search the post content for any matches.
 - **`developer_message`** - This message is used in for fine-tuning the AI as well as generating prompts. If you change this, you will need to run the fine-tuning again with the new value before generating posts.
-- **`user_message`** - This message is used in the same way as `developer_message` and should be treated the same.
+- **`user_message`** - This setting is used and works in the same way as `developer_message`.
 - **`expected_epochs`** - The default value here is the default number of epochs for `base_model`. You may have to change this value if you change `base_model`. After running fine-tuning once, you will see the number of epochs used in the [fine-tuning portal] under *Hyperparameters*. This value will also be updated automatically if you run fine-tuning through this program.
 - **`token_price`** - The default value here is the default token price for `base_model`. You can find the up-to-date value in [OpenAI Pricing], in the *Training* column.
 - **`job_id`** - If there is any value here, this program will resume monitoring the corresponding job, instead of starting a new one. This gets set when starting the fine-tuning and is cleared when it is completed. You can read more in [fine-tuning].
 - **`base_model`** - This value is used to choose the tokenizer for estimating fine-tuning costs. It is also the base model that will be fine-tuned and the model that is used to generate tags. You can find a list of options in the [fine-tuning portal] by pressing `+ Create` and opening the drop-down list for `Base Model`. Be sure to update `token_price` if you change this value.
 - **`fine_tuned_model`** - Set automatically after monitoring fine-tuning if the job has succeeded. You can read more in [fine-tuning].
 - **`tags_chance`** - This should be between 0 and 1. Setting it to 0 corresponds to a 0% chance (never) to add tags to a post. 1 corresponds to a 100% chance (always) to add tags to a post. Adding tags incurs a very small token cost.
+- **`reblog_blog_identifiers`** - Whenever a reblog is attempted, a random blog from this list will be chosen to be reblogged from.
+- **`reblog_chance`** - This setting works the same way as `tags_chance`.
+- **`reblog_user_message`** - This setting is a prefix that is directly prepended to the contents of the post being reblogged.
 
 ## Manual Fine-Tuning
 

{tumblrbot-1.6.0 → tumblrbot-1.8.0}/README.md

@@ -18,6 +18,8 @@
 
 [Tumblr]: https://tumblr.com
 [Tumblr Tokens]: https://tumblr.com/oauth/apps
+[Tumblr API Documentation on Blog Identifiers]: https://tumblr.com/docs/en/api/v2#blog-identifiers
+[Tumblr API Documentation on Rate Limits]: https://tumblr.com/docs/en/api/v2#rate-limits
 
 [Download]: src/tumblrbot/flow/download.py
 [Examples]: src/tumblrbot/flow/examples.py
@@ -40,11 +42,12 @@ Features:
    1. Asks for [OpenAI] and [Tumblr] tokens.
       - Stores API tokens using [keyring].
    1. Retrieves [Tumblr] [OAuth] tokens.
-   1. [Downloads posts][Download] from the [configured][config] [Tumblr] blogs.
+   1. [Downloads posts][Download] from the [configured][config] blogs.
       - Skips redownloading already downloaded posts.
       - Shows progress and previews the current post.
    1. [Creates examples][Examples] to fine-tune the model from your posts.
       - Filters out posts that contain more than just text data.
+      - Filters out posts that contain [configured][config] regular expressions.
       - Adds custom user messages and assistant responses to the dataset from the [configured][config] file.
    1. Filters out any posts flagged by the [OpenAI Moderation API].
    1. [Uploads examples][Fine-Tune] to [OpenAI] and begins the fine-tuning process.
@@ -52,22 +55,23 @@ Features:
       - Resumes monitoring the same fine-tuning process when restarted.
       - Deletes the uploaded examples file if fine-tuning does not succeed (optional).
       - Stores the output model automatically when fine-tuning is completed.
-   1. [Generates and uploads posts][Generate] to the [configured][config] [Tumblr] blog using the [configured][config] fine-tuned model.
+   1. [Generates and uploads posts][Generate] to the [configured][config] blog using the [configured][config] fine-tuned model.
       - Creates tags by extracting keywords at the [configured][config] frequency using the [configured][config] model.
-      - Uploads posts as drafts to the [configured][config] [Tumblr] blog.
-      - Reblog posts at the [configured][config] frequency.
+      - Uploads posts as drafts to the [configured][config] blog.
+      - Reblogs posts from the [configured][config] blogs at the [configured][config] frequency.
       - Shows progress and previews the current post.
 - Colorful output, progress bars, and post previews using [rich].
 - Automatically keeps the [config] file up-to-date and recreates it if missing.
 
 **To-Do:**
 
-- ...
+- Create training data from a sample of posts (possible).
 
 **Known Issues:**
 
 - Sometimes, you will get an error about the training file not being found when starting fine-tuning. We do not currently have a fix or workaround for this. You should instead use the online portal for fine-tuning if this continues to happen. Read more in [fine-tuning].
 - Post counts are incorrect when downloading posts. We are not certain what the cause of this is, but our tests suggest this is a [Tumblr] API problem that is giving inaccurate numbers.
+- During post downloading or post generation, you may receive a "Limit Exceeded" error message from the [Tumblr] API. This is caused by server-side rate-limiting by [Tumblr]. The only workaround is trying again or waiting for a period of time before retrying. In most cases, you either have to wait for a minute or an hour for the limits to reset. You can read more about the limits in the [Tumblr API documentation on rate limits].
 
 **Please submit an issue or contact us for features you want added/reimplemented.**
 
@@ -109,7 +113,7 @@ API tokens can be created here: [Tumblr Tokens].
    1. You now have access to your `consumer key` next to `Oauth Consumer Key`.
    1. Press `Show secret key` to see your `Consumer Secret`.
 
-When running this program, you will be prompted to enter all of these tokens. **The fields are password-protected, so there will be no output to the console.** If something goes wrong while entering the tokens, you can always reset them by running the program again and answering `y` to the relevant prompt.
+When running this program, you will be prompted to enter all of these tokens. If something goes wrong while entering the tokens, you can always reset them by running the program again and answering `y` to the relevant prompt.
 
 After inputting the [Tumblr] tokens, you will be given a URL that you need to open in your browser. Press `Allow`, then copy and paste the URL of the page you are redirected to into the console.
 
@@ -119,6 +123,10 @@ All config options can be found in `config.toml` after running the program once.
 
 All file options can include directories that will be created when the program is run.
 
+All config options that involve *blog identifiers* expect any version of a blog URL, which is explained in more detail in the [Tumblr API documentation on blog identifiers].
+
+Specific Options:
+
 - `custom_prompts_file` This file should follow the following file format:
 
    ```json
@@ -129,14 +137,18 @@ All file options can include directories that will be created when the program i
 
    To be specific, it should follow the [JSON Lines] file format with one collection of name/value pairs (a dictionary) per line. You can validate your file using the [JSON Lines Validator].
 
+- **`filtered_words`** - During training data generation, any posts with the specified words will be removed. Word boundaries are not checked by default, so "the" will also filter out posts with "them" or "thematic". This setting supports regular expressions, so you can explicitly look for word boundaries by surrounding an entry with "\\\b", i.e. "\\\bthe\\\b". Regular expressions have to be escaped like so due to how JSON data is read in. If you are familiar with regular expressions, it could be useful for you to know that every entry is joined with a "|" which is then used to search the post content for any matches.
 - **`developer_message`** - This message is used in for fine-tuning the AI as well as generating prompts. If you change this, you will need to run the fine-tuning again with the new value before generating posts.
-- **`user_message`** - This message is used in the same way as `developer_message` and should be treated the same.
+- **`user_message`** - This setting is used and works in the same way as `developer_message`.
 - **`expected_epochs`** - The default value here is the default number of epochs for `base_model`. You may have to change this value if you change `base_model`. After running fine-tuning once, you will see the number of epochs used in the [fine-tuning portal] under *Hyperparameters*. This value will also be updated automatically if you run fine-tuning through this program.
 - **`token_price`** - The default value here is the default token price for `base_model`. You can find the up-to-date value in [OpenAI Pricing], in the *Training* column.
 - **`job_id`** - If there is any value here, this program will resume monitoring the corresponding job, instead of starting a new one. This gets set when starting the fine-tuning and is cleared when it is completed. You can read more in [fine-tuning].
 - **`base_model`** - This value is used to choose the tokenizer for estimating fine-tuning costs. It is also the base model that will be fine-tuned and the model that is used to generate tags. You can find a list of options in the [fine-tuning portal] by pressing `+ Create` and opening the drop-down list for `Base Model`. Be sure to update `token_price` if you change this value.
 - **`fine_tuned_model`** - Set automatically after monitoring fine-tuning if the job has succeeded. You can read more in [fine-tuning].
 - **`tags_chance`** - This should be between 0 and 1. Setting it to 0 corresponds to a 0% chance (never) to add tags to a post. 1 corresponds to a 100% chance (always) to add tags to a post. Adding tags incurs a very small token cost.
+- **`reblog_blog_identifiers`** - Whenever a reblog is attempted, a random blog from this list will be chosen to be reblogged from.
+- **`reblog_chance`** - This setting works the same way as `tags_chance`.
+- **`reblog_user_message`** - This setting is a prefix that is directly prepended to the contents of the post being reblogged.
 
 ## Manual Fine-Tuning
 

{tumblrbot-1.6.0 → tumblrbot-1.8.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tumblrbot"
-version = "1.6.0"
+version = "1.8.0"
 description = "An updated bot that posts to Tumblr, based on your very own blog!"
 readme = "README.md"
 requires-python = ">= 3.13"

{tumblrbot-1.6.0 → tumblrbot-1.8.0}/src/tumblrbot/flow/examples.py

@@ -1,3 +1,4 @@
+import re
 from collections.abc import Generator
 from itertools import batched
 from json import loads
@@ -54,11 +55,12 @@ class ExamplesWriter(FlowClass):
                 yield from data.items()
 
     def get_valid_posts(self) -> Generator[Post]:
+        pattern = re.compile("|".join(self.config.filtered_words), re.IGNORECASE)
         for data_path in self.get_data_paths():
             with data_path.open("rb") as fp:
                 for line in fp:
                     post = Post.model_validate_json(line)
-                    if post.valid_text_post():
+                    if post.valid_text_post() and not (self.config.filtered_words and pattern.search(post.get_content_text())):
                         yield post
 
     def filter_examples(self) -> None:

{tumblrbot-1.6.0 → tumblrbot-1.8.0}/src/tumblrbot/flow/generate.py

@@ -1,7 +1,10 @@
-from random import random, randrange
+from collections.abc import Iterable
+from functools import cache
+from random import choice, random, sample
 from typing import override
 
 import rich
+from pydantic import ConfigDict
 from rich.prompt import IntPrompt
 
 from tumblrbot.utils.common import FlowClass, PreviewLive
@@ -9,6 +12,8 @@ from tumblrbot.utils.models import Post
 
 
 class DraftGenerator(FlowClass):
+    model_config = ConfigDict(frozen=True)  # Makes this class hashable.
+
     @override
     def main(self) -> None:
         self.config.draft_count = IntPrompt.ask("How many drafts should be generated?", default=self.config.draft_count)
@@ -28,16 +33,16 @@ class DraftGenerator(FlowClass):
         rich.print(f":chart_increasing: [bold green]Generated {self.config.draft_count} draft(s).[/] {message}")
 
     def generate_post(self) -> Post:
-        if random() < self.config.reblog_chance:  # noqa: S311
-            original = self.get_random_post()
+        if original := self.get_random_post():
             user_message = f"{self.config.reblog_user_message}\n\n{original.get_content_text()}"
         else:
             original = Post()
             user_message = self.config.user_message
-
         text = self.generate_text(user_message)
+
         if tags := self.generate_tags(text):
             tags = tags.tags
+
         return Post(
             content=[Post.Block(type="text", text=text)],
             tags=tags or [],
@@ -65,10 +70,22 @@ class DraftGenerator(FlowClass):
 
         return None
 
-    def get_random_post(self) -> Post:
-        total = self.tumblr.retrieve_blog_info(self.config.upload_blog_identifier).response.blog.posts
-        post = self.tumblr.retrieve_published_posts(
-            self.config.upload_blog_identifier,
-            offset=randrange(total),  # noqa: S311
-        ).response.posts[0]
-        return Post.model_validate(post)
+    def get_random_post(self) -> Post | None:
+        if self.config.reblog_blog_identifiers and random() < self.config.reblog_chance:  # noqa: S311
+            blog_identifier = choice(self.config.reblog_blog_identifiers)  # noqa: S311
+            for offset in self.get_offsets(blog_identifier):
+                for raw_post in self.tumblr.retrieve_published_posts(
+                    blog_identifier,
+                    offset,
+                ).response.posts:
+                    post = Post.model_validate(raw_post)
+                    if post.valid_text_post():
+                        return post
+
+        return None
+
+    @cache  # noqa: B019 # This creates a memory leak, but it doesn't matter since this class isn't discarded until the end of the program anyways.
+    def get_offsets(self, blog_identifier: str) -> Iterable[int]:
+        total = self.tumblr.retrieve_blog_info(blog_identifier).response.blog.posts
+        # The same Iterable object is cached, so reading an element will effectively discard it. This prevents checking the same offsets twice.
+        return iter(sample(range(total), total))

{tumblrbot-1.6.0 → tumblrbot-1.8.0}/src/tumblrbot/utils/models.py

@@ -46,12 +46,13 @@ class Config(FileSyncSettings):
     toml_file: ClassVar = Path("config.toml")
 
     # Downloading Posts & Writing Examples
-    download_blog_identifiers: list[str] = Field([], description="The identifiers of the blogs which post data will be downloaded from. These must be blogs associated with the same account as the configured Tumblr secret tokens.")
+    download_blog_identifiers: list[str] = Field([], description="The identifiers of the blogs which post data will be downloaded from.")
     data_directory: Path = Field(Path("data"), description="Where to store downloaded post data.")
 
     # Writing Examples
     max_moderation_batch_size: PositiveInt = Field(100, description="How many posts, at most, to submit to the OpenAI moderation API. This is also capped by the API.")
     custom_prompts_file: Path = Field(Path("custom_prompts.jsonl"), description="Where to read in custom prompts from.")
+    filtered_words: list[str] = Field([], description="A case-insensitive list of disallowed words used to filter out training data. Regular expressions are allowed, but must be escaped.")
 
     # Writing Examples & Fine-Tuning
     examples_file: Path = Field(Path("examples.jsonl"), description="Where to output the examples that will be used to fine-tune the model.")
@@ -72,10 +73,11 @@ class Config(FileSyncSettings):
     # Generating
     upload_blog_identifier: str = Field("", description="The identifier of the blog which generated drafts will be uploaded to. This must be a blog associated with the same account as the configured Tumblr secret tokens.")
     draft_count: PositiveInt = Field(150, description="The number of drafts to process. This will affect the number of tokens used with OpenAI")
-    tags_chance: NonNegativeFloat = Field(0.1, description="The chance to generate tags for any given post. This will incur extra calls to OpenAI.")
+    tags_chance: NonNegativeFloat = Field(0.1, description="The chance to generate tags for any given post. This will use more OpenAI tokens.")
     tags_developer_message: str = Field("You will be provided with a block of text, and your task is to extract a very short list of the most important subjects from it.", description="The developer message used to generate tags.")
-    reblog_chance: NonNegativeFloat = Field(0.05, description="The chance to generate a reblog of a random post.")
-    reblog_user_message: str = Field("Please write a comical Tumblr post in response to the following Tumblr post:", description="The prefix for the user message used to reblog posts.")
+    reblog_blog_identifiers: list[str] = Field([], description="The identifiers of blogs that can be reblogged from when generating drafts.")
+    reblog_chance: NonNegativeFloat = Field(0.1, description="The chance to generate a reblog of a random post. This will use more OpenAI tokens.")
+    reblog_user_message: str = Field("Please write a comical Tumblr post in response to the following post:", description="The prefix for the user message used to reblog posts.")
 
     @classmethod
     @override

{tumblrbot-1.6.0 → tumblrbot-1.8.0}/src/tumblrbot/utils/tumblr.py

@@ -26,7 +26,12 @@ class TumblrSession(OAuth1Session):
         response = self.get(f"https://api.tumblr.com/v2/blog/{blog_identifier}/info")
         return ResponseModel.model_validate_json(response.text)
 
-    def retrieve_published_posts(self, blog_identifier: str, offset: int | None = None, after: int | None = None) -> ResponseModel:
+    def retrieve_published_posts(
+        self,
+        blog_identifier: str,
+        offset: int | None = None,
+        after: int | None = None,
+    ) -> ResponseModel:
         response = self.get(
             f"https://api.tumblr.com/v2/blog/{blog_identifier}/posts",
             params={