tumblrbot 1.7.0__tar.gz → 1.8.0__tar.gz

{tumblrbot-1.7.0 → tumblrbot-1.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tumblrbot
-Version: 1.7.0
+Version: 1.8.0
 Summary: An updated bot that posts to Tumblr, based on your very own blog!
 Requires-Python: >= 3.13
 Description-Content-Type: text/markdown
@@ -65,6 +65,7 @@ Features:
       - Shows progress and previews the current post.
    1. [Creates examples][Examples] to fine-tune the model from your posts.
       - Filters out posts that contain more than just text data.
+      - Filters out posts that contain [configured][config] regular expressions.
       - Adds custom user messages and assistant responses to the dataset from the [configured][config] file.
    1. Filters out any posts flagged by the [OpenAI Moderation API].
    1. [Uploads examples][Fine-Tune] to [OpenAI] and begins the fine-tuning process.
@@ -83,7 +84,6 @@ Features:
 **To-Do:**
 
 - Create training data from a sample of posts (possible).
-- User-specified list of words that will filter out posts.
 
 **Known Issues:**
 
@@ -131,7 +131,7 @@ API tokens can be created here: [Tumblr Tokens].
    1. You now have access to your `consumer key` next to `Oauth Consumer Key`.
    1. Press `Show secret key` to see your `Consumer Secret`.
 
-When running this program, you will be prompted to enter all of these tokens. **The fields are password-protected, so there will be no output to the console.** If something goes wrong while entering the tokens, you can always reset them by running the program again and answering `y` to the relevant prompt.
+When running this program, you will be prompted to enter all of these tokens. If something goes wrong while entering the tokens, you can always reset them by running the program again and answering `y` to the relevant prompt.
 
 After inputting the [Tumblr] tokens, you will be given a URL that you need to open in your browser. Press `Allow`, then copy and paste the URL of the page you are redirected to into the console.
 
@@ -155,6 +155,7 @@ Specific Options:
 
    To be specific, it should follow the [JSON Lines] file format with one collection of name/value pairs (a dictionary) per line. You can validate your file using the [JSON Lines Validator].
 
+- **`filtered_words`** - During training data generation, any posts with the specified words will be removed. Word boundaries are not checked by default, so "the" will also filter out posts with "them" or "thematic". This setting supports regular expressions, so you can explicitly look for word boundaries by surrounding an entry with "\\\b", i.e. "\\\bthe\\\b". Regular expressions have to be escaped like so due to how JSON data is read in. If you are familiar with regular expressions, it could be useful for you to know that every entry is joined with a "|" which is then used to search the post content for any matches.
 - **`developer_message`** - This message is used in for fine-tuning the AI as well as generating prompts. If you change this, you will need to run the fine-tuning again with the new value before generating posts.
 - **`user_message`** - This setting is used and works in the same way as `developer_message`.
 - **`expected_epochs`** - The default value here is the default number of epochs for `base_model`. You may have to change this value if you change `base_model`. After running fine-tuning once, you will see the number of epochs used in the [fine-tuning portal] under *Hyperparameters*. This value will also be updated automatically if you run fine-tuning through this program.
@@ -163,6 +164,7 @@ Specific Options:
 - **`base_model`** - This value is used to choose the tokenizer for estimating fine-tuning costs. It is also the base model that will be fine-tuned and the model that is used to generate tags. You can find a list of options in the [fine-tuning portal] by pressing `+ Create` and opening the drop-down list for `Base Model`. Be sure to update `token_price` if you change this value.
 - **`fine_tuned_model`** - Set automatically after monitoring fine-tuning if the job has succeeded. You can read more in [fine-tuning].
 - **`tags_chance`** - This should be between 0 and 1. Setting it to 0 corresponds to a 0% chance (never) to add tags to a post. 1 corresponds to a 100% chance (always) to add tags to a post. Adding tags incurs a very small token cost.
+- **`reblog_blog_identifiers`** - Whenever a reblog is attempted, a random blog from this list will be chosen to be reblogged from.
 - **`reblog_chance`** - This setting works the same way as `tags_chance`.
 - **`reblog_user_message`** - This setting is a prefix that is directly prepended to the contents of the post being reblogged.
 

{tumblrbot-1.7.0 → tumblrbot-1.8.0}/README.md

@@ -47,6 +47,7 @@ Features:
       - Shows progress and previews the current post.
    1. [Creates examples][Examples] to fine-tune the model from your posts.
       - Filters out posts that contain more than just text data.
+      - Filters out posts that contain [configured][config] regular expressions.
       - Adds custom user messages and assistant responses to the dataset from the [configured][config] file.
    1. Filters out any posts flagged by the [OpenAI Moderation API].
    1. [Uploads examples][Fine-Tune] to [OpenAI] and begins the fine-tuning process.
@@ -65,7 +66,6 @@ Features:
 **To-Do:**
 
 - Create training data from a sample of posts (possible).
-- User-specified list of words that will filter out posts.
 
 **Known Issues:**
 
@@ -113,7 +113,7 @@ API tokens can be created here: [Tumblr Tokens].
    1. You now have access to your `consumer key` next to `Oauth Consumer Key`.
    1. Press `Show secret key` to see your `Consumer Secret`.
 
-When running this program, you will be prompted to enter all of these tokens. **The fields are password-protected, so there will be no output to the console.** If something goes wrong while entering the tokens, you can always reset them by running the program again and answering `y` to the relevant prompt.
+When running this program, you will be prompted to enter all of these tokens. If something goes wrong while entering the tokens, you can always reset them by running the program again and answering `y` to the relevant prompt.
 
 After inputting the [Tumblr] tokens, you will be given a URL that you need to open in your browser. Press `Allow`, then copy and paste the URL of the page you are redirected to into the console.
 
@@ -137,6 +137,7 @@ Specific Options:
 
    To be specific, it should follow the [JSON Lines] file format with one collection of name/value pairs (a dictionary) per line. You can validate your file using the [JSON Lines Validator].
 
+- **`filtered_words`** - During training data generation, any posts with the specified words will be removed. Word boundaries are not checked by default, so "the" will also filter out posts with "them" or "thematic". This setting supports regular expressions, so you can explicitly look for word boundaries by surrounding an entry with "\\\b", i.e. "\\\bthe\\\b". Regular expressions have to be escaped like so due to how JSON data is read in. If you are familiar with regular expressions, it could be useful for you to know that every entry is joined with a "|" which is then used to search the post content for any matches.
 - **`developer_message`** - This message is used in for fine-tuning the AI as well as generating prompts. If you change this, you will need to run the fine-tuning again with the new value before generating posts.
 - **`user_message`** - This setting is used and works in the same way as `developer_message`.
 - **`expected_epochs`** - The default value here is the default number of epochs for `base_model`. You may have to change this value if you change `base_model`. After running fine-tuning once, you will see the number of epochs used in the [fine-tuning portal] under *Hyperparameters*. This value will also be updated automatically if you run fine-tuning through this program.
@@ -145,6 +146,7 @@ Specific Options:
 - **`base_model`** - This value is used to choose the tokenizer for estimating fine-tuning costs. It is also the base model that will be fine-tuned and the model that is used to generate tags. You can find a list of options in the [fine-tuning portal] by pressing `+ Create` and opening the drop-down list for `Base Model`. Be sure to update `token_price` if you change this value.
 - **`fine_tuned_model`** - Set automatically after monitoring fine-tuning if the job has succeeded. You can read more in [fine-tuning].
 - **`tags_chance`** - This should be between 0 and 1. Setting it to 0 corresponds to a 0% chance (never) to add tags to a post. 1 corresponds to a 100% chance (always) to add tags to a post. Adding tags incurs a very small token cost.
+- **`reblog_blog_identifiers`** - Whenever a reblog is attempted, a random blog from this list will be chosen to be reblogged from.
 - **`reblog_chance`** - This setting works the same way as `tags_chance`.
 - **`reblog_user_message`** - This setting is a prefix that is directly prepended to the contents of the post being reblogged.
 

{tumblrbot-1.7.0 → tumblrbot-1.8.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tumblrbot"
-version = "1.7.0"
+version = "1.8.0"
 description = "An updated bot that posts to Tumblr, based on your very own blog!"
 readme = "README.md"
 requires-python = ">= 3.13"

{tumblrbot-1.7.0 → tumblrbot-1.8.0}/src/tumblrbot/flow/examples.py

@@ -1,3 +1,4 @@
+import re
 from collections.abc import Generator
 from itertools import batched
 from json import loads
@@ -54,11 +55,12 @@ class ExamplesWriter(FlowClass):
                 yield from data.items()
 
     def get_valid_posts(self) -> Generator[Post]:
+        pattern = re.compile("|".join(self.config.filtered_words), re.IGNORECASE)
         for data_path in self.get_data_paths():
             with data_path.open("rb") as fp:
                 for line in fp:
                     post = Post.model_validate_json(line)
-                    if post.valid_text_post():
+                    if post.valid_text_post() and not (self.config.filtered_words and pattern.search(post.get_content_text())):
                         yield post
 
     def filter_examples(self) -> None:

{tumblrbot-1.7.0 → tumblrbot-1.8.0}/src/tumblrbot/flow/generate.py

@@ -1,7 +1,10 @@
-from random import choice, random, randrange
+from collections.abc import Iterable
+from functools import cache
+from random import choice, random, sample
 from typing import override
 
 import rich
+from pydantic import ConfigDict
 from rich.prompt import IntPrompt
 
 from tumblrbot.utils.common import FlowClass, PreviewLive
@@ -9,6 +12,8 @@ from tumblrbot.utils.models import Post
 
 
 class DraftGenerator(FlowClass):
+    model_config = ConfigDict(frozen=True)  # Makes this class hashable.
+
     @override
     def main(self) -> None:
         self.config.draft_count = IntPrompt.ask("How many drafts should be generated?", default=self.config.draft_count)
@@ -28,16 +33,16 @@ class DraftGenerator(FlowClass):
         rich.print(f":chart_increasing: [bold green]Generated {self.config.draft_count} draft(s).[/] {message}")
 
     def generate_post(self) -> Post:
-        if self.config.reblog_blog_identifiers and random() < self.config.reblog_chance:  # noqa: S311
-            original = self.get_random_post()
+        if original := self.get_random_post():
             user_message = f"{self.config.reblog_user_message}\n\n{original.get_content_text()}"
         else:
             original = Post()
             user_message = self.config.user_message
-
         text = self.generate_text(user_message)
+
         if tags := self.generate_tags(text):
             tags = tags.tags
+
         return Post(
             content=[Post.Block(type="text", text=text)],
             tags=tags or [],
@@ -65,15 +70,22 @@ class DraftGenerator(FlowClass):
 
         return None
 
-    def get_random_post(self) -> Post:
-        blog_identifier = choice(self.config.reblog_blog_identifiers)  # noqa: S311
-        while True:
-            total = self.tumblr.retrieve_blog_info(blog_identifier).response.blog.posts
-            for raw_post in self.tumblr.retrieve_published_posts(
-                blog_identifier,
-                "text",
-                randrange(total),  # noqa: S311
-            ).response.posts:
-                post = Post.model_validate(raw_post)
-                if post.valid_text_post():
-                    return post
+    def get_random_post(self) -> Post | None:
+        if self.config.reblog_blog_identifiers and random() < self.config.reblog_chance:  # noqa: S311
+            blog_identifier = choice(self.config.reblog_blog_identifiers)  # noqa: S311
+            for offset in self.get_offsets(blog_identifier):
+                for raw_post in self.tumblr.retrieve_published_posts(
+                    blog_identifier,
+                    offset,
+                ).response.posts:
+                    post = Post.model_validate(raw_post)
+                    if post.valid_text_post():
+                        return post
+
+        return None
+
+    @cache  # noqa: B019 # This creates a memory leak, but it doesn't matter since this class isn't discarded until the end of the program anyways.
+    def get_offsets(self, blog_identifier: str) -> Iterable[int]:
+        total = self.tumblr.retrieve_blog_info(blog_identifier).response.blog.posts
+        # The same Iterable object is cached, so reading an element will effectively discard it. This prevents checking the same offsets twice.
+        return iter(sample(range(total), total))

{tumblrbot-1.7.0 → tumblrbot-1.8.0}/src/tumblrbot/utils/models.py

@@ -52,6 +52,7 @@ class Config(FileSyncSettings):
     # Writing Examples
     max_moderation_batch_size: PositiveInt = Field(100, description="How many posts, at most, to submit to the OpenAI moderation API. This is also capped by the API.")
     custom_prompts_file: Path = Field(Path("custom_prompts.jsonl"), description="Where to read in custom prompts from.")
+    filtered_words: list[str] = Field([], description="A case-insensitive list of disallowed words used to filter out training data. Regular expressions are allowed, but must be escaped.")
 
     # Writing Examples & Fine-Tuning
     examples_file: Path = Field(Path("examples.jsonl"), description="Where to output the examples that will be used to fine-tune the model.")
@@ -75,8 +76,8 @@ class Config(FileSyncSettings):
     tags_chance: NonNegativeFloat = Field(0.1, description="The chance to generate tags for any given post. This will use more OpenAI tokens.")
     tags_developer_message: str = Field("You will be provided with a block of text, and your task is to extract a very short list of the most important subjects from it.", description="The developer message used to generate tags.")
     reblog_blog_identifiers: list[str] = Field([], description="The identifiers of blogs that can be reblogged from when generating drafts.")
-    reblog_chance: NonNegativeFloat = Field(0.05, description="The chance to generate a reblog of a random post. This will use more OpenAI tokens.")
-    reblog_user_message: str = Field("Please write a comical Tumblr post in response to the following post from your blog:", description="The prefix for the user message used to reblog posts.")
+    reblog_chance: NonNegativeFloat = Field(0.1, description="The chance to generate a reblog of a random post. This will use more OpenAI tokens.")
+    reblog_user_message: str = Field("Please write a comical Tumblr post in response to the following post:", description="The prefix for the user message used to reblog posts.")
 
     @classmethod
     @override

{tumblrbot-1.7.0 → tumblrbot-1.8.0}/src/tumblrbot/utils/tumblr.py

@@ -1,4 +1,4 @@
-from typing import Literal, Self
+from typing import Self
 
 from requests import HTTPError, Response
 from requests_oauthlib import OAuth1Session
@@ -29,14 +29,12 @@ class TumblrSession(OAuth1Session):
     def retrieve_published_posts(
         self,
         blog_identifier: str,
-        type_: Literal["text", "quote", "link", "answer", "video", "audio", "photo", "chat"] | None = None,
         offset: int | None = None,
         after: int | None = None,
     ) -> ResponseModel:
         response = self.get(
             f"https://api.tumblr.com/v2/blog/{blog_identifier}/posts",
             params={
-                "type": type_,
                 "offset": offset,
                 "after": after,
                 "sort": "asc",