tumblrbot 1.9.0__tar.gz → 1.9.2__tar.gz

{tumblrbot-1.9.0 → tumblrbot-1.9.2}/.gitignore

@@ -1,10 +1,3 @@
-# Custom
-data
-*.lnk
-config.toml
-custom_prompts.jsonl
-examples.jsonl
-
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[codz]
@@ -138,6 +131,19 @@ __pypackages__/
 celerybeat-schedule
 celerybeat.pid
 
+# Redis
+*.rdb
+*.aof
+*.pid
+
+# RabbitMQ
+mnesia/
+rabbitmq/
+rabbitmq-data/
+
+# ActiveMQ
+activemq-data/
+
 # SageMath parsed files
 *.sage.py
 
@@ -189,9 +195,9 @@ cython_debug/
 .abstra/
 
 # Visual Studio Code
-#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore
 #  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
-#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  and can be added to the global gitignore or merged into this file. However, if you prefer,
 #  you could uncomment the following to ignore the entire vscode folder
 .vscode/
 
@@ -207,4 +213,9 @@ marimo/_lsp/
 __marimo__/
 
 # Streamlit
-.streamlit/secrets.toml
+.streamlit/secrets.toml
+
+data
+*.jsonl
+config.toml
+tumblrbot.exe.lnk

{tumblrbot-1.9.0 → tumblrbot-1.9.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tumblrbot
-Version: 1.9.0
+Version: 1.9.2
 Summary: An updated bot that posts to Tumblr, based on your very own blog!
 Requires-Python: >= 3.13
 Description-Content-Type: text/markdown
@@ -39,13 +39,15 @@ Project-URL: Source, https://github.com/MaidScientistIzutsumiMarin/tumblrbot
 [Tumblr API Documentation on Blog Identifiers]: https://tumblr.com/docs/en/api/v2#blog-identifiers
 [Tumblr API Documentation on Rate Limits]: https://tumblr.com/docs/en/api/v2#rate-limits
 
+[Format String]: https://docs.python.org/3/library/string.html#format-string-syntax
+
 [Download]: src/tumblrbot/flow/download.py
 [Examples]: src/tumblrbot/flow/examples.py
 [Fine-Tune]: src/tumblrbot/flow/fine_tune.py
 [Generate]: src/tumblrbot/flow/generate.py
 [Main]: src/tumblrbot/__main__.py
 
-[Config]: #configuration
+[Configurable]: #configuration
 [Fine-Tuning]: #manual-fine-tuning
 [![PyPI - Version](https://img.shields.io/pypi/v/tumblrbot)](https://python.org/pypi/tumblrbot)
 
@@ -60,33 +62,34 @@ Features:
    1. Asks for [OpenAI] and [Tumblr] tokens.
       - Stores API tokens using [keyring].
    1. Retrieves [Tumblr] [OAuth] tokens.
-   1. [Downloads posts][Download] from the [configured][config] blogs.
+   1. [Downloads posts][Download] from specified blogs ([configurable]).
       - Skips redownloading already downloaded posts.
       - Shows progress and previews the current post.
-   1. [Creates examples][Examples] to fine-tune the model from your posts.
+   1. [Creates examples][Examples] to fine-tune the model from the downloaded posts.
       - Filters out posts that contain more than just text data.
-      - Filters out posts that contain [configured][config] regular expressions.
-      - Only uses the most recent posts from each blog as [configured][config].
-      - Adds custom user messages and assistant responses to the dataset from the [configured][config] file.
+      - Filters out posts that contain regular expressions ([configurable]).
+      - Only uses the most recent posts from each blog ([configurable]).
+      - Adds custom user messages and assistant responses to the dataset ([configurable]).
    1. Filters out any posts flagged by the [OpenAI Moderation API].
    1. [Uploads examples][Fine-Tune] to [OpenAI] and begins the fine-tuning process.
-      - Provides cost estimates if the currently saved examples are used to fine-tune the [configured][config] model.
+      - Provides cost estimates if the currently saved examples are used to fine-tune a base model ([configurable]).
       - Resumes monitoring the same fine-tuning process when restarted.
       - Deletes the uploaded examples file if fine-tuning does not succeed (optional).
       - Stores the output model automatically when fine-tuning is completed.
-   1. [Generates and uploads posts][Generate] to the [configured][config] blog using the [configured][config] fine-tuned model.
-      - Creates tags by extracting keywords at the [configured][config] frequency using the [configured][config] model.
-      - Uploads posts as drafts to the [configured][config] blog.
-      - Reblogs posts from the [configured][config] blogs at the [configured][config] frequency.
+   1. [Generates and uploads posts][Generate] to a blog using the fine-tuned model ([configurable]).
+      - Creates tags by extracting keywords using the base model ([configurable]).
+      - Uploads posts as drafts.
+      - Reblogs posts from allowed blogs ([configurable]).
       - Shows progress and previews the current post.
 - Colorful output, progress bars, and post previews using [rich].
-- Automatically keeps the [config] file up-to-date and recreates it if missing.
+- Automatically keeps the [config][configurable] file up-to-date and recreates it if missing (without overriding user settings).
 
 **Known Issues:**
 
 - Sometimes, you will get an error about the training file not being found when starting fine-tuning. We do not currently have a fix or workaround for this. You should instead use the online portal for fine-tuning if this continues to happen. Read more in [fine-tuning].
 - Post counts are incorrect when downloading posts. We are not certain what the cause of this is, but our tests suggest this is a [Tumblr] API problem that is giving inaccurate numbers.
 - During post downloading or post generation, you may receive a "Limit Exceeded" error message from the [Tumblr] API. This is caused by server-side rate-limiting by [Tumblr]. The only workaround is trying again or waiting for a period of time before retrying. In most cases, you either have to wait for a minute or an hour for the limits to reset. You can read more about the limits in the [Tumblr API documentation on rate limits].
+- Similar to the above issue, you may sometimes get a message saying your IP is blocked. This block is temporary and probably follows the same rules as previously described.
 
 **Please submit an issue or contact us for features you want added/reimplemented.**
 
@@ -103,7 +106,7 @@ Features:
 
 ## Usage
 
-Run `tumblrbot` from anywhere. Run `tumblrbot --help` for command-line options. Every command-line option corresponds to a value from the [config].
+Run `tumblrbot` from anywhere. Run `tumblrbot --help` for command-line options. Every command-line option corresponds to a value from the [config][configurable].
 
 ## Obtaining Tokens
 
@@ -171,16 +174,17 @@ Specific Options:
 - **`tags_chance`** - This should be between 0 and 1. Setting it to 0 corresponds to a 0% chance (never) to add tags to a post. 1 corresponds to a 100% chance (always) to add tags to a post. Adding tags incurs a very small token cost.
 - **`reblog_blog_identifiers`** - Whenever a reblog is attempted, a random blog from this list will be chosen to be reblogged from.
 - **`reblog_chance`** - This setting works the same way as `tags_chance`.
-- **`reblog_user_message`** - This setting is a prefix that is directly prepended to the contents of the post being reblogged.
+- **`reblog_user_message`** - This setting is a [format string]. The only argument it is formatted with is the content of the post being reblogged. In simple terms, the `{}` will be replaced with said content. Alternatively, you can leave out the `{}` so that the reblogged post is appended to the end.
+  - *Note: The bot is only given the latest message in a reblog chain due to the required complexity and added costs of including the entire chain.*
 
 ## Manual Fine-Tuning
 
 You can manually upload the examples file to [OpenAI] and start the fine-tuning here: [fine-tuning portal].
 
 1. Press `+ Create`.
-1. Select the desired `Base Model` from the dropdown. This should ideally match the model set in the [config].
-1. Upload the generated examples file to the section under `Training data`. You can find the path for this in the [config].
+1. Select the desired `Base Model` from the dropdown. This should ideally match the model set in the [config][configurable].
+1. Upload the generated examples file to the section under `Training data`. You can find the path for this in the [config][configurable].
 1. Press `Create`.
-1. (Optional) Copy the value next to `Job ID` and paste it into the [config] under `job_id`. You can then run the program and monitor its progress as usual.
-1. If you do not do the above, you will have to copy the value next to `Output model` once the job is complete and paste it into the [config] under `fine_tuned_model`.
+1. (Optional) Copy the value next to `Job ID` and paste it into the [config][configurable] under `job_id`. You can then run the program and monitor its progress as usual.
+1. If you do not do the above, you will have to copy the value next to `Output model` once the job is complete and paste it into the [config][configurable] under `fine_tuned_model`.
 

{tumblrbot-1.9.0 → tumblrbot-1.9.2}/README.md

@@ -21,13 +21,15 @@
 [Tumblr API Documentation on Blog Identifiers]: https://tumblr.com/docs/en/api/v2#blog-identifiers
 [Tumblr API Documentation on Rate Limits]: https://tumblr.com/docs/en/api/v2#rate-limits
 
+[Format String]: https://docs.python.org/3/library/string.html#format-string-syntax
+
 [Download]: src/tumblrbot/flow/download.py
 [Examples]: src/tumblrbot/flow/examples.py
 [Fine-Tune]: src/tumblrbot/flow/fine_tune.py
 [Generate]: src/tumblrbot/flow/generate.py
 [Main]: src/tumblrbot/__main__.py
 
-[Config]: #configuration
+[Configurable]: #configuration
 [Fine-Tuning]: #manual-fine-tuning
 [![PyPI - Version](https://img.shields.io/pypi/v/tumblrbot)](https://python.org/pypi/tumblrbot)
 
@@ -42,33 +44,34 @@ Features:
    1. Asks for [OpenAI] and [Tumblr] tokens.
       - Stores API tokens using [keyring].
    1. Retrieves [Tumblr] [OAuth] tokens.
-   1. [Downloads posts][Download] from the [configured][config] blogs.
+   1. [Downloads posts][Download] from specified blogs ([configurable]).
       - Skips redownloading already downloaded posts.
       - Shows progress and previews the current post.
-   1. [Creates examples][Examples] to fine-tune the model from your posts.
+   1. [Creates examples][Examples] to fine-tune the model from the downloaded posts.
       - Filters out posts that contain more than just text data.
-      - Filters out posts that contain [configured][config] regular expressions.
-      - Only uses the most recent posts from each blog as [configured][config].
-      - Adds custom user messages and assistant responses to the dataset from the [configured][config] file.
+      - Filters out posts that contain regular expressions ([configurable]).
+      - Only uses the most recent posts from each blog ([configurable]).
+      - Adds custom user messages and assistant responses to the dataset ([configurable]).
    1. Filters out any posts flagged by the [OpenAI Moderation API].
    1. [Uploads examples][Fine-Tune] to [OpenAI] and begins the fine-tuning process.
-      - Provides cost estimates if the currently saved examples are used to fine-tune the [configured][config] model.
+      - Provides cost estimates if the currently saved examples are used to fine-tune a base model ([configurable]).
       - Resumes monitoring the same fine-tuning process when restarted.
       - Deletes the uploaded examples file if fine-tuning does not succeed (optional).
       - Stores the output model automatically when fine-tuning is completed.
-   1. [Generates and uploads posts][Generate] to the [configured][config] blog using the [configured][config] fine-tuned model.
-      - Creates tags by extracting keywords at the [configured][config] frequency using the [configured][config] model.
-      - Uploads posts as drafts to the [configured][config] blog.
-      - Reblogs posts from the [configured][config] blogs at the [configured][config] frequency.
+   1. [Generates and uploads posts][Generate] to a blog using the fine-tuned model ([configurable]).
+      - Creates tags by extracting keywords using the base model ([configurable]).
+      - Uploads posts as drafts.
+      - Reblogs posts from allowed blogs ([configurable]).
       - Shows progress and previews the current post.
 - Colorful output, progress bars, and post previews using [rich].
-- Automatically keeps the [config] file up-to-date and recreates it if missing.
+- Automatically keeps the [config][configurable] file up-to-date and recreates it if missing (without overriding user settings).
 
 **Known Issues:**
 
 - Sometimes, you will get an error about the training file not being found when starting fine-tuning. We do not currently have a fix or workaround for this. You should instead use the online portal for fine-tuning if this continues to happen. Read more in [fine-tuning].
 - Post counts are incorrect when downloading posts. We are not certain what the cause of this is, but our tests suggest this is a [Tumblr] API problem that is giving inaccurate numbers.
 - During post downloading or post generation, you may receive a "Limit Exceeded" error message from the [Tumblr] API. This is caused by server-side rate-limiting by [Tumblr]. The only workaround is trying again or waiting for a period of time before retrying. In most cases, you either have to wait for a minute or an hour for the limits to reset. You can read more about the limits in the [Tumblr API documentation on rate limits].
+- Similar to the above issue, you may sometimes get a message saying your IP is blocked. This block is temporary and probably follows the same rules as previously described.
 
 **Please submit an issue or contact us for features you want added/reimplemented.**
 
@@ -85,7 +88,7 @@ Features:
 
 ## Usage
 
-Run `tumblrbot` from anywhere. Run `tumblrbot --help` for command-line options. Every command-line option corresponds to a value from the [config].
+Run `tumblrbot` from anywhere. Run `tumblrbot --help` for command-line options. Every command-line option corresponds to a value from the [config][configurable].
 
 ## Obtaining Tokens
 
@@ -153,15 +156,16 @@ Specific Options:
 - **`tags_chance`** - This should be between 0 and 1. Setting it to 0 corresponds to a 0% chance (never) to add tags to a post. 1 corresponds to a 100% chance (always) to add tags to a post. Adding tags incurs a very small token cost.
 - **`reblog_blog_identifiers`** - Whenever a reblog is attempted, a random blog from this list will be chosen to be reblogged from.
 - **`reblog_chance`** - This setting works the same way as `tags_chance`.
-- **`reblog_user_message`** - This setting is a prefix that is directly prepended to the contents of the post being reblogged.
+- **`reblog_user_message`** - This setting is a [format string]. The only argument it is formatted with is the content of the post being reblogged. In simple terms, the `{}` will be replaced with said content. Alternatively, you can leave out the `{}` so that the reblogged post is appended to the end.
+  - *Note: The bot is only given the latest message in a reblog chain due to the required complexity and added costs of including the entire chain.*
 
 ## Manual Fine-Tuning
 
 You can manually upload the examples file to [OpenAI] and start the fine-tuning here: [fine-tuning portal].
 
 1. Press `+ Create`.
-1. Select the desired `Base Model` from the dropdown. This should ideally match the model set in the [config].
-1. Upload the generated examples file to the section under `Training data`. You can find the path for this in the [config].
+1. Select the desired `Base Model` from the dropdown. This should ideally match the model set in the [config][configurable].
+1. Upload the generated examples file to the section under `Training data`. You can find the path for this in the [config][configurable].
 1. Press `Create`.
-1. (Optional) Copy the value next to `Job ID` and paste it into the [config] under `job_id`. You can then run the program and monitor its progress as usual.
-1. If you do not do the above, you will have to copy the value next to `Output model` once the job is complete and paste it into the [config] under `fine_tuned_model`.
+1. (Optional) Copy the value next to `Job ID` and paste it into the [config][configurable] under `job_id`. You can then run the program and monitor its progress as usual.
+1. If you do not do the above, you will have to copy the value next to `Output model` once the job is complete and paste it into the [config][configurable] under `fine_tuned_model`.

{tumblrbot-1.9.0 → tumblrbot-1.9.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tumblrbot"
-version = "1.9.0"
+version = "1.9.2"
 description = "An updated bot that posts to Tumblr, based on your very own blog!"
 readme = "README.md"
 requires-python = ">= 3.13"

{tumblrbot-1.9.0 → tumblrbot-1.9.2}/src/tumblrbot/flow/examples.py

@@ -30,7 +30,7 @@ class ExamplesWriter(FlowClass):
             for post in self.get_valid_posts():
                 self.write_example(
                     self.config.user_message,
-                    post.get_content_text(),
+                    str(post),
                     fp,
                 )
 
@@ -65,7 +65,7 @@ class ExamplesWriter(FlowClass):
         with path.open("rb") as fp:
             for line in fp:
                 post = Post.model_validate_json(line)
-                if post.valid_text_post() and not (self.config.filtered_words and pattern.search(post.get_content_text())):
+                if post.valid_text_post() and not (post.trail and self.config.filtered_words and pattern.search(str(post))):
                     yield post
 
     def filter_examples(self) -> None:

{tumblrbot-1.9.0 → tumblrbot-1.9.2}/src/tumblrbot/flow/generate.py

@@ -34,7 +34,9 @@ class DraftGenerator(FlowClass):
 
     def generate_post(self) -> Post:
         if original := self.get_random_post():
-            user_message = f"{self.config.reblog_user_message}\n\n{original.get_content_text()}"
+            user_message = self.config.reblog_user_message.format(original)
+            if "{}" not in self.config.reblog_user_message:
+                user_message += str(original)
         else:
             original = Post()
             user_message = self.config.user_message
@@ -79,7 +81,7 @@ class DraftGenerator(FlowClass):
                     offset,
                 ).response.posts:
                     post = Post.model_validate(raw_post)
-                    if post.valid_text_post():
+                    if post.valid_text_post() and self.is_trail_valid(post.trail):
                         return post
 
         return None
@@ -89,3 +91,7 @@ class DraftGenerator(FlowClass):
         total = self.tumblr.retrieve_blog_info(blog_identifier).response.blog.posts
         # The same Iterable object is cached, so reading an element will effectively discard it. This prevents checking the same offsets twice.
         return iter(sample(range(total), total))
+
+    def is_trail_valid(self, trail: list[Post]) -> bool:
+        # Checks if every post in the reblog trail is valid and that the blog that created the post is in the allowed reblog list.
+        return all(post.valid_text_post() and post.blog.name in self.config.reblog_blog_identifiers for post in trail)

{tumblrbot-1.9.0 → tumblrbot-1.9.2}/src/tumblrbot/utils/models.py

@@ -78,7 +78,7 @@ class Config(FileSyncSettings):
     tags_developer_message: str = Field("You will be provided with a block of text, and your task is to extract a very short list of the most important subjects from it.", description="The developer message used to generate tags.")
     reblog_blog_identifiers: list[str] = Field([], description="The identifiers of blogs that can be reblogged from when generating drafts.")
     reblog_chance: NonNegativeFloat = Field(0.1, description="The chance to generate a reblog of a random post. This will use more OpenAI tokens.")
-    reblog_user_message: str = Field("Please write a comical Tumblr post in response to the following post:", description="The prefix for the user message used to reblog posts.")
+    reblog_user_message: str = Field("Please write a comical Tumblr post in response to the following post:\n{}", description="The format string for the user message used to reblog posts.")
 
     @classmethod
     @override
@@ -145,12 +145,16 @@ class Tokens(FileSyncSettings):
     @model_validator(mode="after")
     @override
     def write(self) -> Self:
+        # Check if any tokens are missing or if the user wants to reset them, then set tokens if necessary.
         if not self.openai_api_key or Confirm.ask("Reset OpenAI API key?", default=False):
             (self.openai_api_key,) = self.online_token_prompt("https://platform.openai.com/api-keys", "API key")
 
         if not all(self.tumblr.model_dump().values()) or Confirm.ask("Reset Tumblr API tokens?", default=False):
             self.tumblr.client_key, self.tumblr.client_secret = self.online_token_prompt("https://tumblr.com/oauth/apps", "consumer key", "consumer secret")
 
+            # This is the whole OAuth 1.0 process.
+            # https://requests-oauthlib.readthedocs.io/en/latest/examples/tumblr.html
+            # We tried setting up OAuth 2.0, but the token refresh process is far too unreliable for this sort of program.
             with OAuth1Session(
                 self.tumblr.client_key,
                 self.tumblr.client_secret,
@@ -170,12 +174,15 @@ class Tokens(FileSyncSettings):
 
             self.tumblr.resource_owner_key, self.tumblr.resource_owner_secret = self.get_oauth_tokens(oauth_tokens)
 
+        # Regardless of whether any values were changed, we may as well write to the keyring.
+        # Any unchanged values will be set to the value they already were, since this is run after reading from the keyring.
         set_password(self.service_name, self.username, self.model_dump_json())
 
         return self
 
 
 class Blog(FullyValidatedModel):
+    name: str = ""
     posts: int = 0
     uuid: str = ""
 
@@ -206,24 +213,31 @@ class Post(FullyValidatedModel):
 
     content: SkipJsonSchema[list[Block]] = []
     layout: SkipJsonSchema[list[Block]] = []
-    trail: SkipJsonSchema[list[Any]] = []
+    trail: SkipJsonSchema[list[Self]] = []
 
     is_submission: SkipJsonSchema[bool] = False
 
     def __rich__(self) -> Panel:
         return Panel(
-            self.get_content_text(),
+            str(self),
             title="Preview",
             subtitle=" ".join(f"#{tag}" for tag in self.tags),
             subtitle_align="left",
         )
 
-    def valid_text_post(self) -> bool:
-        return bool(self.content) and all(block.type == "text" for block in self.content) and not (self.is_submission or self.trail or any(block.type == "ask" for block in self.layout))
-
-    def get_content_text(self) -> str:
+    def __str__(self) -> str:
+        # This function is really only relevant when a post is already valid, so we don't have to check the block types.
+        # If it is called on an invalid post, it would also work, but might give strange data.
         return "\n\n".join(block.text for block in self.content)
 
+    def valid_text_post(self) -> bool:
+        # Checks if this post:
+        # - has any content blocks (some glitched empty posts have no content)
+        # - only has content blocks of type 'text' (this excludes photo/video/poll/etc posts)
+        # - is not a submitted post
+        # - has no ask blocks in the content
+        return bool(self.content) and all(block.type == "text" for block in self.content) and not (self.is_submission or any(block.type == "ask" for block in self.layout))
+
 
 class Example(FullyValidatedModel):
     class Message(FullyValidatedModel):