tumblrbot 1.3.2__tar.gz → 1.4.1__tar.gz

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/PKG-INFO

@@ -1,18 +1,19 @@
 Metadata-Version: 2.4
 Name: tumblrbot
-Version: 1.3.2
+Version: 1.4.1
 Summary: An updated bot that posts to Tumblr, based on your very own blog!
 Requires-Python: >= 3.13
 Description-Content-Type: text/markdown
+Requires-Dist: httpx[http2]
 Requires-Dist: keyring
 Requires-Dist: more-itertools
+Requires-Dist: niquests[speedups, http3]
 Requires-Dist: openai
 Requires-Dist: pydantic
 Requires-Dist: pydantic-settings
-Requires-Dist: requests
+Requires-Dist: requests-cache
 Requires-Dist: requests-oauthlib
 Requires-Dist: rich
-Requires-Dist: tenacity
 Requires-Dist: tiktoken
 Requires-Dist: tomlkit
 Project-URL: Source, https://github.com/MaidThatPrograms/tumblrbot
@@ -71,8 +72,9 @@ Features:
 - Automatically keeps the [config] file up-to-date and recreates it if missing.
 
 **To-Do:**
-- Add documentation.
-- Finish updating [README.md].
+- Add code documentation.
+- Fix inaccurate post counts when downloading posts.
+- Fix file not found error when starting fine-tuning.
 
 
 **Please submit an issue or contact us for features you want added/reimplemented.**
@@ -113,5 +115,19 @@ After inputting the [Tumblr] tokens, you will be given a URL that you need to op
 
 ## Configuration
 All config options can be found in `config.toml` after running the program once. This will be kept up-to-date if there are changes to the config's format in a future update. This also means it may be worthwhile to double-check the config file after an update. Any changes to the config should be in the changelog for a given version.
-> WIP: There will be more information about the config options soon.
+
+All file options can include directories that will be created when the program is run.
+
+- `custom_prompts_file` You will have to create this file yourself. It should follow the following format:
+   ```json
+   {"user message 1": "assistant response 1",
+    "user message 2": "assistant response 2"}
+   ```
+- **`developer_message`** - This message is used in for fine-tuning the AI as well as generating prompts. If you change this, you will need to run the fine-tuning again with the new value before generating posts.
+- **`user_message`** - This message is used in the same way as `developer_message` and should be treated the same.
+- **`expected_epochs`** - The default value here is the default number of epochs for `base_model`. You may have to change this value if you change `base_model`. After running fine-tuning once, you will see the number of epochs used in the [fine-tuning portal](https://platform.openai.com/finetune) under *Hyperparameters*. This value will also be updated automatically if you run fine-tuning through this program.
+- **`token_price`** - The default value here is the default token price for `base_model`. You can find the up-to-date value [here](https://platform.openai.com/docs/pricing#fine-tuning), in the *Training* column.
+- **`job_id`** - If there is any value here, this program will resume monitoring the corresponding job, instead of starting a new one. This gets set when starting the fine-tuning and is cleared when it is completed. You can find job IDs in the [fine-tuning portal](https://platform.openai.com/finetune).
+- **`base_model`** - This value is used to choose the tokenizer for estimating fine-tuning costs. It is also the base model that will be fine-tuned and the model that is used to generate tags. You can find a list of options in the [fine-tuning portal](https://platform.openai.com/finetune) by pressing *+ Create* and opening the drop-down list for *Base Model*. Be sure to update `token_price` if you change this value.
+- **`tags_chance`** - This should be between 0 and 1. Setting it to 0 corresponds to a 0% chance (never) to add tags to a post. 1 corresponds to a 100% chance (always) to add tags to a post. Adding tags incurs a very small token cost.
 

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/README.md

@@ -52,8 +52,9 @@ Features:
 - Automatically keeps the [config] file up-to-date and recreates it if missing.
 
 **To-Do:**
-- Add documentation.
-- Finish updating [README.md].
+- Add code documentation.
+- Fix inaccurate post counts when downloading posts.
+- Fix file not found error when starting fine-tuning.
 
 
 **Please submit an issue or contact us for features you want added/reimplemented.**
@@ -94,4 +95,18 @@ After inputting the [Tumblr] tokens, you will be given a URL that you need to op
 
 ## Configuration
 All config options can be found in `config.toml` after running the program once. This will be kept up-to-date if there are changes to the config's format in a future update. This also means it may be worthwhile to double-check the config file after an update. Any changes to the config should be in the changelog for a given version.
-> WIP: There will be more information about the config options soon.
+
+All file options can include directories that will be created when the program is run.
+
+- `custom_prompts_file` You will have to create this file yourself. It should follow the following format:
+   ```json
+   {"user message 1": "assistant response 1",
+    "user message 2": "assistant response 2"}
+   ```
+- **`developer_message`** - This message is used in for fine-tuning the AI as well as generating prompts. If you change this, you will need to run the fine-tuning again with the new value before generating posts.
+- **`user_message`** - This message is used in the same way as `developer_message` and should be treated the same.
+- **`expected_epochs`** - The default value here is the default number of epochs for `base_model`. You may have to change this value if you change `base_model`. After running fine-tuning once, you will see the number of epochs used in the [fine-tuning portal](https://platform.openai.com/finetune) under *Hyperparameters*. This value will also be updated automatically if you run fine-tuning through this program.
+- **`token_price`** - The default value here is the default token price for `base_model`. You can find the up-to-date value [here](https://platform.openai.com/docs/pricing#fine-tuning), in the *Training* column.
+- **`job_id`** - If there is any value here, this program will resume monitoring the corresponding job, instead of starting a new one. This gets set when starting the fine-tuning and is cleared when it is completed. You can find job IDs in the [fine-tuning portal](https://platform.openai.com/finetune).
+- **`base_model`** - This value is used to choose the tokenizer for estimating fine-tuning costs. It is also the base model that will be fine-tuned and the model that is used to generate tags. You can find a list of options in the [fine-tuning portal](https://platform.openai.com/finetune) by pressing *+ Create* and opening the drop-down list for *Base Model*. Be sure to update `token_price` if you change this value.
+- **`tags_chance`** - This should be between 0 and 1. Setting it to 0 corresponds to a 0% chance (never) to add tags to a post. 1 corresponds to a 100% chance (always) to add tags to a post. Adding tags incurs a very small token cost.

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/pyproject.toml

@@ -1,19 +1,20 @@
 [project]
 name = "tumblrbot"
-version = "1.3.2"
+version = "1.4.1"
 description = "An updated bot that posts to Tumblr, based on your very own blog!"
 readme = "README.md"
 requires-python = ">= 3.13"
 dependencies = [
+  "httpx[http2]",
   "keyring",
   "more-itertools",
+  "niquests[speedups,http3]",
   "openai",
   "pydantic",
   "pydantic-settings",
-  "requests",
+  "requests-cache",
   "requests-oauthlib",
   "rich",
-  "tenacity",
   "tiktoken",
   "tomlkit",
 ]

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/src/tumblrbot/__main__.py

@@ -1,4 +1,4 @@
-from openai import OpenAI
+from openai import DefaultHttpxClient, OpenAI
 from rich.prompt import Confirm
 from rich.traceback import install
 
@@ -13,8 +13,11 @@ from tumblrbot.utils.tumblr import TumblrClient
 def main() -> None:
     install()
 
-    tokens = Tokens()
-    with OpenAI(api_key=tokens.openai_api_key.get_secret_value()) as openai, TumblrClient(tokens=tokens) as tumblr:
+    tokens = Tokens.read_from_keyring()
+    with (
+        OpenAI(api_key=tokens.openai_api_key.get_secret_value(), http_client=DefaultHttpxClient(http2=True)) as openai,
+        TumblrClient(tokens=tokens) as tumblr,
+    ):
         post_downloader = PostDownloader(openai, tumblr)
         if Confirm.ask("Download latest posts?", default=False):
             post_downloader.download()

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/src/tumblrbot/flow/download.py

@@ -7,32 +7,6 @@ from tumblrbot.utils.models import Post
 
 
 class PostDownloader(FlowClass):
-    def paginate_posts(self, blog_identifier: str, completed: int, after: int, fp: TextIOBase, live: PreviewLive) -> None:
-        task_id = live.progress.add_task(f"Downloading posts from '{blog_identifier}'...", total=None, completed=completed)
-
-        while True:
-            response = self.tumblr.retrieve_published_posts(blog_identifier, after=after).json()["response"]
-            live.progress.update(task_id, total=response["blog"]["posts"], completed=completed)
-
-            if posts := response["posts"]:
-                for post in posts:
-                    dump(post, fp)
-                    fp.write("\n")
-
-                    model = Post.model_validate(post)
-                    after = model.timestamp
-                    live.custom_update(model)
-
-                completed += len(posts)
-            else:
-                return
-
-    def get_data_path(self, blog_identifier: str) -> Path:
-        return (self.config.data_directory / blog_identifier).with_suffix(".jsonl")
-
-    def get_data_paths(self) -> list[Path]:
-        return list(map(self.get_data_path, self.config.download_blog_identifiers))
-
     def download(self) -> None:
         self.config.data_directory.mkdir(parents=True, exist_ok=True)
 
@@ -56,3 +30,29 @@ class PostDownloader(FlowClass):
                         fp,
                         live,
                     )
+
+    def paginate_posts(self, blog_identifier: str, completed: int, after: int, fp: TextIOBase, live: PreviewLive) -> None:
+        task_id = live.progress.add_task(f"Downloading posts from '{blog_identifier}'...", total=None, completed=completed)
+
+        while True:
+            response = self.tumblr.retrieve_published_posts(blog_identifier, after=after).json()["response"]
+            live.progress.update(task_id, total=response["blog"]["posts"], completed=completed)
+
+            if posts := response["posts"]:
+                for post in posts:
+                    dump(post, fp)
+                    fp.write("\n")
+
+                    model = Post.model_validate(post)
+                    after = model.timestamp
+                    live.custom_update(model)
+
+                completed += len(posts)
+            else:
+                return
+
+    def get_data_paths(self) -> list[Path]:
+        return list(map(self.get_data_path, self.config.download_blog_identifiers))
+
+    def get_data_path(self, blog_identifier: str) -> Path:
+        return (self.config.data_directory / blog_identifier).with_suffix(".jsonl")

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/src/tumblrbot/flow/examples.py

@@ -8,7 +8,7 @@ from typing import IO
 
 import rich
 from more_itertools import chunked
-from openai import BadRequestError, OpenAI
+from openai import BadRequestError
 from rich.console import Console
 from rich.prompt import Confirm
 from tiktoken import encoding_for_model, get_encoding
@@ -19,47 +19,47 @@ from tumblrbot.utils.models import Example, Post
 
 @dataclass
 class ExamplesWriter(FlowClass):
-    openai: OpenAI
     data_paths: list[Path]
 
-    def count_tokens(self) -> Generator[int]:
-        # Based on https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken
-        # and https://cookbook.openai.com/examples/chat_finetuning_data_prep
-        try:
-            encoding = encoding_for_model(self.config.base_model)
-        except KeyError as error:
-            encoding = get_encoding("o200k_base")
-            Console(stderr=True, style="logging.level.warning").print(f"[Warning] Using encoding '{encoding.name}': {''.join(error.args)}\n")
+    def write_examples(self) -> None:
+        self.config.examples_file.parent.mkdir(parents=True, exist_ok=True)
 
-        with self.config.examples_file.open(encoding="utf_8") as fp:
-            for line in fp:
-                example = Example.model_validate_json(line)
-                yield len(encoding.encode("assistant"))  # every reply is primed with <|start|>assistant<|message|>
-                for message in example.messages:
-                    yield 4 + len(encoding.encode(message.content))
+        with self.config.examples_file.open("w", encoding="utf_8") as fp:
+            for user_message, assistant_response in self.get_custom_prompts():
+                self.write_example(
+                    user_message,
+                    assistant_response,
+                    fp,
+                )
 
-    def get_moderation_chunk_limit(self) -> int:
-        test_n = 1000
-        try:
-            self.openai.moderations.create(input=[""] * test_n)
-        except BadRequestError as error:
-            message = error.response.json()["error"]["message"]
-            if match := search(r"(\d+)\.", message):
-                return int(match.group(1))
-        return test_n
+            for post in self.get_filtered_posts():
+                self.write_example(
+                    self.config.user_message,
+                    post.get_content_text(),
+                    fp,
+                )
 
-    def get_valid_posts(self) -> Generator[Post]:
-        for data_path in self.data_paths:
-            with data_path.open(encoding="utf_8") as fp:
-                for line in fp:
-                    post = Post.model_validate_json(line)
-                    if not (post.is_submission or post.trail) and post.only_text_blocks() and post.get_content_text():
-                        yield post
+        rich.print(f"[bold]The examples file can be found at: '{self.config.examples_file}'\n")
+
+    def write_example(self, user_message: str, assistant_message: str, fp: IO[str]) -> None:
+        example = Example(
+            messages=[
+                Example.Message(role="developer", content=self.config.developer_message),
+                Example.Message(role="user", content=user_message),
+                Example.Message(role="assistant", content=assistant_message),
+            ],
+        )
+        fp.write(f"{example.model_dump_json()}\n")
+
+    def get_custom_prompts(self) -> Generator[tuple[str, str]]:
+        if self.config.custom_prompts_file.exists():
+            text = self.config.custom_prompts_file.read_text(encoding="utf_8")
+            yield from loads(text).items()
 
     def get_filtered_posts(self) -> Generator[Post]:
         posts = list(self.get_valid_posts())
 
-        if Confirm.ask("Remove posts flagged by the OpenAI moderation? This can sometimes resolve errors with fine-tuning validation, but is slow.", default=False):
+        if Confirm.ask("[gray62]Remove posts flagged by the OpenAI moderation? This can sometimes resolve errors with fine-tuning validation, but is slow.", default=False):
             removed = 0
             chunk_size = self.get_moderation_chunk_limit()
             with PreviewLive() as live:
@@ -79,37 +79,36 @@ class ExamplesWriter(FlowClass):
         else:
             yield from posts
 
-    def get_custom_prompts(self) -> Generator[tuple[str, str]]:
-        if self.config.custom_prompts_file.exists():
-            text = self.config.custom_prompts_file.read_text(encoding="utf_8")
-            yield from loads(text).items()
-
-    def write_examples(self) -> None:
-        self.config.examples_file.parent.mkdir(parents=True, exist_ok=True)
-
-        with self.config.examples_file.open("w", encoding="utf_8") as fp:
-            for post in self.get_filtered_posts():
-                self.write_example(
-                    self.config.user_message,
-                    post.get_content_text(),
-                    fp,
-                )
+    def get_valid_posts(self) -> Generator[Post]:
+        for data_path in self.data_paths:
+            with data_path.open(encoding="utf_8") as fp:
+                for line in fp:
+                    post = Post.model_validate_json(line)
+                    if not (post.is_submission or post.trail) and post.only_text_blocks() and post.get_content_text():
+                        yield post
 
-            for user_message, assistant_response in self.get_custom_prompts():
-                self.write_example(
-                    user_message,
-                    assistant_response,
-                    fp,
-                )
+    def get_moderation_chunk_limit(self) -> int:
+        test_n = 1000
+        try:
+            self.openai.moderations.create(input=[""] * test_n)
+        except BadRequestError as error:
+            message = error.response.json()["error"]["message"]
+            if match := search(r"(\d+)\.", message):
+                return int(match.group(1))
+        return test_n
 
-        rich.print(f"[bold]The examples file can be found at: '{self.config.examples_file}'\n")
+    def count_tokens(self) -> Generator[int]:
+        # Based on https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken
+        # and https://cookbook.openai.com/examples/chat_finetuning_data_prep
+        try:
+            encoding = encoding_for_model(self.config.base_model)
+        except KeyError as error:
+            encoding = get_encoding("o200k_base")
+            Console(stderr=True, style="logging.level.warning").print(f"[Warning] Using encoding '{encoding.name}': {''.join(error.args)}\n")
 
-    def write_example(self, user_message: str, assistant_message: str, fp: IO[str]) -> None:
-        example = Example(
-            messages=[
-                Example.Message(role="developer", content=self.config.developer_message),
-                Example.Message(role="user", content=user_message),
-                Example.Message(role="assistant", content=assistant_message),
-            ],
-        )
-        fp.write(f"{example.model_dump_json()}\n")
+        with self.config.examples_file.open(encoding="utf_8") as fp:
+            for line in fp:
+                example = Example.model_validate_json(line)
+                yield len(encoding.encode("assistant"))  # every reply is primed with <|start|>assistant<|message|>
+                for message in example.messages:
+                    yield 4 + len(encoding.encode(message.content))

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/src/tumblrbot/flow/fine_tune.py

@@ -4,10 +4,7 @@ from textwrap import dedent
 from time import sleep
 
 import rich
-from openai import BadRequestError
-from openai.types import FileObject
 from openai.types.fine_tuning import FineTuningJob
-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_fixed, wait_random
 
 from tumblrbot.utils.common import FlowClass, PreviewLive
 
@@ -20,46 +17,33 @@ class FineTuner(FlowClass):
     def dedent_print(text: str) -> None:
         rich.print(dedent(text).lstrip())
 
-    def process_completed_job(self, job: FineTuningJob) -> None:
-        if job.trained_tokens is not None:
-            self.dedent_print(f"""
-                Trained Tokens: {job.trained_tokens:,}
-                Cost: {self.get_cost_string(job.trained_tokens)}
-            """)
-
-        self.config.job_id = ""
+    def fine_tune(self) -> None:
+        with PreviewLive() as live:
+            job = self.create_job(live)
 
-        if job.status == "failed" and job.error is not None:
-            raise RuntimeError(job.error.message)
+            self.dedent_print(f"""
+                [bold]Fine-tuning is starting...[/]
+                View it online at: https://platform.openai.com/finetune/{job.id}
+                    Created at: {datetime.fromtimestamp(job.created_at)}
+                    Base Model: {job.model}
 
-        if job.fine_tuned_model:
-            self.config.fine_tuned_model = job.fine_tuned_model or ""
+                [italic dim]Closing this terminal will not stop the fine-tuning. This will take a while...
+            """)  # noqa: DTZ006
 
-    def poll_job_status(self) -> FineTuningJob:
-        job = self.openai.fine_tuning.jobs.retrieve(self.config.job_id)
+            task_id = live.progress.add_task("", total=None)
 
-        if self.config.expected_epochs != job.hyperparameters.n_epochs and isinstance(job.hyperparameters.n_epochs, int):
-            self.config.expected_epochs = job.hyperparameters.n_epochs
+            while job.status not in {"succeeded", "failed", "cancelled"}:
+                job = self.poll_job_status()
 
-            self.dedent_print(f"""
-                The number of epochs has been updated to {job.hyperparameters.n_epochs}!
-                [cyan]Updated the config.
-            """)
-            self.print_estimates()
+                live.progress.update(
+                    task_id,
+                    total=job.estimated_finish,
+                    description=f"Fine-tuning: [italic]{job.status.replace('_', ' ').title()}[/]...",
+                )
 
-        return job
+                sleep(1)
 
-    @retry(
-        stop=stop_after_attempt(5),
-        wait=wait_fixed(1.5) + wait_random(),
-        retry=retry_if_exception_type(BadRequestError),
-        reraise=True,
-    )
-    def attempt_submit_job(self, file: FileObject) -> FineTuningJob:
-        return self.openai.fine_tuning.jobs.create(
-            model=self.config.base_model,
-            training_file=file.id,
-        )
+        self.process_completed_job(job)
 
     def create_job(self, live: PreviewLive) -> FineTuningJob:
         if self.config.job_id:
@@ -71,41 +55,42 @@ class FineTuner(FlowClass):
                 purpose="fine-tune",
             )
 
-        job = self.attempt_submit_job(file)
+        job = self.openai.fine_tuning.jobs.create(
+            model=self.config.base_model,
+            training_file=file.id,
+        )
 
         self.config.job_id = job.id
         return job
 
-    def fine_tune(self) -> None:
-        with PreviewLive() as live:
-            job = self.create_job(live)
-
-            self.dedent_print(f"""
-                [bold]Fine-tuning is starting...[/]
-                View it online at: https://platform.openai.com/finetune/{job.id}
-                    Created at: {datetime.fromtimestamp(job.created_at)}
-                    Base Model: {job.model}
+    def poll_job_status(self) -> FineTuningJob:
+        job = self.openai.fine_tuning.jobs.retrieve(self.config.job_id)
 
-                [italic dim]Closing this terminal will not stop the fine-tuning. This will take a while...
-            """)  # noqa: DTZ006
+        if self.config.expected_epochs != job.hyperparameters.n_epochs and isinstance(job.hyperparameters.n_epochs, int):
+            self.config.expected_epochs = job.hyperparameters.n_epochs
 
-            task_id = live.progress.add_task("", total=None)
+            self.dedent_print(f"""
+                The number of epochs has been updated to {job.hyperparameters.n_epochs}!
+                [cyan]Updated the config.
+            """)
+            self.print_estimates()
 
-            while job.status not in {"succeeded", "failed", "cancelled"}:
-                job = self.poll_job_status()
+        return job
 
-                live.progress.update(
-                    task_id,
-                    total=job.estimated_finish,
-                    description=f"Fine-tuning: [italic]{job.status.replace('_', ' ').title()}[/]...",
-                )
+    def process_completed_job(self, job: FineTuningJob) -> None:
+        if job.trained_tokens is not None:
+            self.dedent_print(f"""
+                Trained Tokens: {job.trained_tokens:,}
+                Cost: {self.get_cost_string(job.trained_tokens)}
+            """)
 
-                sleep(1)
+        self.config.job_id = ""
 
-        self.process_completed_job(job)
+        if job.status == "failed" and job.error is not None:
+            raise RuntimeError(job.error.message)
 
-    def get_cost_string(self, total_tokens: int) -> str:
-        return f"${self.config.token_price / 1000000 * total_tokens:.2f}"
+        if job.fine_tuned_model:
+            self.config.fine_tuned_model = job.fine_tuned_model or ""
 
     def print_estimates(self) -> None:
         total_tokens = self.config.expected_epochs * self.estimated_tokens
@@ -118,3 +103,6 @@ class FineTuner(FlowClass):
             NOTE: Token values are approximate and may not be 100% accurate, please be aware of this when using the data.
                     [italic red]Amelia, Mutsumi, and Marin are not responsible for any inaccuracies in the token count or estimated price.[/]
         """)
+
+    def get_cost_string(self, total_tokens: int) -> str:
+        return f"${self.config.token_price / 1000000 * total_tokens:.2f}"

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/src/tumblrbot/flow/generate.py

@@ -7,26 +7,20 @@ from tumblrbot.utils.models import Post
 
 
 class DraftGenerator(FlowClass):
-    def generate_tags(self, content: Post.Block) -> Post | None:
-        if random() < self.config.tags_chance:  # noqa: S311
-            return self.openai.responses.parse(
-                input=f"Extract the most important subjects from the following text:\n\n{content.text}",
-                model=self.config.base_model,
-                text_format=Post,
-                instructions="You are an advanced text summarization tool. You return the requested data to the user as a list of comma-separated strings.",
-                temperature=0.5,
-            ).output_parsed
-
-        return None
+    def create_drafts(self) -> None:
+        message = f"View drafts here: https://tumblr.com/blog/{self.config.upload_blog_identifier}/drafts"
 
-    def generate_content(self) -> Post.Block:
-        content = self.openai.responses.create(
-            input=self.config.user_message,
-            instructions=self.config.developer_message,
-            model=self.config.fine_tuned_model,
-        ).output_text
+        with PreviewLive() as live:
+            for i in live.progress.track(range(self.config.draft_count), description="Generating drafts..."):
+                try:
+                    post = self.generate_post()
+                    self.tumblr.create_post(self.config.upload_blog_identifier, post)
+                    live.custom_update(post)
+                except BaseException as exception:
+                    exception.add_note(f"📉 An error occurred! Generated {i} draft(s) before failing. {message}")
+                    raise
 
-        return Post.Block(type="text", text=content)
+        rich.print(f":chart_increasing: [bold green]Generated {self.config.draft_count} draft(s).[/] {message}")
 
     def generate_post(self) -> Post:
         content = self.generate_content()
@@ -38,17 +32,23 @@ class DraftGenerator(FlowClass):
             post.tags = tags.tags
         return post
 
-    def create_drafts(self) -> None:
-        message = f"View drafts here: https://tumblr.com/blog/{self.config.upload_blog_identifier}/drafts"
+    def generate_content(self) -> Post.Block:
+        content = self.openai.responses.create(
+            input=self.config.user_message,
+            instructions=self.config.developer_message,
+            model=self.config.fine_tuned_model,
+        ).output_text
 
-        with PreviewLive() as live:
-            for i in live.progress.track(range(self.config.draft_count), description="Generating drafts..."):
-                try:
-                    post = self.generate_post()
-                    self.tumblr.create_post(self.config.upload_blog_identifier, post)
-                    live.custom_update(post)
-                except BaseException as exception:
-                    exception.add_note(f"📉 An error occurred! Generated {i} draft(s) before failing. {message}")
-                    raise
+        return Post.Block(type="text", text=content)
 
-        rich.print(f":chart_increasing: [bold green]Generated {self.config.draft_count} draft(s).[/] {message}")
+    def generate_tags(self, content: Post.Block) -> Post | None:
+        if random() < self.config.tags_chance:  # noqa: S311
+            return self.openai.responses.parse(
+                text_format=Post,
+                input=f"Extract the most important subjects from the following text:\n\n{content.text}",
+                instructions="You are an advanced text summarization tool. You return the requested data to the user as a list of comma-separated strings.",
+                model=self.config.base_model,
+                temperature=0.5,
+            ).output_parsed
+
+        return None

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/src/tumblrbot/utils/common.py

@@ -13,6 +13,14 @@ from tumblrbot.utils.config import Config
 from tumblrbot.utils.tumblr import TumblrClient
 
 
+@dataclass
+class FlowClass:
+    config: ClassVar = Config()  # pyright: ignore[reportCallIssue]
+
+    openai: OpenAI
+    tumblr: TumblrClient
+
+
 class PreviewLive(Live):
     def __init__(self) -> None:
         super().__init__()
@@ -38,11 +46,3 @@ class PreviewLive(Live):
         table.add_row(self.progress)
         table.add_row(*renderables)
         self.update(table)
-
-
-@dataclass
-class FlowClass:
-    config: ClassVar = Config()  # pyright: ignore[reportCallIssue]
-
-    openai: OpenAI
-    tumblr: TumblrClient

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/src/tumblrbot/utils/config.py

@@ -26,28 +26,33 @@ class Config(BaseSettings):
         toml_file="config.toml",
     )
 
-    fine_tuned_model: str = Field("", description="The name of the OpenAI model that was fine-tuned with your posts.")
-    upload_blog_identifier: str = Field(
-        "",
-        description="The identifier of the blog which generated drafts will be uploaded to. This must be a blog associated with the same account as the configured Tumblr secret tokens.",
-    )
-    draft_count: PositiveInt = Field(150, description="The number of drafts to process. This will affect the number of tokens used with OpenAI")
-    tags_chance: NonNegativeFloat = Field(0.1, description="The chance to generate tags for any given post. This will incur extra calls to OpenAI.")
-
-    download_blog_identifiers: list[str] = Field(
-        [],
-        description="The identifiers of the blogs which post data will be downloaded from. These must be blogs associated with the same account as the configured Tumblr secret tokens.",
-    )
+    # Downloading Posts & Writing Examples
+    download_blog_identifiers: list[str] = Field([], description="The identifiers of the blogs which post data will be downloaded from. These must be blogs associated with the same account as the configured Tumblr secret tokens.")
     data_directory: Path = Field(Path("data"), description="Where to store downloaded post data.")
+
+    # Writing Examples
     custom_prompts_file: Path = Field(Path("custom_prompts.json"), description="Where to read in custom prompts from.")
+
+    # Writing Examples & Fine-Tuning
     examples_file: Path = Field(Path("examples.jsonl"), description="Where to output the examples that will be used to fine-tune the model.")
-    job_id: str = Field("", description="The fine-tuning job ID that will be polled on next run.")
+
+    # Writing Examples & Generating
+    developer_message: str = Field("You are a Tumblr post bot. Please generate a Tumblr post in accordance with the user's request.", description="The developer message used by the OpenAI API to generate drafts.")
+    user_message: str = Field("Please write a comical Tumblr post.", description="The user input used by the OpenAI API to generate drafts.")
+
+    # Fine-Tuning
     expected_epochs: PositiveInt = Field(3, description="The expected number of epochs fine-tuning will be run for. This will be updated during fine-tuning.")
     token_price: PositiveFloat = Field(3, description="The expected price in USD per million tokens during fine-tuning for the current model.")
+    job_id: str = Field("", description="The fine-tuning job ID that will be polled on next run.")
 
+    # Fine-Tuning & Generating
     base_model: ChatModel = Field("gpt-4o-mini-2024-07-18", description="The name of the model that will be fine-tuned by the generated training data.")
-    developer_message: str = Field("You are a Tumblr post bot. Please generate a Tumblr post in accordance with the user's request.", description="The developer message used by the OpenAI API to generate drafts.")
-    user_message: str = Field("Please write a comical Tumblr post.", description="The user input used by the OpenAI API to generate drafts.")
+    fine_tuned_model: str = Field("", description="The name of the OpenAI model that was fine-tuned with your posts.")
+
+    # Generating
+    upload_blog_identifier: str = Field("", description="The identifier of the blog which generated drafts will be uploaded to. This must be a blog associated with the same account as the configured Tumblr secret tokens.")
+    draft_count: PositiveInt = Field(150, description="The number of drafts to process. This will affect the number of tokens used with OpenAI")
+    tags_chance: NonNegativeFloat = Field(0.1, description="The chance to generate tags for any given post. This will incur extra calls to OpenAI.")
 
     @override
     @classmethod
@@ -58,11 +63,11 @@ class Config(BaseSettings):
     def write_to_file(self) -> Self:
         if not self.download_blog_identifiers:
             rich.print("Enter the [cyan]identifiers of your blogs[/] that data should be [bold purple]downloaded[/] from, separated by commas.")
-            self.download_blog_identifiers = list(map(str.strip, Prompt.ask("[bold]Example: staff.tumblr.com,changes").split(",")))
+            self.download_blog_identifiers = list(map(str.strip, Prompt.ask("[bold][Example] [dim]staff.tumblr.com,changes").split(",")))
 
         if not self.upload_blog_identifier:
             rich.print("Enter the [cyan]identifier of your blog[/] that drafts should be [bold purple]uploaded[/] to.")
-            self.upload_blog_identifier = Prompt.ask("[bold]Examples: staff.tumblr.com or changes").strip()
+            self.upload_blog_identifier = Prompt.ask("[bold][Example] [dim]staff.tumblr.com or changes").strip()
 
         toml_files = self.model_config.get("toml_file")
         if isinstance(toml_files, (Path, str)):
@@ -86,6 +91,6 @@ class Config(BaseSettings):
             toml_table[name] = value.get_secret_value() if isinstance(value, Secret) else dumped_model[name]
 
         Path(toml_file).write_text(
-            tomlkit.dumps(toml_table),  # pyright: ignore[reportUnknownMemberType]
+            tomlkit.dumps(toml_table),
             encoding="utf_8",
         )

{tumblrbot-1.3.2 → tumblrbot-1.4.1}/src/tumblrbot/utils/models.py

@@ -1,5 +1,5 @@
 from collections.abc import Generator
-from typing import Annotated, Any, ClassVar, Literal, override
+from typing import Annotated, Any, ClassVar, Literal, Self, override
 
 import rich
 from keyring import get_password, set_password
@@ -10,6 +10,14 @@ from requests_oauthlib import OAuth1Session
 from rich.panel import Panel
 from rich.prompt import Confirm, Prompt
 
+type SerializableSecretStr = Annotated[
+    SecretStr,
+    PlainSerializer(
+        SecretStr.get_secret_value,
+        when_used="json-unless-none",
+    ),
+]
+
 
 class FullyValidatedModel(BaseModel):
     model_config = ConfigDict(
@@ -22,13 +30,17 @@ class FullyValidatedModel(BaseModel):
 
 
 class Tokens(FullyValidatedModel):
+    class Tumblr(FullyValidatedModel):
+        client_key: SerializableSecretStr = SecretStr("")
+        client_secret: SerializableSecretStr = SecretStr("")
+        resource_owner_key: SerializableSecretStr = SecretStr("")
+        resource_owner_secret: SerializableSecretStr = SecretStr("")
+
     service_name: ClassVar = "tumblrbot"
+    username: ClassVar = "tokens"
 
-    openai_api_key: SecretStr = SecretStr("")
-    tumblr_client_key: SecretStr = SecretStr("")
-    tumblr_client_secret: SecretStr = SecretStr("")
-    tumblr_resource_owner_key: SecretStr = SecretStr("")
-    tumblr_resource_owner_secret: SecretStr = SecretStr("")
+    openai_api_key: SerializableSecretStr = SecretStr("")
+    tumblr: Tumblr = Tumblr()
 
     @staticmethod
     def online_token_prompt(url: str, *tokens: str) -> Generator[SecretStr]:
@@ -42,46 +54,44 @@ class Tokens(FullyValidatedModel):
 
         rich.print()
 
+    @classmethod
+    def read_from_keyring(cls) -> Self:
+        if json_data := get_password(cls.service_name, cls.username):
+            return cls.model_validate_json(json_data)
+        return cls()
+
     @override
     def model_post_init(self, context: object) -> None:
         super().model_post_init(context)
 
-        for name, _ in self:
-            if value := get_password(self.service_name, name):
-                setattr(self, name, value)
-
         if not self.openai_api_key.get_secret_value() or Confirm.ask("Reset OpenAI API key?", default=False):
             (self.openai_api_key,) = self.online_token_prompt("https://platform.openai.com/api-keys", "API key")
 
-        if not all(self.get_tumblr_tokens()) or Confirm.ask("Reset Tumblr API tokens?", default=False):
-            self.tumblr_client_key, self.tumblr_client_secret = self.online_token_prompt("https://tumblr.com/oauth/apps", "consumer key", "consumer secret")
-
-            oauth_session = OAuth1Session(*self.get_tumblr_tokens()[:2])
-            fetch_response = oauth_session.fetch_request_token("http://tumblr.com/oauth/request_token")  # pyright: ignore[reportUnknownMemberType]
-            full_authorize_url = oauth_session.authorization_url("http://tumblr.com/oauth/authorize")  # pyright: ignore[reportUnknownMemberType]
-            (redirect_response,) = self.online_token_prompt(full_authorize_url, "full redirect URL")
-            oauth_response = oauth_session.parse_authorization_response(redirect_response.get_secret_value())
-            oauth_session = OAuth1Session(
-                *self.get_tumblr_tokens()[:2],
+        if not all(self.tumblr.model_dump(mode="json").values()) or Confirm.ask("Reset Tumblr API tokens?", default=False):
+            self.tumblr.client_key, self.tumblr.client_secret = self.online_token_prompt("https://tumblr.com/oauth/apps", "consumer key", "consumer secret")
+
+            with OAuth1Session(
+                self.tumblr.client_key.get_secret_value(),
+                self.tumblr.client_secret.get_secret_value(),
+            ) as oauth_session:
+                fetch_response = oauth_session.fetch_request_token("http://tumblr.com/oauth/request_token")
+                full_authorize_url = oauth_session.authorization_url("http://tumblr.com/oauth/authorize")
+                (redirect_response,) = self.online_token_prompt(full_authorize_url, "full redirect URL")
+                oauth_response = oauth_session.parse_authorization_response(redirect_response.get_secret_value())
+
+            with OAuth1Session(
+                self.tumblr.client_key.get_secret_value(),
+                self.tumblr.client_secret.get_secret_value(),
                 fetch_response["oauth_token"],
                 fetch_response["oauth_token_secret"],
                 verifier=oauth_response["oauth_verifier"],
-            )
-            oauth_tokens = oauth_session.fetch_access_token("http://tumblr.com/oauth/access_token")  # pyright: ignore[reportUnknownMemberType]
-            self.tumblr_resource_owner_key = oauth_tokens["oauth_token"]
-            self.tumblr_resource_owner_secret = oauth_tokens["oauth_token_secret"]
-
-        for name, value in self:
-            if isinstance(value, SecretStr):
-                set_password(self.service_name, name, value.get_secret_value())
-
-    def get_tumblr_tokens(self) -> tuple[str, str, str, str]:
-        return (
-            self.tumblr_client_key.get_secret_value(),
-            self.tumblr_client_secret.get_secret_value(),
-            self.tumblr_resource_owner_key.get_secret_value(),
-            self.tumblr_resource_owner_secret.get_secret_value(),
-        )
+            ) as oauth_session:
+                oauth_tokens = oauth_session.fetch_access_token("http://tumblr.com/oauth/access_token")
+
+            self.tumblr.resource_owner_key = oauth_tokens["oauth_token"]
+            self.tumblr.resource_owner_secret = oauth_tokens["oauth_token_secret"]
+
+        set_password(self.service_name, self.username, self.model_dump_json())
 
 
 class Post(FullyValidatedModel):

tumblrbot-1.4.1/src/tumblrbot/utils/tumblr.py

@@ -0,0 +1,49 @@
+from dataclasses import dataclass
+from typing import Self
+
+from niquests import HTTPError, PreparedRequest, Response, Session
+from requests_cache import CacheMixin
+from requests_oauthlib import OAuth1
+
+from tumblrbot.utils.models import Post, Tokens
+
+
+@dataclass
+class TumblrClient(Session, CacheMixin):  # pyright: ignore[reportIncompatibleMethodOverride, reportIncompatibleVariableOverride]
+    tokens: Tokens
+
+    def __post_init__(self) -> None:
+        super().__init__(happy_eyeballs=True)
+        CacheMixin.__init__(self, use_cache_dir=True)
+
+        self.auth = OAuth1(**self.tokens.tumblr.model_dump(mode="json"))
+        self.hooks["response"].append(self.response_hook)
+
+    def __enter__(self) -> Self:
+        super().__enter__()
+        return self
+
+    def response_hook(self, response: PreparedRequest | Response) -> None:
+        if isinstance(response, Response):
+            try:
+                response.raise_for_status()
+            except HTTPError as error:
+                if response.text:
+                    error.add_note(response.text)
+                raise
+
+    def retrieve_published_posts(self, blog_identifier: str, after: int) -> Response:
+        return self.get(
+            f"https://api.tumblr.com/v2/blog/{blog_identifier}/posts",
+            params={
+                "after": str(after),
+                "sort": "asc",
+                "npf": str(True),
+            },
+        )
+
+    def create_post(self, blog_identifier: str, post: Post) -> Response:
+        return self.post(
+            f"https://api.tumblr.com/v2/blog/{blog_identifier}/posts",
+            json=post.model_dump(mode="json"),
+        )

tumblrbot-1.3.2/src/tumblrbot/utils/tumblr.py

@@ -1,39 +0,0 @@
-from dataclasses import dataclass
-
-from requests import HTTPError, Response
-from requests_oauthlib import OAuth1Session
-
-from tumblrbot.utils.models import Post, Tokens
-
-
-@dataclass
-class TumblrClient(OAuth1Session):
-    tokens: Tokens
-
-    def __post_init__(self) -> None:
-        super().__init__(*self.tokens.get_tumblr_tokens())  # pyright: ignore[reportUnknownMemberType]
-
-        self.hooks["response"].append(self.response_hook)
-
-    def response_hook(self, response: Response, **_: object) -> None:
-        try:
-            response.raise_for_status()
-        except HTTPError as error:
-            error.add_note(response.text)
-            raise
-
-    def retrieve_published_posts(self, blog_identifier: str, after: int) -> Response:
-        return self.get(
-            f"https://api.tumblr.com/v2/blog/{blog_identifier}/posts",
-            params={
-                "after": after,
-                "sort": "asc",
-                "npf": True,
-            },
-        )
-
-    def create_post(self, blog_identifier: str, post: Post) -> Response:
-        return self.post(
-            f"https://api.tumblr.com/v2/blog/{blog_identifier}/posts",
-            json=post.model_dump(mode="json"),
-        )