forecasting-tools 0.2.89__tar.gz → 0.2.91__tar.gz

{forecasting_tools-0.2.89 → forecasting_tools-0.2.91}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: forecasting-tools
-Version: 0.2.89
+Version: 0.2.91
 Summary: AI forecasting and research tools to help humans reason about and forecast the future
 License: MIT
 License-File: LICENSE
@@ -31,7 +31,7 @@ Requires-Dist: litellm (>=1.59.1,<2.0.0,!=1.76.*,!=1.77.*,!=1.82.7,!=1.82.8)
 Requires-Dist: nest-asyncio (>=1.5.8,<2.0.0)
 Requires-Dist: numpy (>=1.26.0,<3.0.0)
 Requires-Dist: openai (>=1.51.0,<3.0.0)
-Requires-Dist: openai-agents[litellm] (>=0.2.0,<0.14.0)
+Requires-Dist: openai-agents[litellm] (>=0.2.0,<0.20.0)
 Requires-Dist: pandas (>=2.2.3,<4.0.0)
 Requires-Dist: pendulum (>=3.1.0,<4.0.0)
 Requires-Dist: pillow (>=9.0.0,<13.0.0)
@@ -43,7 +43,7 @@ Requires-Dist: requests (>=2.32.3,<3.0.0)
 Requires-Dist: scikit-learn (>=1.5.2,<2.0.0)
 Requires-Dist: streamlit (>=1.20.0,<2.0.0)
 Requires-Dist: tenacity (>=8.0.0,<10.0.0)
-Requires-Dist: tiktoken (>=0.8.0,<0.13.0)
+Requires-Dist: tiktoken (>=0.8.0,<0.20.0)
 Requires-Dist: typeguard (>=4.3.0,<5.0.0)
 Requires-Dist: unidecode (>=1.4.0,<2.0.0)
 Project-URL: Repository, https://github.com/Metaculus/forecasting-tools
@@ -71,18 +71,11 @@ This repository contains forecasting and research tools built with Python and St
 Here are the tools most likely to be useful to you:
 - 🎯 **Forecasting Bot:** General forecaster that integrates with the Metaculus AI benchmarking competition and provides a number of utilities. You can forecast with a pre-existing bot or override the class to customize your own (without redoing all the aggregation/API code, etc)
 - 🔌 **Metaculus API Wrapper:** for interacting with questions and tournaments
-- 📊 **Benchmarking:** Randomly sample quality questions from Metaculus and run your bot against them so you can get an early sense of how your bot is doing by comparing to the community prediction and expected baseline scores.
 - 🤖 **In-House Metaculus Bots**: You can see all the bots that Metaculus is running on their site in `run_bots.py`
 
 Here are some other features of the project (not all are documented yet):
-- **Smart Searcher:** A custom AI-powered internet-informed llm powered by Exa.ai and GPT. It is more configurable than Perplexity AI, allowing you to use any AI model, instruct the AI to decide on filters, get citations linking to exact paragraphs, etc.
-- **Key Factor Analysis:** Key Factors Analysis for scoring, ranking, and prioritizing important variables in forecasting questions
-- **Base Rate Researcher:** for calculating event probabilities (still experimental)
-- **Niche List Researcher:** for analyzing very specific lists of past events or items (still experimental)
-- **Fermi Estimator:** for breaking down numerical estimates (still experimental)
+- **General LLM Wrapper:** A unified interface around litellm with retry logic, the Metaculus proxy, structured outputs, and cost tracking
 - **Monetary Cost Manager:** for tracking AI and API expenses
-- **Prompt Optimizer:** for letting AI iterate through 100+ forecasting bot prompts
-- **Question Decomposer/Operationalizer:** To turn a question or topic into relevant forecastable sub-questions
 - **Other experimental tools:** See the demo site for other AI forecasting tools that this project supports (not all are documented). Also see the `scripts` folder for other common workflows and entry points into the code.
 
 All the examples below are in a Jupyter Notebook called `README.ipynb` which you can run locally to test the package (make sure to run the first cell though).
@@ -108,9 +101,8 @@ They both have roughly the same parameters. See below on how to use the Template
 
 
 ```python
-from forecasting_tools import TemplateBot, MetaculusApi, GeneralLlm
+from forecasting_tools import TemplateBot, MetaculusClient, GeneralLlm
 
-# Initialize the bot
 bot = TemplateBot(
     research_reports_per_question=3,  # Number of separate research attempts per question
     predictions_per_research_report=5,  # Number of predictions to make per research report
@@ -123,10 +115,10 @@ bot = TemplateBot(
     }
 )
 
-TOURNAMENT_ID = MetaculusApi.CURRENT_QUARTERLY_CUP_ID
+TOURNAMENT_ID = MetaculusClient.CURRENT_METACULUS_CUP_ID
 reports = await bot.forecast_on_tournament(TOURNAMENT_ID)
 
-# Print results (if the tournament is not active, no reports will be returned)
+# If the tournament is not active, no reports will be returned
 for report in reports:
     print(f"\nQuestion: {report.question.question_text}")
     print(f"Prediction: {report.prediction}")
@@ -151,19 +143,18 @@ for report in reports:
 from forecasting_tools import (
     TemplateBot,
     BinaryQuestion,
-    MetaculusApi,
-    DataOrganizer
+    MetaculusClient,
+    DataOrganizer,
 )
 
-# Initialize the bot
 bot = TemplateBot(
     research_reports_per_question=3,
     predictions_per_research_report=5,
     publish_reports_to_metaculus=False,
 )
 
-# Get and forecast a specific question
-question1 = MetaculusApi.get_question_by_url(
+metaculus_client = MetaculusClient()
+question1 = metaculus_client.get_question_by_url(
     "https://www.metaculus.com/questions/578/human-extinction-by-2100/"
 )
 question2 = BinaryQuestion(
@@ -175,17 +166,14 @@ question2 = BinaryQuestion(
 
 reports = await bot.forecast_questions([question1, question2])
 
-
-# Print results
 for report in reports:
     print(f"Question: {report.question.question_text}")
     print(f"Prediction: {report.prediction}")
     shortened_explanation = report.explanation.replace('\n', ' ')[:100]
     print(f"Reasoning: {shortened_explanation}...")
 
-# You can also save and load questions and reports
 file_path = "temp/reports.json"
-DataOrganizer.save_reports_to_file_path(reports, file_path) # This will overwrite the file if it already exists
+DataOrganizer.save_reports_to_file_path(reports, file_path) # Overwrites the file if it already exists
 loaded_reports = DataOrganizer.load_reports_from_file_path(file_path)
 ```
 
@@ -208,7 +196,7 @@ Note: You'll need to have your environment variables set up (see the section bel
 
 ## Customizing the Bot
 ### General Customization
-Generally all you have to do to make your own bot is inherit from the TemplateBot and override any combination of the 3 forecasting methods and the 1 research method. This saves you the headache of interacting with the Metaculus API, implementing aggregation of predictions, creating benchmarking interfaces, etc. Below is an example. It may also be helpful to look at the TemplateBot code (forecasting_tools/forecasting/forecast_bots/template_bot.py) for a more complete example.
+Generally all you have to do to make your own bot is inherit from the TemplateBot and override any combination of the 3 forecasting methods and the 1 research method. This saves you the headache of interacting with the Metaculus API, implementing aggregation of predictions, creating benchmarking interfaces, etc. Below is an example. It may also be helpful to look at the `TemplateBot` code (`forecasting_tools/forecast_bots/template_bot.py`) for a more complete example.
 
 
 ```python
@@ -222,9 +210,9 @@ from forecasting_tools import (
     PredictedOptionList,
     NumericDistribution,
     SmartSearcher,
-    MetaculusApi,
+    MetaculusClient,
     GeneralLlm,
-    PredictionExtractor
+    PredictionExtractor,
 )
 from forecasting_tools.util.misc import clean_indents
 
@@ -277,7 +265,7 @@ class MyCustomBot(TemplateBot):
         ...
 
 custom_bot = MyCustomBot()
-question = MetaculusApi.get_question_by_url(
+question = MetaculusClient().get_question_by_url(
     "https://www.metaculus.com/questions/578/human-extinction-by-2100/"
 )
 report = await custom_bot.forecast_question(question)
@@ -322,9 +310,9 @@ class NotepadBot(TemplateBot):
         notepad = await self._get_notepad(question)
 
         if notepad.total_predictions_attempted % 2 == 0:
-            model = "metaculus/gpt-4o"
+            model = "openrouter/openai/gpt-4o"
         else:
-            model = "metaculus/claude-3-5-sonnet-20240620"
+            model = "anthropic/claude-3-5-sonnet-20240620"
 
         personality = notepad.note_entries["personality"]
         prompt = f"You are a {personality}. Forecast this question: {question.question_text}. The last thing you write is your final answer as: 'Probability: ZZ%', 0-100"
@@ -345,127 +333,29 @@ Whether running locally or through Github actions, you will need to set environm
 
 # Important Utilities
 
-## Benchmarking
-Below is an example of how to run the benchmarker
-
-
-```python
-from forecasting_tools import Benchmarker, TemplateBot, BenchmarkForBot
-
-class CustomBot(TemplateBot):
-    ...
-
-# Run benchmark on multiple bots
-bots = [TemplateBot(), CustomBot()]  # Add your custom bots here
-benchmarker = Benchmarker(
-    forecast_bots=bots,
-    number_of_questions_to_use=2,  # Recommended 100+ for meaningful results
-    file_path_to_save_reports="benchmarks/",
-        # It will create a file name for you if given a folder.
-        # If a file name is given, and the file already exists, it will overwrite it.
-    concurrent_question_batch_size=5,
-)
-benchmarks: list[BenchmarkForBot] = await benchmarker.run_benchmark()
-
-# View results
-for benchmark in benchmarks[:2]:
-    print("--------------------------------")
-    print(f"Bot: {benchmark.name}")
-    print(f"Score: {benchmark.average_expected_baseline_score}") # Higher is better
-    print(f"Num reports in benchmark: {len(benchmark.forecast_reports)}")
-    print(f"Time: {benchmark.time_taken_in_minutes}min")
-    print(f"Cost: ${benchmark.total_cost}")
-```
-
-    --------------------------------
-    Bot: TemplateBot
-    Score: 53.24105782939477
-    Num reports in benchmark: 2
-    Time: 0.23375582297643024min
-    Cost: $0.03020605
-    --------------------------------
-    Bot: CustomBot
-    Score: 53.24105782939476
-    Num reports in benchmark: 2
-    Time: 0.20734789768854778min
-    Cost: $0.019155650000000003
-
-
-The ideal number of questions to get a good sense of whether one bot is better than another can vary. 100+ should tell your something decent. See [this analysis](https://forum.effectivealtruism.org/posts/DzqSh7akX28JEHf9H/comparing-two-forecasters-in-an-ideal-world) for exploration of the numbers. With too few questions, the results could just be statistical noise, though how many questions you need depends highly on the difference in skill of your bot versions.
-
-If you use the average expected baseline score, higher score is better. The scoring measures the expected value of your score without needing an actual resolution by assuming that the community prediction is the 'true probability'. Under this assumption, expected baseline scores are a proper score (see analysis in `scripts/simulate_a_tournament.ipynb`)
-
-As of May 29, 2025 the benchmarker automatically selects a random set of questions from Metaculus that:
-- Are binary questions (yes/no)
-- Are currently open
-- Opened within the last year
-- Have at least 30 forecasters
-- Have a community prediction
-- Are not part of a group question
-
-Note that sometimes there are not many questions matching these filters (e.g. at the beginning of a new year when a majority of open questions were just resolved). As of last edit there are plans to expand this to numeric and multiple choice, but right now it just benchmarks binary questions.
-
-You can grab these questions without using the Benchmarker by running the below
-
+## Metaculus Client
+The `MetaculusClient` wraps the Metaculus API for interacting with questions and tournaments. Grabbing questions returns a pydantic object that supports the important fields for Binary, Multiple Choice, Numeric, and Date questions. Instantiate the client once and reuse it (it picks up `METACULUS_TOKEN` from the environment by default).
 
 
 ```python
-from forecasting_tools import MetaculusApi
-
-questions = MetaculusApi.get_benchmark_questions(
-    num_of_questions_to_return=100,
-)
-```
-
-You can also save/load benchmarks to/from json
-
-
-```python
-from forecasting_tools import BenchmarkForBot
-
-# Load
-file_path = "benchmarks/benchmark.json"
-benchmarks: list[BenchmarkForBot] = BenchmarkForBot.load_json_from_file_path(file_path)
-
-# Save
-new_benchmarks: list[BenchmarkForBot] = benchmarks
-BenchmarkForBot.save_object_list_to_file_path(new_benchmarks, file_path) # Will overwrite the file if it already exists
-
-# To/From Json String
-single_benchmark = benchmarks[0]
-json_object: dict = single_benchmark.to_json()
-new_benchmark: BenchmarkForBot = BenchmarkForBot.from_json(json_object)
-```
-
-Once you have benchmark files in your project directory you can run `streamlit run forecasting_tools/benchmarking/benchmark_displayer.py` to get a UI with the benchmarks. You can also put `forecasting-tools.run_benchmark_streamlit_page()` into a new file, and run this file with streamlit to achieve the same results. This will allow you to see metrics side by side, explore code of past bots, see the actual bot responses, etc. It will pull in any files in your directory that contain "bench" in the name and are json. Results may take a while to load for large benchmark files.
-
-![Benchmark Displayer Top](./docs/images/benchmark_top_screen.png)
-![Benchmark Displayer Bottom](./docs/images/benchmark_bottom_screen.png)
-
-## Metaculus API
-The Metaculus API wrapper helps interact with Metaculus questions and tournaments. Grabbing questions returns a pydantic object, and supports important information for Binary, Multiple Choice, Numeric,and Date questions.
-
-
-```python
-from forecasting_tools import MetaculusApi, ApiFilter, DataOrganizer
+from forecasting_tools import MetaculusClient, ApiFilter, DataOrganizer
 from datetime import datetime
 
+metaculus_client = MetaculusClient()
 
-# Get a question by post id
-question = MetaculusApi.get_question_by_post_id(578)
+question = metaculus_client.get_question_by_post_id(578)
 print(f"Question found with url: {question.page_url}")
 
-# Get a question by url
-question = MetaculusApi.get_question_by_url("https://www.metaculus.com/questions/578/human-extinction-by-2100/")
+question = metaculus_client.get_question_by_url(
+    "https://www.metaculus.com/questions/578/human-extinction-by-2100/"
+)
 print(f"Question found with url: {question.page_url}")
 
-# Get all open questions from a tournament
-questions = MetaculusApi.get_all_open_questions_from_tournament(
-    tournament_id=MetaculusApi.CURRENT_QUARTERLY_CUP_ID
+questions = metaculus_client.get_all_open_questions_from_tournament(
+    tournament_id=MetaculusClient.CURRENT_METACULUS_CUP_ID
 )
 print(f"Num tournament questions: {len(questions)}")
 
-# Get questions matching a filter
 api_filter = ApiFilter(
     num_forecasters_gte=40,
     close_time_gt=datetime(2023, 12, 31),
@@ -474,35 +364,31 @@ api_filter = ApiFilter(
     allowed_types=["binary", "multiple_choice", "numeric", "date"],
     allowed_statuses=["resolved"],
 )
-questions = await MetaculusApi.get_questions_matching_filter(
+questions = await metaculus_client.get_questions_matching_filter(
     api_filter=api_filter,
     num_questions=50, # Remove this field to make it not error if you don't get 50 questions. However it will only go through one page of questions which may miss questions matching the ApiFilter since some filters are handled locally.
-    randomly_sample=False
+    randomly_sample=False,
 )
 print(f"Num filtered questions: {len(questions)}")
 
-# Load and save questions/reports
 file_path = "temp/questions.json"
-DataOrganizer.save_questions_to_file_path(questions, file_path) # Will overwrite the file if it already exists
+DataOrganizer.save_questions_to_file_path(questions, file_path) # Overwrites the file if it already exists
 questions = DataOrganizer.load_questions_from_file_path(file_path)
 
-# Get benchmark questions
-benchmark_questions = MetaculusApi.get_benchmark_questions(
+benchmark_questions = metaculus_client.get_benchmark_questions(
     num_of_questions_to_return=20
 )
 print(f"Num benchmark questions: {len(benchmark_questions)}")
 
-# Post a prediction
-MetaculusApi.post_binary_question_prediction(
+metaculus_client.post_binary_question_prediction(
     question_id=578, # Note that the question ID is not always the same as the post ID
-    prediction_in_decimal=0.012  # Must be between 0.01 and 0.99
+    prediction_in_decimal=0.012,  # Must be between 0.001 and 0.999
 )
 print("Posted prediction")
 
-# Post a comment
-MetaculusApi.post_question_comment(
+metaculus_client.post_question_comment(
     post_id=578,
-    comment_text="Here's example reasoning for testing... This will be a private comment..."
+    comment_text="Here's example reasoning for testing... This will be a private comment...",
 )
 print("Posted comment")
 ```
@@ -516,237 +402,39 @@ print("Posted comment")
     Posted comment
 
 
-# AI Research Tools/Agents
-
-## Smart Searcher
-The Smart Searcher acts like an LLM with internet access. It works a lot like Perplexity.ai API, except:
-- It has clickable citations that highlights and links directly to the paragraph cited using text fragments
-- You can ask the AI to use filters for domain, date, and keywords
-- There are options for structured output (Pydantic objects, lists, dict, list\[dict\], etc.)
-- Concurrent search execution for faster results
-- Optional detailed works cited list
-
-
-```python
-
-searcher = SmartSearcher(
-    temperature=0,
-    num_searches_to_run=2,
-    num_sites_per_search=10,  # Results returned per search
-    include_works_cited_list=False  # Add detailed citations at the end
-)
-
-response = await searcher.invoke(
-    "What is the recent news for Apple?"
-)
-
-print(response)
-```
-
-Example output:
-> Recent news about Apple includes several significant developments:
->
-> 1. **Expansion in India**: Apple is planning to open four more stores in India, with two in Delhi and Mumbai, and two in Bengaluru and Pune. This decision follows record revenues in India for the September 2024 quarter, driven by strong iPhone sales. Tim Cook, Apple's CEO, highlighted the enthusiasm and growth in the Indian market during the company's earnings call \[[1](https://telecomtalk.info/tim-cook-makes-major-announcement-for-apple-in-india/984260/#:~:text=This%20is%20not%20a%20new,first%20time%20Apple%20confirmed%20it.)\]\[[4](https://telecomtalk.info/tim-cook-makes-major-announcement-for-apple-in-india/984260/#:~:text=This%20is%20not%20a%20new,set%20an%20all%2Dtime%20revenue%20record.)\]\[[5](https://telecomtalk.info/tim-cook-makes-major-announcement-for-apple-in-india/984260/#:~:text=Previously%2C%20Diedre%20O%27Brien%2C%20Apple%27s%20senior,East%2C%20India%20and%20South%20Asia.)\]\[[8](https://telecomtalk.info/tim-cook-makes-major-announcement-for-apple-in-india/984260/#:~:text=At%20the%20company%27s%20earnings%20call,four%20new%20stores%20in%20India.)\].
->
-> 2. **Product Launches**: Apple is set to launch new iMac, Mac mini, and MacBook Pro models with M4 series chips on November 8, 2024. Additionally, the Vision Pro headset will be available in South Korea and the United Arab Emirates starting November 15, 2024. The second season of the Apple TV+ sci-fi series "Silo" will also premiere on November 15, 2024 \[[2](https://www.macrumors.com/2024/11/01/what-to-expect-from-apple-this-november/#:~:text=And%20the%20Vision%20Pro%20launches,the%20App%20Store%2C%20and%20more.)\]\[[12](https://www.macrumors.com/2024/11/01/what-to-expect-from-apple-this-november/#:~:text=As%20for%20hardware%2C%20the%20new,announcements%20in%20store%20this%20November.)\].
->
-> ... etc ...
+### Group Questions
 
-You can also use structured outputs by providing a Pydantic model (or any other simpler type hint) and using the schema formatting helper:
+Several of the methods above accept a `group_question_mode` parameter that controls how Metaculus group questions (e.g. "How many people will die of coronavirus in [period]?") are handled:
+- `"exclude"` — drop group questions from the result.
+- `"unpack_subquestions"` — turn each subquestion into a separate normal question.
 
+For backwards compatibility, the default is `"exclude"` for `get_question_by_post_id`, `get_question_by_url`, `ApiFilter` (used by `get_questions_matching_filter`), and `get_benchmark_questions` — so group questions don't get overweighted in benchmarks. The exception is `get_all_open_questions_from_tournament`, which defaults to `"unpack_subquestions"` so all subquestions are forecasted as normal questions.
 
 ```python
-from pydantic import BaseModel, Field
-from forecasting_tools import SmartSearcher
-
-class Company(BaseModel):
-    name: str = Field(description="Full company name")
-    market_cap: float = Field(description="Market capitalization in billions USD")
-    key_products: list[str] = Field(description="Main products or services")
-    relevance: str = Field(description="Why this company is relevant to the search")
-
-searcher = SmartSearcher(temperature=0, num_searches_to_run=4, num_sites_per_search=10)
-
-schema_instructions = searcher.get_schema_format_instructions_for_pydantic_type(Company)
-prompt = f"""Find companies that are leading the development of autonomous vehicles.
-Return as a list of companies with their details. Remember to give me a list of the schema provided.
-
-{schema_instructions}"""
-
-companies = await searcher.invoke_and_return_verified_type(prompt, list[Company])
-
-for company in companies:
-    print(f"\n{company.name} (${company.market_cap}B)")
-    print(f"Relevance: {company.relevance}")
-    print("Key Products:")
-    for product in company.key_products:
-        print(f"- {product}")
-```
-
-The schema instructions will format the Pydantic model into clear instructions for the AI about the expected output format and field descriptions.
-
+from forecasting_tools import MetaculusClient, ApiFilter
 
-## Key Factors Researcher
-The Key Factors Researcher helps identify and analyze key factors that should be considered for a forecasting question. As of last update, this is the most reliable of the tools, and gives something useful and accurate almost every time. It asks a lot of questions, turns search results into a long list of bullet points, rates each bullet point on ~8 criteria, and returns the top results.
+metaculus_client = MetaculusClient()
 
+# Unpack a group question into its subquestions
+result = metaculus_client.get_question_by_post_id(
+    post_id=...,  # a group-question post
+    group_question_mode="unpack_subquestions",
+)  # returns list[MetaculusQuestion] for group posts
 
-```python
-from forecasting_tools import (
-    KeyFactorsResearcher,
-    BinaryQuestion,
-    ScoredKeyFactor
-)
-
-# Consider using MetaculusApi.get_question_by_id or MetaculusApi.get_question_by_url instead
-question = BinaryQuestion(
-    question_text="Will YouTube be blocked in Russia?",
-    background_info="...", # Or 'None'
-    resolution_criteria="...", # Or 'None'
-    fine_print="...", # Or 'None'
-)
-
-# Find key factors
-key_factors = await KeyFactorsResearcher.find_and_sort_key_factors(
-    metaculus_question=question,
-    num_key_factors_to_return=5,  # Number of final factors to return
-    num_questions_to_research_with=26  # Number of research questions to generate
-)
-
-print(ScoredKeyFactor.turn_key_factors_into_markdown_list(key_factors))
-```
-
-Example output:
-> - The Russian authorities have slowed YouTube speeds to near unusable levels, indicating a potential groundwork for a future ban. [Source Published on 2024-09-12](https://meduza.io/en/feature/2024/09/12/the-russian-authorities-slowed-youtube-speeds-to-near-unusable-levels-so-why-are-kremlin-critics-getting-more-views#:~:text=Kolezev%20attributed%20this%20to%20the,suddenly%20stopped%20working%20in%20Russia.)
-> - Russian lawmaker Alexander Khinshtein stated that YouTube speeds would be deliberately slowed by up to 70% due to Google's non-compliance with Russian demands, indicating escalating measures against YouTube. [Source Published on 2024-07-25](https://www.yahoo.com/news/russia-slow-youtube-speeds-google-180512830.html#:~:text=Russia%20will%20deliberately%20slow%20YouTube,forces%20and%20promoting%20extremist%20content.)
-> - The press secretary of President Vladimir Putin, Dmitry Peskov, denied that the authorities intended to block YouTube, attributing access issues to outdated equipment due to sanctions. [Source Published on 2024-08-17](https://www.wsws.org/en/articles/2024/08/17/pbyj-a17.html#:~:text=%5BAP%20Photo%2FAP%20Photo%5D%20On%20July,two%20years%20due%20to%20sanctions.)
-> - YouTube is currently the last Western social media platform still operational in Russia, with over 93 million users in the country. [Source Published on 2024-07-26](https://www.techradar.com/pro/vpn/youtube-is-getting-throttled-in-russia-heres-how-to-unblock-it#:~:text=If%20you%27re%20in%20Russia%20and,platform%20to%20work%20in%20Russia.)
-> - Russian users reported mass YouTube outages amid growing official criticism, with reports of thousands of glitches in August 2024. [Source Published on 2024-08-09](https://www.aljazeera.com/news/2024/8/9/russian-users-report-mass-youtube-outage-amid-growing-official-criticism?traffic_source=rss#:~:text=Responding%20to%20this%2C%20a%20YouTube,reported%20about%20YouTube%20in%20Russia.)
-
-
-The simplified pydantic structure of the scored key factors is:
-```python
-class ScoredKeyFactor():
-    text: str
-    factor_type: KeyFactorType (Pro, Con, or Base_Rate)
-    citation: str
-    source_publish_date: datetime | None
-    url: str
-    score_card: ScoreCard
-    score: int
-    display_text: str
-```
-
-## Base Rate Researcher
-The Base Rate Researcher helps calculate historical base rates for events. As of last update, it gives decent results around 50% of the time. It orchestrates the Niche List Researcher and the Fermi Estimator to find base rate.
-
-
-```python
-from forecasting_tools import BaseRateResearcher
-
-# Initialize researcher
-researcher = BaseRateResearcher(
-    "How often has Apple been successfully sued for patent violations?"
-)
-
-# Get base rate analysis
-report = await researcher.make_base_rate_report()
-
-print(f"Historical rate: {report.historical_rate:.2%}")
-print(report.markdown_report)
-```
-
-## Niche List Researcher
-The Niche List Researcher helps analyze specific lists of events or items. The researcher will:
-1. Generate a comprehensive list of potential matches
-2. Remove duplicates
-3. Fact check each item against multiple criteria
-4. Return only validated items (unless include_incorrect_items=True)
-
-
-```python
-from forecasting_tools import NicheListResearcher
-
-researcher = NicheListResearcher(
-    type_of_thing_to_generate="Times Apple was successfully sued for patent violations between 2000-2024"
-)
-
-fact_checked_items = await researcher.research_niche_reference_class(
-    return_invalid_items=False
-)
-
-for item in fact_checked_items:
-    print(item)
-```
-
-The simplified pydantic structure of the fact checked items is:
-```python
-class FactCheckedItem():
-    item_name: str
-    description: str
-    is_uncertain: bool | None = None
-    initial_citations: list[str] | None = None
-    fact_check: FactCheck
-    type_description: str
-    is_valid: bool
-    supporting_urls: list[str]
-    one_line_fact_check_summary: str
-
-class FactCheck(BaseModel):
-    criteria_assessments: list[CriteriaAssessment]
-    is_valid: bool
-
-class CriteriaAssessment():
-    short_name: str
-    description: str
-    validity_assessment: str
-    is_valid_or_unknown: bool | None
-    citation_proving_assessment: str | None
-    url_proving_assessment: str | None:
-```
-
-## Fermi Estimator
-The Fermi Estimator helps break down numerical estimates using Fermi estimation techniques.
-
-
-
-```python
-from forecasting_tools import Estimator
-
-estimator = Estimator(
-    type_of_thing_to_estimate="books published worldwide each year",
-    previous_research=None  # Optional: Pass in existing research
+# Same option on a filtered query
+api_filter = ApiFilter(
+    allowed_statuses=["open"],
+    group_question_mode="unpack_subquestions",
 )
-
-size, explanation = await estimator.estimate_size()
-
-print(f"Estimate: {size:,}")
-print(explanation)
+questions = await metaculus_client.get_questions_matching_filter(api_filter=api_filter)
 ```
 
-Example output (Fake data with links not added):
-> I estimate that there are 2,750,000 'books published worldwide each year'.
->
-> **Facts**:
-> - Traditional publishers release approximately 500,000 new titles annually in English-speaking countries [1]
-> - China publishes around 450,000 new books annually [2]
-> - The global book market was valued at $92.68 billion in 2023 [3]
-> - Self-published titles have grown by 264% in the last 5 years [4]
-> - Non-English language markets account for about 50% of global publishing [5]
->
-> **Estimation Steps and Assumptions**:
-> 1. Start with traditional English publishing: 500,000 titles
-> 2. Add Chinese market: 500,000 + 450,000 = 950,000
-> 3. Account for other major languages (50% of market): 950,000 * 2 = 1,900,000
-> 4. Add self-published titles (estimated 45% of total): 1,900,000 * 1.45 = 2,755,000
->
-> **Background Research**: [Additional research details...]
-
 ## General LLM
-The `GeneralLlm` class is a wrapper around around litellm's acompletion function that adds some functionality like retry logic, calling the metaculus proxy, and cost callback handling. Litellm supports every model, most every parameter, and acts as one interface for every provider. See the litellm's acompletion function for a full list of parameters. Not all models will support all parameters. Additionally the Metaculus proxy doesn't support all models.
+The `GeneralLlm` class is a wrapper around litellm's acompletion function that adds some functionality like retry logic, calling the metaculus proxy, and cost callback handling. Litellm supports every model, most every parameter, and acts as one interface for every provider. See the litellm's acompletion function for a full list of parameters. Not all models will support all parameters. Additionally the Metaculus proxy doesn't support all models.
 
 
 ```python
-
+prompt = "What is the weather in Tokyo?"
 result = await GeneralLlm(model="gpt-4o").invoke(prompt)
 result = await GeneralLlm(model="claude-3-5-sonnet-20241022").invoke(prompt)
 result = await GeneralLlm(model="metaculus/claude-3-5-sonnet-20241022").invoke(prompt) # Adding 'metaculus' Calls the Metaculus proxy
@@ -859,7 +547,7 @@ The `MonetaryCostManager` helps to track AI and API costs. It tracks expenses an
 ```python
 from forecasting_tools import MonetaryCostManager
 from forecasting_tools import (
-    ExaSearcher, SmartSearcher, GeneralLlm
+    ExaSearcher, GeneralLlm
 )
 
 max_cost = 5.00
@@ -867,7 +555,6 @@ max_cost = 5.00
 with MonetaryCostManager(max_cost) as cost_manager:
     prompt = "What is the weather in Tokyo?"
     result = await GeneralLlm(model="gpt-4o").invoke(prompt)
-    result = await SmartSearcher(model="claude-3-5-sonnet-20241022").invoke(prompt)
     result = await ExaSearcher().invoke(prompt)
     # ... etc ...