pydoll-python 1.2.3__tar.gz → 1.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. pydoll_python-1.3.0/PKG-INFO +864 -0
  2. pydoll_python-1.3.0/README.md +844 -0
  3. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/browser/base.py +82 -9
  4. pydoll_python-1.3.0/pydoll/browser/chrome.py +65 -0
  5. pydoll_python-1.3.0/pydoll/browser/managers.py +296 -0
  6. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/browser/page.py +121 -9
  7. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/commands/dom.py +129 -22
  8. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/commands/network.py +30 -0
  9. pydoll_python-1.3.0/pydoll/commands/runtime.py +92 -0
  10. pydoll_python-1.3.0/pydoll/commands/storage.py +54 -0
  11. pydoll_python-1.3.0/pydoll/commands/target.py +101 -0
  12. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/connection/connection.py +191 -11
  13. pydoll_python-1.3.0/pydoll/connection/managers.py +262 -0
  14. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/element.py +129 -11
  15. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/mixins/find_elements.py +79 -18
  16. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pyproject.toml +1 -1
  17. pydoll_python-1.2.3/PKG-INFO +0 -207
  18. pydoll_python-1.2.3/README.md +0 -187
  19. pydoll_python-1.2.3/pydoll/browser/chrome.py +0 -33
  20. pydoll_python-1.2.3/pydoll/browser/managers.py +0 -154
  21. pydoll_python-1.2.3/pydoll/commands/runtime.py +0 -45
  22. pydoll_python-1.2.3/pydoll/commands/storage.py +0 -18
  23. pydoll_python-1.2.3/pydoll/commands/target.py +0 -35
  24. pydoll_python-1.2.3/pydoll/connection/managers.py +0 -136
  25. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/LICENSE +0 -0
  26. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/__init__.py +0 -0
  27. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/browser/__init__.py +0 -0
  28. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/browser/options.py +0 -0
  29. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/commands/__init__.py +0 -0
  30. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/commands/browser.py +0 -0
  31. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/commands/fetch.py +0 -0
  32. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/commands/input.py +0 -0
  33. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/commands/page.py +0 -0
  34. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/connection/__init__.py +0 -0
  35. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/constants.py +0 -0
  36. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/events/__init__.py +0 -0
  37. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/events/browser.py +0 -0
  38. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/events/dom.py +0 -0
  39. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/events/fetch.py +0 -0
  40. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/events/network.py +0 -0
  41. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/events/page.py +0 -0
  42. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/exceptions.py +0 -0
  43. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/mixins/__init__.py +0 -0
  44. {pydoll_python-1.2.3 → pydoll_python-1.3.0}/pydoll/utils.py +0 -0
@@ -0,0 +1,864 @@
1
+ Metadata-Version: 2.3
2
+ Name: pydoll-python
3
+ Version: 1.3.0
4
+ Summary:
5
+ Author: Thalison Fernandes
6
+ Author-email: thalissfernandes99@gmail.com
7
+ Requires-Python: >=3.10,<4.0
8
+ Classifier: Programming Language :: Python :: 3
9
+ Classifier: Programming Language :: Python :: 3.10
10
+ Classifier: Programming Language :: Python :: 3.11
11
+ Classifier: Programming Language :: Python :: 3.12
12
+ Classifier: Programming Language :: Python :: 3.13
13
+ Requires-Dist: aiofiles (>=23.2.1,<24.0.0)
14
+ Requires-Dist: aiohttp (>=3.9.5,<4.0.0)
15
+ Requires-Dist: bs4 (>=0.0.2,<0.0.3)
16
+ Requires-Dist: requests (>=2.31.0,<3.0.0)
17
+ Requires-Dist: websockets (>=13.1,<14.0)
18
+ Description-Content-Type: text/markdown
19
+
20
+ <p align="center">
21
+ <h1>🚀 Pydoll: Async Web Automation in Python!</h1>
22
+ </p>
23
+ <br>
24
+ <p align="center">
25
+ <img src="https://github.com/user-attachments/assets/c4615101-d932-4e79-8a08-f50fbc686e3b" alt="Alt text" /> <br><br>
26
+ </p>
27
+
28
+ <p align="center">
29
+ <img src="https://codecov.io/github/thalissonvs/pydoll/graph/badge.svg?token=40I938OGM9"/>
30
+ <img src="https://github.com/thalissonvs/pydoll/actions/workflows/tests.yml/badge.svg" alt="Tests">
31
+ <img src="https://github.com/thalissonvs/pydoll/actions/workflows/ruff-ci.yml/badge.svg" alt="Ruff CI">
32
+ <img src="https://github.com/thalissonvs/pydoll/actions/workflows/release.yml/badge.svg" alt="Release">
33
+ <img src="https://tokei.rs/b1/github/thalissonvs/pydoll" alt="Total lines">
34
+ <img src="https://tokei.rs/b1/github/thalissonvs/pydoll?category=files" alt="Files">
35
+ <img src="https://tokei.rs/b1/github/thalissonvs/pydoll?category=comments" alt="Comments">
36
+ <img src="https://img.shields.io/github/issues/thalissonvs/pydoll?label=Issues" alt="GitHub issues">
37
+ <img src="https://img.shields.io/github/issues-closed/thalissonvs/pydoll?label=Closed issues" alt="GitHub closed issues">
38
+ <img src="https://img.shields.io/github/issues/thalissonvs/pydoll/bug?label=Bugs&color=red" alt="GitHub bug issues">
39
+ <img src="https://img.shields.io/github/issues/thalissonvs/pydoll/enhancement?label=Enhancements&color=purple" alt="GitHub enhancement issues">
40
+ </p>
41
+ <p align="center">
42
+ <a href="https://trendshift.io/repositories/13125" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13125" alt="thalissonvs%2Fpydoll | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
43
+ </p>
44
+
45
+ Pydoll is an innovative Python library that's redefining Chromium browser automation! Unlike other solutions, Pydoll **completely eliminates the need for webdrivers**, providing a much more fluid and reliable automation experience.
46
+
47
+
48
+
49
+ ## ⭐ Extraordinary Features
50
+
51
+ - **Zero Webdrivers!** Say goodbye to webdriver compatibility and configuration headaches
52
+ - **Native Captcha Bypass!** Naturally passes through Cloudflare Turnstile and reCAPTCHA v3 *
53
+ - **Performance** thanks to native asynchronous programming
54
+ - **Realistic Interactions** that simulate human behavior
55
+ - **Advanced Event System** for complex and reactive automations
56
+
57
+ > Note: for cloudflare captcha, you have to perform a click in the checkbox. Just find a div containing the iframe and use the `.click()` method. Automatic detection and click coming soon!
58
+
59
+ ## Table of Contents
60
+
61
+ - [Installation](#-installation)
62
+ - [Quick Start](#-quick-start)
63
+ - [Core Components](#-core-components)
64
+ - [Browser Interface](#browser-interface)
65
+ - [Page Interface](#page-interface)
66
+ - [WebElement Interface](#webelement-interface)
67
+ - [Advanced Features](#-advanced-features)
68
+ - [Event System](#event-system)
69
+ - [Concurrent Scraping](#concurrent-scraping)
70
+ - [Best Practices](#-best-practices)
71
+ - [Contributing](#-contributing)
72
+
73
+ ## 🔥 Installation
74
+
75
+ ```bash
76
+ pip install pydoll-python
77
+ ```
78
+
79
+ ## ⚡ Quick Start
80
+
81
+ See how simple it is to get started - no webdriver configuration needed!
82
+
83
+ ```python
84
+ import asyncio
85
+ from pydoll.browser.chrome import Chrome
86
+ from pydoll.constants import By
87
+
88
+ async def main():
89
+ # Start the browser with no additional webdriver configuration!
90
+ async with Chrome() as browser:
91
+ await browser.start()
92
+ page = await browser.get_page()
93
+
94
+ # Navigate through captcha-protected sites without worry
95
+ await page.go_to('https://example-with-cloudflare.com')
96
+ button = await page.find_element(By.CSS_SELECTOR, 'button')
97
+ await button.click()
98
+
99
+ asyncio.run(main())
100
+ ```
101
+
102
+ If you need to configure the browser, you can do it like this:
103
+
104
+ ```python
105
+ from pydoll.browser.chrome import Chrome
106
+ from pydoll.browser.options import Options
107
+
108
+ options = Options()
109
+ # Public or private proxy, you choose!
110
+ options.add_argument('--proxy-server=username:password@ip:port')
111
+ # Needs to change the default browser location? No worries!
112
+ options.binary_location = '/path/to/your/browser'
113
+ async with Chrome(options=options) as browser:
114
+ await browser.start()
115
+ ```
116
+
117
+ Here you can find all the options available: [Chromium Command Line Switches](https://peter.sh/experiments/chromium-command-line-switches/)
118
+
119
+ ## 🎯 Core Components
120
+
121
+ ### Browser Interface
122
+
123
+ The Browser domain provides direct control over the browser itself, offering global methods to manage the entire browser instance. Unlike page-specific operations, these methods affect the browser as a whole, allowing you to control multiple pages, handle window properties, manage cookies across all domains, and monitor events throughout the entire browsing session.
124
+
125
+ Here's an example of how to use the Browser domain:
126
+
127
+ ```python
128
+ async def browser_examples():
129
+ async with Chrome() as browser:
130
+ await browser.start()
131
+ # Control multiple pages with incredible ease
132
+ pages = [await browser.get_page() for _ in range(3)]
133
+
134
+ # Advanced settings with a simple command
135
+ await browser.set_window_maximized()
136
+ ```
137
+
138
+ #### Browser Management
139
+
140
+ Now, let's dive into the methods of the Browser domain.
141
+
142
+ ##### `async start() -> None`
143
+ Fires up your browser and gets everything ready for automation magic!
144
+
145
+ ```python
146
+ async with Chrome() as browser:
147
+ await browser.start() # Starts the browser
148
+ ```
149
+
150
+ ##### `async stop() -> None`
151
+ Gracefully shuts down the browser when you're done.
152
+
153
+ ```python
154
+ await browser.stop() # Manually stops the browser
155
+ ```
156
+
157
+ ##### `async get_page() -> Page`
158
+ Grabs an existing page or creates a fresh one if needed - super convenient!
159
+
160
+ ```python
161
+ # Gets an existing page or creates a new one
162
+ page = await browser.get_page()
163
+ await page.go_to("https://www.example.com")
164
+ ```
165
+
166
+ ##### `async new_page(url: str = '') -> str`
167
+ Opens a brand new page and returns its ID for future reference.
168
+ Always prefer using the `get_page` method to get a page instance.
169
+
170
+ ```python
171
+ # Creates a new page and navigates directly to a URL
172
+ page_id = await browser.new_page("https://www.example.com")
173
+ ```
174
+
175
+ ##### `async get_page_by_id(page_id: str) -> Page`
176
+ Lets you access any specific page using its ID - perfect for multi-tab automation!
177
+
178
+ ```python
179
+ # Gets a specific page by ID
180
+ page = await browser.get_page_by_id(page_id)
181
+ ```
182
+
183
+ ##### `async get_targets() -> list`
184
+ Shows you all open pages in the browser - great for keeping track of everything.
185
+
186
+ ```python
187
+ # Lists all open pages in the browser
188
+ targets = await browser.get_targets()
189
+ for target in targets:
190
+ print(f"Page: {target.get('title')} - URL: {target.get('url')}")
191
+ ```
192
+
193
+ Want to switch between tabs or pages? It's super easy! First, get all your targets:
194
+
195
+ ```python
196
+ targets = await browser.get_targets()
197
+ ```
198
+
199
+ You'll get something like this:
200
+
201
+ ```python
202
+ [{'targetId': 'F4729A95E0E4F9456BB6A853643518AF', 'type': 'page', 'title': 'New Tab', 'url': 'chrome://newtab/', 'attached': False, 'canAccessOpener': False, 'browserContextId': 'C76015D1F1C690B7BC295E1D81C8935F'}, {'targetId': '1C44D55BEEE43F44C52D69D8FC5C3685', 'type': 'iframe', 'title': 'chrome-untrusted://new-tab-page/one-google-bar?paramsencoded=', 'url': 'chrome-untrusted://new-tab-page/one-google-bar?paramsencoded=', 'attached': False, 'canAccessOpener': False, 'browserContextId': 'C76015D1F1C690B7BC295E1D81C8935F'}]
203
+ ```
204
+
205
+ Then just pick the page you want:
206
+
207
+ ```python
208
+ target = next(target for target in targets if target['title'] == 'New Tab')
209
+ ```
210
+
211
+ And switch to it:
212
+
213
+ ```python
214
+ new_tab_page = await browser.get_page_by_id(target['targetId'])
215
+ ```
216
+
217
+ Now you can control this page as if it were the only one open! Switch between tabs effortlessly by keeping references to each page.
218
+
219
+ ##### `async set_window_bounds(bounds: dict) -> None`
220
+ Position and size your browser window exactly how you want it!
221
+
222
+ ```python
223
+ # Sets the size and position of the window
224
+ await browser.set_window_bounds({
225
+ 'left': 100,
226
+ 'top': 100,
227
+ 'width': 1024,
228
+ 'height': 768
229
+ })
230
+ ```
231
+
232
+ ##### `async set_window_maximized() -> None`
233
+ Make your browser take up the full screen with one simple command.
234
+
235
+ ```python
236
+ # Maximizes the browser window
237
+ await browser.set_window_maximized()
238
+ ```
239
+
240
+ ##### `async set_window_minimized() -> None`
241
+ Hide the browser window when you don't need to see it.
242
+
243
+ ```python
244
+ # Minimizes the browser window
245
+ await browser.set_window_minimized()
246
+ ```
247
+
248
+ ##### `async get_cookies() -> list[dict]`
249
+ Peek into all cookies stored by the browser - great for debugging or session management!
250
+
251
+ ```python
252
+ # Gets all cookies
253
+ cookies = await browser.get_cookies()
254
+ for cookie in cookies:
255
+ print(f"Name: {cookie['name']}, Value: {cookie['value']}")
256
+ ```
257
+
258
+ ##### `async set_cookies(cookies: list[dict]) -> None`
259
+ Set up custom cookies for authentication or testing scenarios.
260
+
261
+ ```python
262
+ # Sets cookies in the browser
263
+ await browser.set_cookies([
264
+ {
265
+ 'name': 'session_id',
266
+ 'value': '12345',
267
+ 'domain': 'example.com',
268
+ 'path': '/',
269
+ 'expires': -1, # Session
270
+ 'secure': True
271
+ }
272
+ ])
273
+ ```
274
+
275
+ ##### `async delete_all_cookies() -> None`
276
+ Wipe all cookies clean - perfect for testing from a fresh state!
277
+
278
+ ```python
279
+ # Clears all cookies
280
+ await browser.delete_all_cookies()
281
+ ```
282
+
283
+ ##### `async set_download_path(path: str) -> None`
284
+ Tell your browser exactly where to save downloaded files.
285
+
286
+ ```python
287
+ # Sets the directory to save downloads
288
+ await browser.set_download_path("/path/to/downloads")
289
+ ```
290
+
291
+ ##### `async on(event_name: str, callback: callable, temporary: bool = False) -> int`
292
+ Registers a callback for a specific event. You can read more about the events in the [Event Monitoring](#event-monitoring) section.
293
+
294
+
295
+ ### Page Interface
296
+
297
+ The Page Interface is the heart of your automation! 🌟 This domain provides precise control over individual pages, representing a single tab in the browser and enabling interactions with web content, event handling, screenshots, and much more. This is where the real magic happens!
298
+
299
+ Here's how to use the Page interface:
300
+
301
+ ```python
302
+ async def page_examples():
303
+ # Smooth navigation, even on protected sites
304
+ await page.go_to('https://example.com')
305
+
306
+ # Retrieve information about the page
307
+ current_url = await page.current_url
308
+ content = await page.page_source
309
+
310
+ # Capture perfect screenshots
311
+ await page.get_screenshot('/screenshots/evidence.png')
312
+
313
+ # Handle cookies with ease
314
+ cookies = await page.get_cookies()
315
+ await page.set_cookies([{'name': 'session', 'value': 'xyz123'}])
316
+
317
+ # Execute JavaScript on the page
318
+ await page.execute_script('return document.title')
319
+ ```
320
+
321
+ #### Page Management
322
+
323
+ Let's explore the awesome ways to control your pages:
324
+
325
+ ##### `async go_to(url: str, timeout=300) -> None`
326
+ Navigate to any URL with smart loading detection - works beautifully with protected sites!
327
+
328
+ ```python
329
+ # Navigate to a URL with custom timeout
330
+ await page.go_to("https://www.example.com", timeout=60)
331
+ ```
332
+
333
+ ##### `async refresh() -> None`
334
+ Refresh your page with a single command - perfect for testing state changes.
335
+
336
+ ```python
337
+ # Refresh the page when needed
338
+ await page.refresh()
339
+ ```
340
+
341
+ ##### `async close() -> None`
342
+ Close the current tab while keeping your browser session alive.
343
+
344
+ ```python
345
+ # Close the page when done
346
+ await page.close()
347
+ ```
348
+
349
+ ##### `async current_url -> str`
350
+ Instantly know exactly where you are in your automation journey.
351
+
352
+ ```python
353
+ # Check the current URL
354
+ url = await page.current_url
355
+ print(f"We are at: {url}")
356
+ ```
357
+
358
+ ##### `async page_source -> str`
359
+ Get the complete HTML source with one command - perfect for scraping or debugging.
360
+
361
+ ```python
362
+ # Capture the complete HTML of the page
363
+ html = await page.page_source
364
+ ```
365
+
366
+ ##### `async get_cookies() -> list[dict]`
367
+ See all cookies for the current page - great for debugging authentication issues!
368
+
369
+ ```python
370
+ # Get all cookies
371
+ cookies = await page.get_cookies()
372
+ for cookie in cookies:
373
+ print(f"Name: {cookie['name']}, Value: {cookie['value']}")
374
+ ```
375
+
376
+ ##### `async set_cookies(cookies: list[dict]) -> None`
377
+ Set up exactly the cookies you need for testing logged-in states or specific scenarios.
378
+
379
+ ```python
380
+ # Set custom cookies
381
+ await page.set_cookies([
382
+ {
383
+ 'name': 'session_id',
384
+ 'value': '12345',
385
+ 'domain': 'example.com',
386
+ 'path': '/',
387
+ 'expires': -1, # Session
388
+ 'secure': True
389
+ }
390
+ ])
391
+ ```
392
+
393
+ ##### `async delete_all_cookies() -> None`
394
+ Start fresh with a clean cookie state - perfect for testing login flows!
395
+
396
+ ```python
397
+ # Clear all cookies to start fresh
398
+ await page.delete_all_cookies()
399
+ ```
400
+
401
+ ##### `async get_screenshot(path: str) -> None`
402
+ Capture visual evidence of your automation in action!
403
+
404
+ ```python
405
+ # Capture a screenshot of the entire page
406
+ await page.get_screenshot("/path/to/screenshot.png")
407
+ ```
408
+
409
+ ##### `async print_to_pdf(path: str) -> None`
410
+ Generate beautiful PDF reports directly from your pages.
411
+
412
+ ```python
413
+ # Convert the page to PDF
414
+ await page.print_to_pdf("/path/to/document.pdf")
415
+ ```
416
+
417
+ ##### `async get_pdf_base64() -> str`
418
+ Get PDF data ready for processing or sending in API requests.
419
+
420
+ ```python
421
+ # Get PDF in base64 format for processing
422
+ pdf_base64 = await page.get_pdf_base64()
423
+ ```
424
+
425
+ ##### `async has_dialog() -> bool`
426
+ Never be surprised by unexpected alerts or confirmations again!
427
+
428
+ ```python
429
+ # Check if there's an alert or confirmation
430
+ if await page.has_dialog():
431
+ # Handle the dialog
432
+ ```
433
+
434
+ ##### `async get_dialog_message() -> str`
435
+ Read what those pesky dialogs are trying to tell you.
436
+
437
+ ```python
438
+ # Read the dialog message
439
+ if await page.has_dialog():
440
+ message = await page.get_dialog_message()
441
+ print(f"Dialog says: {message}")
442
+ ```
443
+
444
+ ##### `async accept_dialog() -> None`
445
+ Automatically dismiss dialogs to keep your automation flowing smoothly.
446
+
447
+ ```python
448
+ # Automatically accept any dialog
449
+ if await page.has_dialog():
450
+ await page.accept_dialog()
451
+ ```
452
+
453
+
454
+ ##### `async set_download_path(path: str) -> None`
455
+ Control exactly where your files go - per page or globally!
456
+
457
+ ```python
458
+ # Configure custom download folder
459
+ await page.set_download_path("/my_downloads")
460
+ ```
461
+
462
+ ##### `async get_network_logs(matches: list[str] = []) -> list`
463
+ See exactly what requests your page is making - filter for just what you need!
464
+
465
+ ```python
466
+ # Capture requests to specific APIs
467
+ await page.enable_network_events() # this is obligatory!
468
+ await page.go_to('https://example.com')
469
+ logs = await page.get_network_logs(['/example-api', '/another-api'])
470
+ ```
471
+
472
+ This will give you detailed insights like:
473
+
474
+ ```python
475
+ {
476
+ "method": "Network.requestWillBeSent",
477
+ "params": {
478
+ "requestId": "764F3179D5C6D0A1A3F67E7C0ECD88DB",
479
+ "loaderId": "764F3179D5C6D0A1A3F67E7C0ECD88DB",
480
+ "documentURL": "https://google.com/",
481
+ "request": {
482
+ "url": "https://google.com/",
483
+ "method": "GET",
484
+ "headers": {
485
+ "Upgrade-Insecure-Requests": "1",
486
+ "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
487
+ "sec-ch-ua": "\"Chromium\";v=\"130\", \"Google Chrome\";v=\"130\", \"Not?A_Brand\";v=\"99\"",
488
+ "sec-ch-ua-mobile": "?0",
489
+ "sec-ch-ua-platform": "\"Linux\""
490
+ },
491
+ "mixedContentType": "none",
492
+ "initialPriority": "VeryHigh",
493
+ "referrerPolicy": "strict-origin-when-cross-origin",
494
+ "isSameSite": true
495
+ },
496
+ "timestamp": 152714.232433,
497
+ "wallTime": 1741737850.444337,
498
+ "initiator": {
499
+ "type": "other"
500
+ },
501
+ "redirectHasExtraInfo": false,
502
+ "type": "Document",
503
+ "frameId": "2EC0B50381EB2D8CF48BE3CF1D933B59",
504
+ "hasUserGesture": false
505
+ }
506
+ }
507
+ ```
508
+
509
+ ##### `async get_network_response_bodies(matches: list[str] = []) -> list`
510
+ Capture the actual response data from your API calls - perfect for validation!
511
+
512
+ ```python
513
+ # Get responses from API requests
514
+ responses = await page.get_network_response_bodies(['google.com'])
515
+ ```
516
+
517
+ You'll get something like this:
518
+
519
+ ```python
520
+ [
521
+ {
522
+ 'body': '...',
523
+ 'base64Encoded': False
524
+ }
525
+ ]
526
+ ```
527
+
528
+ ##### `async get_network_response_body(request_id: str) -> tuple`
529
+ Target specific requests for detailed analysis.
530
+
531
+ ```python
532
+ # Retrieve a specific response body
533
+ logs = await page.get_network_logs(['api/products'])
534
+ body, encoded = await page.get_network_response_body(logs[0]['params']['requestId'])
535
+ ```
536
+
537
+ ##### `async on(event_name: str, callback: callable, temporary: bool = False) -> int`
538
+ Make your automation reactive by responding to page events in real-time! Check out the [Event System](#event-system) section for more details.
539
+
540
+ ##### `async execute_script(script: str, element: WebElement = None)`
541
+ Unleash the full power of JavaScript directly from your Python code!
542
+
543
+ ```python
544
+ # Execute JavaScript on the page
545
+ title = await page.execute_script('return document.title')
546
+
547
+ # Execute JavaScript on a specific element
548
+ button = await page.find_element(By.CSS_SELECTOR, 'button')
549
+ await page.execute_script('arguments[0].click()', button)
550
+ ```
551
+
552
+ ### WebElement Interface
553
+
554
+ A WebElement represents a DOM element in the browser, allowing you to interact with it in ways that simulate real human behavior. This section explores the most useful methods for interacting with web elements.
555
+
556
+
557
+ ##### `value -> str`
558
+ Get the current value of the element - perfect for checking form fields.
559
+
560
+ ```python
561
+ # Check the value of an input field
562
+ input_element = await page.find_element(By.CSS_SELECTOR, 'input')
563
+ print(f"Current value: {input_element.value}")
564
+ ```
565
+
566
+ ##### `class_name -> str`
567
+ Access the element's CSS classes - useful for style verification.
568
+
569
+ ```python
570
+ # Check CSS classes
571
+ button = await page.find_element(By.CSS_SELECTOR, 'button')
572
+ print(f"Classes: {button.class_name}")
573
+ ```
574
+
575
+ ##### `id -> str`
576
+ Retrieve the element's ID - perfect for unique identification.
577
+
578
+ ```python
579
+ # Get the element's ID
580
+ element = await page.find_element(By.CSS_SELECTOR, 'div')
581
+ print(f"ID: {element.id}")
582
+ ```
583
+
584
+ ##### `is_enabled -> bool`
585
+ Quickly check if the element is enabled for interaction.
586
+
587
+ ```python
588
+ # Check if a button is enabled before clicking
589
+ button = await page.find_element(By.CSS_SELECTOR, 'button')
590
+ if button.is_enabled:
591
+ await button.click()
592
+ ```
593
+
594
+ ##### `async bounds -> list`
595
+ Get the exact coordinates of the element's bounding box.
596
+
597
+ ```python
598
+ # Get the element's coordinates
599
+ element = await page.find_element(By.CSS_SELECTOR, 'div')
600
+ bounds = await element.bounds
601
+ print(f"Coordinates: {bounds}")
602
+ ```
603
+
604
+ ##### `async inner_html -> str`
605
+ Retrieve the element's inner HTML - perfect for scraping!
606
+
607
+ ```python
608
+ # Get the inner HTML of a container
609
+ container = await page.find_element(By.CSS_SELECTOR, '.container')
610
+ html = await container.inner_html
611
+ print(f"Inner HTML: {html}")
612
+ ```
613
+
614
+ ##### `async get_element_text() -> str`
615
+ Get the text contained in the element - clean and ready to use.
616
+
617
+ ```python
618
+ # Get the element's text
619
+ element = await page.find_element(By.CSS_SELECTOR, 'p')
620
+ text = await element.get_element_text()
621
+ print(f"Text: {text}")
622
+ ```
623
+
624
+ ##### `get_attribute(name: str) -> str`
625
+ Access any attribute of the element directly.
626
+
627
+ ```python
628
+ # Get a custom attribute value
629
+ link = await page.find_element(By.CSS_SELECTOR, 'a')
630
+ href = link.get_attribute('href')
631
+ data_id = link.get_attribute('data-id')
632
+ ```
633
+
634
+ ##### `async scroll_into_view()`
635
+ Ensure the element is visible before interacting with it.
636
+
637
+ ```python
638
+ # Make sure the element is visible
639
+ button = await page.find_element(By.CSS_SELECTOR, 'button.footer')
640
+ await button.scroll_into_view()
641
+ await button.click()
642
+ ```
643
+
644
+ ##### `async click(x_offset: int = 0, y_offset: int = 0)`
645
+ Click elements with amazing precision - even with offsets.
646
+
647
+ ```python
648
+ # Click with custom offset
649
+ slider = await page.find_element(By.CSS_SELECTOR, '.slider')
650
+ # Click 10px to the right of the element's center
651
+ await slider.click(x_offset=10, y_offset=0)
652
+ ```
653
+
654
+ ##### `async click_using_js()`
655
+ Click problematic elements that might be obscured or dynamic.
656
+
657
+ ```python
658
+ # Use JavaScript to click difficult elements
659
+ overlay_button = await page.find_element(By.CSS_SELECTOR, '.overlay-button')
660
+ await overlay_button.click_using_js()
661
+ ```
662
+
663
+ ##### `async send_keys(text: str)`
664
+ Send text to form fields quickly.
665
+
666
+ ```python
667
+ # Fill a text field
668
+ input_field = await page.find_element(By.CSS_SELECTOR, 'input[name="username"]')
669
+ await input_field.send_keys("user123")
670
+ ```
671
+
672
+ ##### `async type_keys(text: str)`
673
+ Type realistically, key by key, simulating human input.
674
+
675
+ ```python
676
+ # Type like a real human - with pauses between each key
677
+ password_field = await page.find_element(By.CSS_SELECTOR, 'input[type="password"]')
678
+ await password_field.type_keys("secure_password123") # Realistic typing!
679
+ ```
680
+
681
+ ##### `async get_screenshot(path: str)`
682
+ Capture an image of just the element - perfect for testing or evidence.
683
+
684
+ ```python
685
+ # Capture screenshots of specific elements
686
+ error_message = await page.find_element(By.CSS_SELECTOR, '.error-message')
687
+ await error_message.get_screenshot('/screenshots/error.png')
688
+ ```
689
+
690
+ The WebElement interface also inherits element finding capabilities from FindElementsMixin, allowing you to find child elements:
691
+
692
+ ##### `async find_element(by: By, value: str, raise_exc: bool = True)`
693
+ Find a child element within this element.
694
+
695
+ ```python
696
+ # Find an element within another
697
+ container = await page.find_element(By.CSS_SELECTOR, '.container')
698
+ button = await container.find_element(By.CSS_SELECTOR, 'button')
699
+ await button.click()
700
+ ```
701
+
702
+ ##### `async find_elements(by: By, value: str, raise_exc: bool = True)`
703
+ Find all matching child elements within this element.
704
+
705
+ ```python
706
+ # Find all items in a list
707
+ list_container = await page.find_element(By.CSS_SELECTOR, 'ul.items')
708
+ list_items = await list_container.find_elements(By.CSS_SELECTOR, 'li')
709
+ # Iterate through found items
710
+ for item in list_items:
711
+ text = await item.get_element_text()
712
+ print(f"Item: {text}")
713
+ ```
714
+
715
+ ##### `async wait_element(by: By, value: str, timeout: int = 10, raise_exc: bool = True)`
716
+ Wait until a child element appears within this element.
717
+
718
+ ```python
719
+ # Wait for an element to appear within another
720
+ modal = await page.find_element(By.CSS_SELECTOR, '.modal')
721
+ # Wait up to 5 seconds for confirm button to appear in the modal
722
+ confirm_button = await modal.wait_element(By.CSS_SELECTOR, '.confirm', timeout=5)
723
+ await confirm_button.click()
724
+ ```
725
+
726
+ ## 🚀 Advanced Features
727
+
728
+ ### Event System
729
+
730
+ Pydoll's event system is where the magic really happens! Monitor and react to browser events in real-time for incredibly dynamic automation.
731
+
732
+ ##### `async on(event_name: str, callback: callable, temporary: bool = False) -> int`
733
+ Register custom callbacks for any browser event. First, enable the events you want to track. Keep in mind that you can use the global or local event listener.
734
+ If you want to use the local listener, enable the events on the page instance.
735
+ It's really useful, don't you think?
736
+
737
+ ```python
738
+ from pydoll.events.page import PageEvents
739
+ # Monitors all navigation events on any page
740
+ async def on_page_loaded(event):
741
+ print(f"🌐 Navigating to: {event['params'].get('url')}")
742
+
743
+ await browser.enable_page_events() # Activates page events
744
+ await browser.on(PageEvents.PAGE_LOADED, on_page_loaded) # Global listener!
745
+ # Needs to be locally? Use the page.on method!
746
+ await page.on(PageEvents.PAGE_LOADED, on_page_loaded)
747
+ ```
748
+
749
+ Need to pass extra parameters to your callback? No problem!
750
+
751
+ ```python
752
+ from functools import partial
753
+
754
+ async def on_page_loaded(page, event):
755
+ print(f"📄 Page loaded: {await page.current_url}")
756
+
757
+ await browser.on(PageEvents.PAGE_LOADED, partial(on_page_loaded, page))
758
+ ```
759
+
760
+ ##### `async enable_page_events() -> None`
761
+ Track everything happening on your pages - loading states, navigation, DOM changes, and more! This works globally or locally. Just use the browser or the page instance to enable the events.
762
+
763
+ ```python
764
+ # Enables page event monitoring
765
+ await browser.enable_page_events() # global
766
+ ```
767
+
768
+
769
+ ##### `async enable_network_events() -> None`
770
+ See all network activity in real-time - perfect for debugging or monitoring specific API calls!
771
+
772
+ ```python
773
+ from pydoll.events.network import NetworkEvents
774
+
775
+ async def on_request(event):
776
+ print(f"🔄 Request to: {event['params']['request']['url']} will be sent")
777
+
778
+ await browser.enable_network_events()
779
+ await browser.on(NetworkEvents.REQUEST_WILL_BE_SENT, on_request)
780
+
781
+ await page.go_to('https://www.google.com') # This will trigger the on_request callback
782
+ ```
783
+
784
+ ##### `async enable_dom_events() -> None`
785
+ Watch the page structure change in real-time and react accordingly!
786
+
787
+ ```python
788
+ from pydoll.events.dom import DomEvents
789
+
790
+ async def on_dom_event(event):
791
+ print(f"🔄 The DOM has been updated!")
792
+
793
+ await browser.enable_dom_events()
794
+ await browser.on(DomEvents.DOCUMENT_UPDATED, on_dom_event)
795
+ ```
796
+
797
+ ##### `async enable_fetch_events(handle_auth_requests: bool = False, resource_type: str = '') -> None`
798
+ The ultimate power tool - intercept and modify network requests before they're even sent!
799
+
800
+ ```python
801
+ # Intercepts all network requests
802
+ from pydoll.events.fetch import FetchEvents
803
+ from pydoll.commands.fetch import FetchCommands
804
+ from functools import partial
805
+
806
+ async def interceptor(page, event):
807
+ request_id = event['params']['requestId']
808
+ request_url = event['params']['request']['url']
809
+ print(f"🕵️ Intercepting request to: {request_url}")
810
+
811
+ # Customize the request however you want!
812
+ await page._execute_command(
813
+ FetchCommands.continue_request(
814
+ request_id=request_id,
815
+ method='GET', # change the HTTP method
816
+ headers={'X-Custom-Header': 'CustomValue'}, # add your own headers
817
+ post_data='Hello World', # modify the request body
818
+ url='https://www.google.com', # even change the destination URL!
819
+ intercept_response=True # and intercept the response too
820
+ )
821
+ )
822
+
823
+ await browser.enable_fetch_events(resource_type='xhr') # only intercept XHR requests
824
+ await browser.on(FetchEvents.REQUEST_PAUSED, partial(interceptor, page))
825
+ ```
826
+
827
+ With this power, you can transform your automation into something truly intelligent!
828
+
829
+ ##### `async disable_fetch_events() -> None`
830
+ Turn off request interception when you're done.
831
+
832
+ ```python
833
+ # Disables request interception
834
+ await browser.disable_fetch_events()
835
+ ```
836
+
837
+ ### Concurrent Scraping
838
+
839
+ Scrape multiple pages simultaneously with extraordinary performance:
840
+
841
+ ```python
842
+ async def concurrent_example():
843
+ pages = [await browser.get_page() for _ in range(10)]
844
+ # Parallel scraping with intelligent resource management
845
+ results = await asyncio.gather(
846
+ *(scrape_page(page) for page in pages)
847
+ )
848
+ # Just declare the scrape_page method and see the magic happens!
849
+ ```
850
+
851
+ ## 🌟 Best Practices
852
+
853
+ Get the most out of Pydoll with these tips:
854
+
855
+ - Use asynchronous patterns throughout your code for maximum performance
856
+ - Prefer specific selectors (IDs, data attributes) over generic ones
857
+ - Add proper error handling for robust automation
858
+ - Use the event system for reactive scenarios rather than constant polling
859
+ - Close browser instances when done to avoid memory leaks
860
+
861
+ ## 🤝 Contributing
862
+
863
+ We'd love your help making Pydoll even better! Check out our contribution guidelines to get started. Whether it's fixing bugs, adding features, or improving documentation - all contributions are welcome!
864
+