mcp-query-table 0.3.6__tar.gz → 0.3.8__tar.gz

mcp_query_table-0.3.8/.gitignore

@@ -0,0 +1,174 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mcp_query_table
-Version: 0.3.6
+Version: 0.3.8
 Summary: query table from website, support MCP
 Author-email: wukan <wu-kan@163.com>
 License: MIT License
@@ -24,24 +24,25 @@ License: MIT License
         LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
-        
-Keywords: playwright,mcp,table,iwencai,tdx,eastmoney
+License-File: LICENSE
+Keywords: eastmoney,iwencai,mcp,playwright,table,tdx
 Classifier: Development Status :: 4 - Beta
 Classifier: Programming Language :: Python
 Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: pandas
 Requires-Dist: loguru
-Requires-Dist: playwright
 Requires-Dist: mcp
-Dynamic: license-file
+Requires-Dist: pandas
+Requires-Dist: playwright
+Requires-Dist: playwright-stealth
+Requires-Dist: setuptools
+Requires-Dist: tabulate
+Description-Content-Type: text/markdown
 
 # mcp_query_table
 
 1. 基于`playwright`实现的财经网页表格爬虫，支持`Model Context Protocol (MCP) `。目前可查询来源为
 
-    - [同花顺i问财](http://iwencai.com/)
+    - [同花顺问财](http://iwencai.com/)
     - [通达信问小达](https://wenda.tdx.com.cn/)
     - [东方财富条件选股](https://xuangu.eastmoney.com/)
 
@@ -70,7 +71,7 @@ from mcp_query_table import *
 
 
 async def main() -> None:
-    async with BrowserManager(endpoint="http://127.0.0.1:9222", executable_path=None, debug=True) as bm:
+    async with BrowserManager(endpoint="http://127.0.0.1:9222", executable_path=None, devtools=True) as bm:
         # 问财需要保证浏览器宽度>768，防止界面变成适应手机
         page = await bm.get_page()
         df = await query(page, '收益最好的200只ETF', query_type=QueryType.ETF, max_page=1, site=Site.THS)
@@ -128,19 +129,21 @@ if __name__ == '__main__':
 
 后期会根据不同的网站改版情况，使用更适合的方法。
 
-## MCP支持
+## 无头模式
 
-确保可以在控制台中执行`python -m mcp_query_table -h`。如果不能，可能要先`pip install mcp_query_table`
+无头模式运行速度更快，但部分网站需要提前登录，所以，无头模式一定要指定`user_data_dir`，否则会出现需要登录的情况。
 
-在`Cline`中可以配置如下。其中`command`是`python`的绝对路径，`executable_path`是`Chrome`的绝对路径，`timeout`是超时时间，单位为秒。
-在各`AI`平台中由于返回时间常需1分钟以上，所以需要设置大的超时时间。
+- `endpoint=None`时，`headless=True`可无头启动新浏览器实例。指定`executable_path`和`user_data_dir`，才能确保无头模式下正常运行。
+- `endpoint`以`http://`开头，连接`CDP`模式启动的有头浏览器，参数必有`--remote-debugging-port`。`executable_path`为本地浏览器路径。
+- `endpoint`以`ws://`开头，连接远程`Playwright Server`。也是无头模式，但无法指定`user_data_dir`，所以使用受限
+    - 参考：https://playwright.dev/python/docs/docker#running-the-playwright-server
 
-`endpoint`支持两方式，一种是`cdp_endpoint`方式，一种是`ws_endpoint`方式。
+## MCP支持
 
-- cdp方式：通过启动时加参数`--remote-debugging-port=9222`来启动浏览器
-- ws方式：服务器上`docker run -p 3000:3000 --rm --init -it --workdir /home/pwuser --user pwuser mcr.microsoft.com/playwright:v1.51.0-noble /bin/sh -c "npx -y playwright@1.51.0 run-server --port 3000 --host 0.0.0.0"`
+确保可以在控制台中执行`python -m mcp_query_table -h`。如果不能，可能要先`pip install mcp_query_table`
 
-参考：https://playwright.dev/python/docs/docker#remote-connection
+在`Cline`中可以配置如下。其中`command`是`python`的绝对路径，`timeout`是超时时间，单位为秒。 在各`AI`
+平台中由于返回时间常需1分钟以上，所以需要设置大的超时时间。
 
 ### STDIO方式
 
@@ -170,7 +173,7 @@ if __name__ == '__main__':
 先在控制台中执行如下命令，启动`MCP`服务
 
 ```commandline
-python -m mcp_query_table --format markdown --transport sse --port 8000
+python -m mcp_query_table --format markdown --transport sse --port 8000 --endpoint http://127.0.0.1:9222
 ```
 
 然后就可以连接到`MCP`服务了
@@ -189,7 +192,7 @@ python -m mcp_query_table --format markdown --transport sse --port 8000
 ## 使用`MCP Inspector`进行调试
 
 ```commandline
-npx @modelcontextprotocol/inspector python -m mcp_query_table --format markdown
+npx @modelcontextprotocol/inspector python -m mcp_query_table --format markdown --endpoint http://127.0.0.1:9222
 ```
 
 打开浏览器并翻页是一个比较耗时的操作，会导致`MCP Inspector`页面超时，可以`http://localhost:5173/?timeout=300000`
@@ -219,6 +222,5 @@ npx @modelcontextprotocol/inspector python -m mcp_query_table --format markdown
 ![streamlit](docs/img/streamlit.png)
 
 ## 参考
-
-- [Playwright](https://playwright.dev/python/docs/intro)
 - [Selenium webdriver无法附加到edge实例，edge的--remote-debugging-port选项无效](https://blog.csdn.net/qq_30576521/article/details/142370538)
+- https://github.com/AtuboDad/playwright_stealth/issues/31

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/README.md

@@ -2,7 +2,7 @@
 
 1. 基于`playwright`实现的财经网页表格爬虫，支持`Model Context Protocol (MCP) `。目前可查询来源为
 
-    - [同花顺i问财](http://iwencai.com/)
+    - [同花顺问财](http://iwencai.com/)
     - [通达信问小达](https://wenda.tdx.com.cn/)
     - [东方财富条件选股](https://xuangu.eastmoney.com/)
 
@@ -31,7 +31,7 @@ from mcp_query_table import *
 
 
 async def main() -> None:
-    async with BrowserManager(endpoint="http://127.0.0.1:9222", executable_path=None, debug=True) as bm:
+    async with BrowserManager(endpoint="http://127.0.0.1:9222", executable_path=None, devtools=True) as bm:
         # 问财需要保证浏览器宽度>768，防止界面变成适应手机
         page = await bm.get_page()
         df = await query(page, '收益最好的200只ETF', query_type=QueryType.ETF, max_page=1, site=Site.THS)
@@ -89,19 +89,21 @@ if __name__ == '__main__':
 
 后期会根据不同的网站改版情况，使用更适合的方法。
 
-## MCP支持
+## 无头模式
 
-确保可以在控制台中执行`python -m mcp_query_table -h`。如果不能，可能要先`pip install mcp_query_table`
+无头模式运行速度更快，但部分网站需要提前登录，所以，无头模式一定要指定`user_data_dir`，否则会出现需要登录的情况。
 
-在`Cline`中可以配置如下。其中`command`是`python`的绝对路径，`executable_path`是`Chrome`的绝对路径，`timeout`是超时时间，单位为秒。
-在各`AI`平台中由于返回时间常需1分钟以上，所以需要设置大的超时时间。
+- `endpoint=None`时，`headless=True`可无头启动新浏览器实例。指定`executable_path`和`user_data_dir`，才能确保无头模式下正常运行。
+- `endpoint`以`http://`开头，连接`CDP`模式启动的有头浏览器，参数必有`--remote-debugging-port`。`executable_path`为本地浏览器路径。
+- `endpoint`以`ws://`开头，连接远程`Playwright Server`。也是无头模式，但无法指定`user_data_dir`，所以使用受限
+    - 参考：https://playwright.dev/python/docs/docker#running-the-playwright-server
 
-`endpoint`支持两方式，一种是`cdp_endpoint`方式，一种是`ws_endpoint`方式。
+## MCP支持
 
-- cdp方式：通过启动时加参数`--remote-debugging-port=9222`来启动浏览器
-- ws方式：服务器上`docker run -p 3000:3000 --rm --init -it --workdir /home/pwuser --user pwuser mcr.microsoft.com/playwright:v1.51.0-noble /bin/sh -c "npx -y playwright@1.51.0 run-server --port 3000 --host 0.0.0.0"`
+确保可以在控制台中执行`python -m mcp_query_table -h`。如果不能，可能要先`pip install mcp_query_table`
 
-参考：https://playwright.dev/python/docs/docker#remote-connection
+在`Cline`中可以配置如下。其中`command`是`python`的绝对路径，`timeout`是超时时间，单位为秒。 在各`AI`
+平台中由于返回时间常需1分钟以上，所以需要设置大的超时时间。
 
 ### STDIO方式
 
@@ -131,7 +133,7 @@ if __name__ == '__main__':
 先在控制台中执行如下命令，启动`MCP`服务
 
 ```commandline
-python -m mcp_query_table --format markdown --transport sse --port 8000
+python -m mcp_query_table --format markdown --transport sse --port 8000 --endpoint http://127.0.0.1:9222
 ```
 
 然后就可以连接到`MCP`服务了
@@ -150,7 +152,7 @@ python -m mcp_query_table --format markdown --transport sse --port 8000
 ## 使用`MCP Inspector`进行调试
 
 ```commandline
-npx @modelcontextprotocol/inspector python -m mcp_query_table --format markdown
+npx @modelcontextprotocol/inspector python -m mcp_query_table --format markdown --endpoint http://127.0.0.1:9222
 ```
 
 打开浏览器并翻页是一个比较耗时的操作，会导致`MCP Inspector`页面超时，可以`http://localhost:5173/?timeout=300000`
@@ -180,6 +182,5 @@ npx @modelcontextprotocol/inspector python -m mcp_query_table --format markdown
 ![streamlit](docs/img/streamlit.png)
 
 ## 参考
-
-- [Playwright](https://playwright.dev/python/docs/intro)
-- [Selenium webdriver无法附加到edge实例，edge的--remote-debugging-port选项无效](https://blog.csdn.net/qq_30576521/article/details/142370538)
+- [Selenium webdriver无法附加到edge实例，edge的--remote-debugging-port选项无效](https://blog.csdn.net/qq_30576521/article/details/142370538)
+- https://github.com/AtuboDad/playwright_stealth/issues/31

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/mcp_query_table/__init__.py

@@ -5,3 +5,7 @@ from .tool import BrowserManager, query, chat
 
 TIMEOUT = 1000 * 60 * 3  # 3分钟，在抓取EventStream数据时等待数据返回，防止外层30秒超时
 TIMEOUT_60 = 1000 * 60  # 1分钟
+
+# TODO 临时测试
+# TIMEOUT = None
+# TIMEOUT_60 = None

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/mcp_query_table/__main__.py

@@ -1,3 +1,5 @@
+import getpass
+
 from mcp_query_table.server import serve
 
 
@@ -11,10 +13,11 @@ def main():
     parser.add_argument("--format", type=str, help="输出格式",
                         default='markdown', choices=['markdown', 'csv', 'json'])
     parser.add_argument("--endpoint", type=str, help="浏览器CDP地址/WS地址",
-                        default="http://127.0.0.1:9222")
-    parser.add_argument("--executable_path", type=str, help="浏览器类型",
-                        default=r'C:\Program Files\Google\Chrome\Application\chrome.exe')
-
+                        nargs="?", default=r'http://127.0.0.1:9222')
+    parser.add_argument("--executable_path", type=str, help="浏览器路径",
+                        nargs="?", default=r'C:\Program Files\Google\Chrome\Application\chrome.exe')
+    parser.add_argument("--user_data_dir", type=str, help="浏览器用户数据目录",
+                        nargs="?", default=rf'C:\Users\{getpass.getuser()}\AppData\Local\Google\Chrome\User Data')
     parser.add_argument("--transport", type=str, help="传输类型",
                         default='stdio', choices=['stdio', 'sse'])
     parser.add_argument("--host", type=str, help="MCP服务端绑定地址",
@@ -22,7 +25,8 @@ def main():
     parser.add_argument("--port", type=int, help="MCP服务端绑定端口",
                         default='8000')
     args = parser.parse_args()
-    serve(args.format, args.endpoint, args.executable_path,
+    serve(args.format, args.endpoint,
+          args.executable_path, args.user_data_dir,
           args.transport, args.host, args.port)
 
 

mcp_query_table-0.3.8/mcp_query_table/_version.py

@@ -0,0 +1 @@
+__version__ = "0.3.8"

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/mcp_query_table/enums.py

@@ -18,7 +18,7 @@ class Site(Enum):
     """站点"""
     EastMoney = '东方财富'  # 东方财富 条件选股
     TDX = '通达信'  # 通达信 问小达
-    THS = '同花顺'  # 同花顺 i问财
+    THS = '同花顺'  # 同花顺 问财
 
 
 class Provider(Enum):

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/mcp_query_table/providers/baidu.py

@@ -8,7 +8,7 @@ import json
 from playwright.async_api import Page
 
 import mcp_query_table
-from mcp_query_table.tool import GlobalVars, split_images
+from mcp_query_table.utils import split_images, GlobalVars
 
 _PAGE0_ = "https://chat.baidu.com/search"
 _PAGE1_ = "https://chat.baidu.com/aichat/api/conversation"

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/mcp_query_table/providers/n.py

@@ -3,10 +3,11 @@
 """
 import json
 
+from loguru import logger
 from playwright.async_api import Page
 
 import mcp_query_table
-from mcp_query_table.tool import GlobalVars, is_image
+from mcp_query_table.utils import is_image, GlobalVars
 
 _PAGE0_ = "https://www.n.cn"
 _PAGE1_ = "https://www.n.cn/search"
@@ -78,6 +79,8 @@ async def chat(page: Page,
     str
         回答
     """
+    logger.warning("纳米搜索。不登录可以使用。但无头模式要指定`user_data_dir`才能正常工作")
+
     if not create:
         if not page.url.startswith(_PAGE1_):
             create = True
@@ -102,6 +105,7 @@ async def chat(page: Page,
         textbox = page.get_by_role("textbox", name=name)
         await textbox.fill(prompt)
         await textbox.press("Enter")
+        # await page.screenshot(path="n.png")
     await on_response(await response_info.value)
 
     return G.get_text()

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/mcp_query_table/providers/yuanbao.py

@@ -4,10 +4,11 @@
 import json
 import re
 
+from loguru import logger
 from playwright.async_api import Page
 
 import mcp_query_table
-from mcp_query_table.tool import GlobalVars, split_images
+from mcp_query_table.utils import split_images, GlobalVars
 
 _PAGE0_ = "https://yuanbao.tencent.com/"
 _PAGE1_ = "https://yuanbao.tencent.com/api/chat"
@@ -69,6 +70,8 @@ async def chat(page: Page,
                create: bool,
                files: list[str]
                ) -> str:
+    logger.info("腾讯元宝。登录才可以使用。无头模式时要指定`user_data_dir`才能正常工作")
+
     if not page.url.startswith(_PAGE0_):
         create = True
 

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/mcp_query_table/server.py

@@ -14,9 +14,13 @@ class QueryServer:
         self.format: str = "markdown"
         self.browser = None
 
-    def start(self, format, endpoint, executable_path):
+    def start(self, format, endpoint, executable_path, user_data_dir):
         self.format: str = format
-        self.browser = BrowserManager(endpoint=endpoint, executable_path=executable_path, debug=False)
+        self.browser = BrowserManager(endpoint=endpoint,
+                                      executable_path=executable_path,
+                                      user_data_dir=user_data_dir,
+                                      devtools=False,
+                                      headless=True)
 
     async def query(self, query_input: str, query_type: QueryType, max_page: int, site: Site):
         page = await self.browser.get_page()
@@ -54,7 +58,8 @@ async def query(
     return await qsv.query(query_input, query_type, max_page, site)
 
 
-@mcp.tool(description="大语言模型对话")
+# chat功能不通过mcp暴露，因为在Cline等客户端中本就有LLM功能，反而导致返回的数据没有正确提交
+# @mcp.tool(description="大语言模型对话")
 async def chat(
         prompt: Annotated[str, Field(description="提示词。如：`9.9大还是9.11大？`")],
         create: Annotated[bool, Field(default=False, description="是否创建新对话")],
@@ -65,12 +70,15 @@ async def chat(
     return await qsv.chat(prompt, create, files, provider)
 
 
-def serve(format, endpoint, executable_path, transport, host, port):
-    qsv.start(format, endpoint, executable_path)
-    logger.info(f"{format=},{transport=}")
-    logger.info(f"{endpoint=},{executable_path=}")
+def serve(format, endpoint, executable_path, user_data_dir, transport, host, port):
+    qsv.start(format, endpoint, executable_path, user_data_dir)
+    logger.info(f"{endpoint=}")
+    logger.info(f"{executable_path=}")
+    logger.info(f"{user_data_dir=}")
     if transport == 'sse':
-        logger.info(f"{host=},{port=}", transport, host, port)
+        logger.info(f"{transport=},{format=},{host=},{port=}")
+    else:
+        logger.info(f"{transport=},{format=}")
 
     mcp.settings.host = host
     mcp.settings.port = port

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/mcp_query_table/sites/iwencai.py

@@ -1,5 +1,5 @@
 """
-同花顺 i问财
+同花顺问财
 https://www.iwencai.com/
 
 1. 一定要保证浏览器宽度>768，防止界面变成适应手机
@@ -10,8 +10,10 @@ import re
 import pandas as pd
 from loguru import logger
 from playwright.async_api import Page
+from playwright_stealth import stealth_async
 
 from mcp_query_table.enums import QueryType
+from mcp_query_table.utils import FixedConfig
 
 # 初次查询页面
 _PAGE1_ = 'https://www.iwencai.com/customized/chart/get-robot-data'
@@ -140,7 +142,6 @@ json_data['answer']['components'][0]['data']['meta']['extra']['row_count']
 
 async def on_response(response):
     if response.url == _PAGE1_:
-        # TODO 不支持headless模式，需要以后解决
         P.update(*get_robot_data(await response.json()))
     if response.url == _PAGE2_:
         P.update(*getDataList(await response.json()))
@@ -153,6 +154,8 @@ async def query(page: Page,
     querytype = _querytype_.get(type_, None)
     assert querytype is not None, f"不支持的类型:{type_}"
 
+    await stealth_async(page, FixedConfig())
+
     await page.route(re.compile(r'.*\.(?:jpg|jpeg|png|gif|webp)(?:$|\?)'), lambda route: route.abort())
 
     P.reset()

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/mcp_query_table/tool.py

@@ -1,8 +1,9 @@
+import getpass
 import subprocess
 import sys
 import time
 from pathlib import Path
-from typing import Optional, List, Tuple
+from typing import Optional
 from urllib.parse import urlparse
 
 import pandas as pd
@@ -26,6 +27,7 @@ def create_detached_process(command):
         kwargs.update({
             'start_new_session': True  # 创建新的会话
         })
+    logger.info(f"Popen: {command}")
     return subprocess.Popen(command, **kwargs)
 
 
@@ -59,6 +61,21 @@ def get_executable_path(executable_path) -> Optional[str]:
     return None
 
 
+def get_user_data_dir(user_data_dir) -> Optional[str]:
+    """获取浏览器可用户目录"""
+    browsers = {
+        "default": user_data_dir,
+        "chrome.exe": rf'C:\Users\{getpass.getuser()}\AppData\Local\Google\Chrome\User Data\Default',
+        "msedge.exe": rf"C:\Users\{getpass.getuser()}\AppData\Local\Microsoft\Edge\User Data\Default",
+    }
+    for k, v in browsers.items():
+        if v is None:
+            continue
+        if Path(v).exists():
+            return v
+    return None
+
+
 class BrowserManager:
     async def __aenter__(self):
         return self
@@ -67,24 +84,36 @@ class BrowserManager:
         await self.cleanup()
 
     def __init__(self,
-                 endpoint: Optional[str] = None,
+                 endpoint: Optional[str],
                  executable_path: Optional[str] = None,
-                 debug: bool = False):
+                 devtools: bool = False,
+                 headless: bool = True,
+                 user_data_dir: Optional[str] = None):
         """
 
         Parameters
         ----------
-        endpoint:str
-            浏览器CDP地址/WS地址
+        endpoint:str or None
+            浏览器CDP地址/WS地址。
+            如果为None，则直接启动浏览器实例。可用无头模式。建议指定用户数据目录，否则可能无法使用某些需要登录的网站
         executable_path:str
             浏览器可执行文件路径。推荐使用chrome，因为Microsoft Edge必须在任务管理器中完全退出才能启动调试端口
-        debug:bool
+        devtools:bool
             是否显示开发者工具
+        headless:bool
+            是否无头模式启动浏览器
+        user_data_dir:str
+            浏览器用户数据目录。无头模式。强烈建议指定用户数据目录，否则可能无法使用某些需要登录的网站
 
         """
-        self.endpoint = endpoint or 'http://127.0.0.1:9222'
+        if devtools:
+            headless = False
+
+        self.endpoint = endpoint
         self.executable_path = executable_path
-        self.debug = debug
+        self.devtools = devtools
+        self.headless = headless
+        self.user_data_dir = user_data_dir
 
         self.playwright: Optional[Playwright] = None
         self.browser = None
@@ -102,8 +131,9 @@ class BrowserManager:
         """连接本地浏览器"""
         port = urlparse(self.endpoint).port
         executable_path = get_executable_path(self.executable_path)
+        name = Path(executable_path).name
         command = [executable_path, f'--remote-debugging-port={port}', '--start-maximized']
-        if self.debug:
+        if self.devtools:
             command.append('--auto-open-devtools-for-tabs')
 
         for i in range(2):
@@ -113,13 +143,12 @@ class BrowserManager:
                 break
             except:
                 if i == 0:
-                    logger.info(f"start browser:{command}")
                     create_detached_process(command)
-                    time.sleep(3)
+                    time.sleep(5)
                     continue
                 if i == 1:
                     raise ConnectionError(
-                        f"已提前打开了浏览器，但未开启远程调试端口？请关闭浏览器全部进程后重试 `taskkill /f /im {Path(executable_path).name}`")
+                        f"已提前打开了浏览器，但未开启远程调试端口？请关闭浏览器全部进程后重试 `taskkill /f /im {name}`")
 
     async def _connect_to_remote(self) -> None:
         """连接远程浏览器"""
@@ -133,6 +162,26 @@ class BrowserManager:
         except:
             raise ConnectionError(f"连接远程浏览器失败，请检查CDP/WS地址和端口是否正确。{self.endpoint}")
 
+    async def _connect_to_launch(self) -> None:
+        logger.info("executable_path={}", self.executable_path)
+        if self.user_data_dir:
+            logger.info("user_data_dir={}", self.user_data_dir)
+            try:
+                self.context = await self.playwright.chromium.launch_persistent_context(
+                    user_data_dir=self.user_data_dir,
+                    executable_path=self.executable_path,
+                    headless=self.headless,
+                    devtools=self.devtools,
+                    timeout=10000, slow_mo=1000)
+            except:
+                raise ConnectionError(f"launch失败，可能已经有浏览器已经打开了数据目录。{self.user_data_dir}")
+        else:
+            logger.warning("未指定浏览器用户数据目录，部分需要的网站可能无法使用")
+            self.browser = await self.playwright.chromium.launch(
+                executable_path=self.executable_path,
+                headless=self.headless,
+                devtools=self.devtools)
+
     async def _launch(self) -> None:
         """启动浏览器，并连接CDP协议
 
@@ -142,16 +191,20 @@ class BrowserManager:
 
         """
         self.playwright = await async_playwright().start()
-
-        if is_local_url(self.endpoint) and is_cdp_url(self.endpoint):
+        if self.endpoint is None:
+            await self._connect_to_launch()
+        elif is_local_url(self.endpoint) and is_cdp_url(self.endpoint):
             await self._connect_to_local()
         else:
             await self._connect_to_remote()
 
-        if len(self.browser.contexts) == 0:
+        if self.browser is None:
+            pass
+        elif len(self.browser.contexts) == 0:
             self.context = await self.browser.new_context()
         else:
             self.context = self.browser.contexts[0]
+
         # 复用打开的page
         for page in self.context.pages:
             # 防止开发者工具被使用
@@ -165,15 +218,10 @@ class BrowserManager:
                 continue
             self.pages.append(page)
 
-    async def _try_launch(self) -> None:
-        if self.browser is None:
-            await self._launch()
-        if not self.browser.is_connected():
-            await self._launch()
-
     async def get_page(self) -> Page:
         """获取可用Page。无空闲标签时会打开新标签"""
-        await self._try_launch()
+        if self.context is None:
+            await self._launch()
 
         # 反复取第一个tab
         while len(self.pages) > 0:
@@ -193,19 +241,6 @@ class BrowserManager:
         self.pages.append(page)
 
 
-class GlobalVars:
-    """全局变量"""
-
-    def __init__(self):
-        self.text = ""
-
-    def set_text(self, text):
-        self.text = text
-
-    def get_text(self):
-        return self.text
-
-
 async def query(
         page: Page,
         query_input: str = "收盘价>100元",
@@ -289,22 +324,3 @@ async def chat(
         return await chat(page, prompt, create, files)
 
     raise ValueError(f"未支持的提供商:{provider}")
-
-
-def is_image(path: str) -> bool:
-    """判断是否是图片文件"""
-    img_ext = ['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.webp']
-    ext = Path(path).suffix.lower()
-    return ext in img_ext
-
-
-def split_images(files: List[str]) -> Tuple[List[str], List[str]]:
-    """图片列表分成两部分"""
-    imgs = []
-    docs = []
-    for f in files:
-        if is_image(f):
-            imgs.append(f)
-        else:
-            docs.append(f)
-    return imgs, docs

mcp_query_table-0.3.8/mcp_query_table/utils.py

@@ -0,0 +1,51 @@
+import random
+import string
+from pathlib import Path
+from typing import List, Tuple
+
+from playwright_stealth import StealthConfig
+
+
+def is_image(path: str) -> bool:
+    """判断是否是图片文件"""
+    img_ext = ['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.webp']
+    ext = Path(path).suffix.lower()
+    return ext in img_ext
+
+
+def split_images(files: List[str]) -> Tuple[List[str], List[str]]:
+    """图片列表分成两部分"""
+    imgs = []
+    docs = []
+    for f in files:
+        if is_image(f):
+            imgs.append(f)
+        else:
+            docs.append(f)
+    return imgs, docs
+
+
+class GlobalVars:
+    """全局变量"""
+
+    def __init__(self):
+        self.text = ""
+
+    def set_text(self, text):
+        self.text = text
+
+    def get_text(self):
+        return self.text
+
+
+# https://github.com/AtuboDad/playwright_stealth/issues/31#issuecomment-2342541305
+class FixedConfig(StealthConfig):
+
+    @property
+    def enabled_scripts(self):
+        key = "".join(random.choices(string.ascii_letters, k=10))
+        for script in super().enabled_scripts:
+            if "const opts" in script:
+                yield script.replace("const opts", f"window.{key}")
+                continue
+            yield script.replace("opts", f"window.{key}")

{mcp_query_table-0.3.6 → mcp_query_table-0.3.8}/pyproject.toml

@@ -1,7 +1,3 @@
-[build-system]
-requires = ["setuptools"]
-build-backend = "setuptools.build_meta"
-
 [project]
 name = "mcp_query_table"
 authors = [
@@ -20,17 +16,23 @@ dependencies = [
     "pandas",
     "loguru",
     "playwright",
+    "playwright-stealth",
     "mcp",
+    "setuptools", # playwright-stealth中要使用pkg_resources，而pkg_resources在setuptools中
+    "tabulate"
 ]
 dynamic = ["version"]
 
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
 
-[tool.setuptools]
-packages = [
-    "mcp_query_table",
-    "mcp_query_table.sites",
-    "mcp_query_table.providers",
-]
+[tool.hatch.version]
+path = "mcp_query_table/_version.py"
+
+[tool.hatch.build.targets.wheel]
+packages = ["mcp_query_table"]
+include-package-data = true
 
-[tool.setuptools.dynamic]
-version = { attr = "mcp_query_table._version.__version__" }
+[tool.hatch.build.targets.sdist]
+include = ["mcp_query_table*"]

mcp_query_table-0.3.6/mcp_query_table/_version.py

@@ -1 +0,0 @@
-__version__ = "0.3.6"

mcp_query_table-0.3.6/mcp_query_table.egg-info/PKG-INFO

@@ -1,224 +0,0 @@
-Metadata-Version: 2.4
-Name: mcp_query_table
-Version: 0.3.6
-Summary: query table from website, support MCP
-Author-email: wukan <wu-kan@163.com>
-License: MIT License
-        
-        Copyright (c) 2025 伍侃
-        
-        Permission is hereby granted, free of charge, to any person obtaining a copy
-        of this software and associated documentation files (the "Software"), to deal
-        in the Software without restriction, including without limitation the rights
-        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-        copies of the Software, and to permit persons to whom the Software is
-        furnished to do so, subject to the following conditions:
-        
-        The above copyright notice and this permission notice shall be included in all
-        copies or substantial portions of the Software.
-        
-        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-        SOFTWARE.
-        
-Keywords: playwright,mcp,table,iwencai,tdx,eastmoney
-Classifier: Development Status :: 4 - Beta
-Classifier: Programming Language :: Python
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: pandas
-Requires-Dist: loguru
-Requires-Dist: playwright
-Requires-Dist: mcp
-Dynamic: license-file
-
-# mcp_query_table
-
-1. 基于`playwright`实现的财经网页表格爬虫，支持`Model Context Protocol (MCP) `。目前可查询来源为
-
-    - [同花顺i问财](http://iwencai.com/)
-    - [通达信问小达](https://wenda.tdx.com.cn/)
-    - [东方财富条件选股](https://xuangu.eastmoney.com/)
-
-   实盘时，如果某网站宕机或改版，可以立即切换到其他网站。(注意：不同网站的表格结构不同，需要提前做适配)
-
-2. 基于`playwright`实现的大语言模型调用爬虫。目前可用来源为
-    - [纳米搜索](https://www.n.cn/)
-    - [腾讯元宝](https://yuanbao.tencent.com/)
-    - [百度AI搜索](https://chat.baidu.com/)
-
-   `RooCode`提供了`Human Reply`功能。但发现`纳米搜索`网页版复制时格式破坏，所以研发了此功能
-
-## 安装
-
-```commandline
-pip install -i https://pypi.org/simple --upgrade mcp_query_table
-pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --upgrade mcp_query_table
-```
-
-## 使用
-
-```python
-import asyncio
-
-from mcp_query_table import *
-
-
-async def main() -> None:
-    async with BrowserManager(endpoint="http://127.0.0.1:9222", executable_path=None, debug=True) as bm:
-        # 问财需要保证浏览器宽度>768，防止界面变成适应手机
-        page = await bm.get_page()
-        df = await query(page, '收益最好的200只ETF', query_type=QueryType.ETF, max_page=1, site=Site.THS)
-        print(df.to_markdown())
-        df = await query(page, '年初至今收益率前50', query_type=QueryType.Fund, max_page=1, site=Site.TDX)
-        print(df.to_csv())
-        df = await query(page, '流通市值前10的行业板块', query_type=QueryType.Index, max_page=1, site=Site.TDX)
-        print(df.to_csv())
-        # TODO 东财翻页要提前登录
-        df = await query(page, '今日涨幅前5的概念板块;', query_type=QueryType.Board, max_page=3, site=Site.EastMoney)
-        print(df)
-
-        output = await chat(page, "1+2等于多少？", provider=Provider.YuanBao)
-        print(output)
-        output = await chat(page, "3+4等于多少？", provider=Provider.YuanBao, create=True)
-        print(output)
-
-        print('done')
-        bm.release_page(page)
-        await page.wait_for_timeout(2000)
-
-
-if __name__ == '__main__':
-    asyncio.run(main())
-
-```
-
-## 注意事项
-
-1. 浏览器最好是`Chrome`。如一定要使用`Edge`,除了关闭`Edge`所有窗口外，还要在任务管理器关闭`Microsoft Edge`
-   的所有进程，即`taskkill /f /im msedge.exe`
-2. 浏览器要保证窗口宽度，防止部分网站自动适配成手机版，导致表格查询失败
-3. 如有网站账号，请提前登录。此工具无自动登录功能
-4. 不同网站的表格结构不同，同条件返回股票数量也不同。需要查询后做适配
-
-## 工作原理
-
-不同于`requests`，`playwright`是基于浏览器的，模拟用户在浏览器中的操作。
-
-1. 不需要解决登录问题
-2. 不需要解决请求构造、响应解析
-3. 可以直接获取表格数据，所见即所得
-4. 运行速度慢于`requests`，但开发效率高
-
-数据的获取有：
-
-1. 直接解析HTML表格
-    1. 数字文本化了，不利于后期研究
-    2. 适用性最强
-2. 截获请求，获取返回的`json`数据
-    1. 类似于`requests`，需要做响应解析
-    2. 灵活性差点，网站改版后，需要重新做适配
-
-此项目采用的是模拟点击浏览器来发送请求，使用截获响应并解析的方法来获取数据。
-
-后期会根据不同的网站改版情况，使用更适合的方法。
-
-## MCP支持
-
-确保可以在控制台中执行`python -m mcp_query_table -h`。如果不能，可能要先`pip install mcp_query_table`
-
-在`Cline`中可以配置如下。其中`command`是`python`的绝对路径，`executable_path`是`Chrome`的绝对路径，`timeout`是超时时间，单位为秒。
-在各`AI`平台中由于返回时间常需1分钟以上，所以需要设置大的超时时间。
-
-`endpoint`支持两方式，一种是`cdp_endpoint`方式，一种是`ws_endpoint`方式。
-
-- cdp方式：通过启动时加参数`--remote-debugging-port=9222`来启动浏览器
-- ws方式：服务器上`docker run -p 3000:3000 --rm --init -it --workdir /home/pwuser --user pwuser mcr.microsoft.com/playwright:v1.51.0-noble /bin/sh -c "npx -y playwright@1.51.0 run-server --port 3000 --host 0.0.0.0"`
-
-参考：https://playwright.dev/python/docs/docker#remote-connection
-
-### STDIO方式
-
-```json
-{
-  "mcpServers": {
-    "mcp_query_table": {
-      "timeout": 300,
-      "command": "D:\\Users\\Kan\\miniconda3\\envs\\py312\\python.exe",
-      "args": [
-        "-m",
-        "mcp_query_table",
-        "--format",
-        "markdown",
-        "--endpoint",
-        "http://127.0.0.1:9222",
-        "--executable_path",
-        "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe"
-      ]
-    }
-  }
-}
-```
-
-### SSE方式
-
-先在控制台中执行如下命令，启动`MCP`服务
-
-```commandline
-python -m mcp_query_table --format markdown --transport sse --port 8000
-```
-
-然后就可以连接到`MCP`服务了
-
-```json
-{
-  "mcpServers": {
-    "mcp_query_table": {
-      "timeout": 300,
-      "url": "http://127.0.0.1:8000/sse"
-    }
-  }
-}
-```
-
-## 使用`MCP Inspector`进行调试
-
-```commandline
-npx @modelcontextprotocol/inspector python -m mcp_query_table --format markdown
-```
-
-打开浏览器并翻页是一个比较耗时的操作，会导致`MCP Inspector`页面超时，可以`http://localhost:5173/?timeout=300000`
-表示超时时间为300秒
-
-第一次尝试编写`MCP`项目，可能会有各种问题，欢迎大家交流。
-
-## `MCP`使用技巧
-
-1. 2024年涨幅最大的100只股票按2024年12月31日总市值排名。三个网站的结果都不一样
-    - 同花顺：显示了2201只股票。前5个是工商银行、农业银行、中国移动、中国石油、建设银行
-    - 通达信：显示了100只股票，前5个是寒武纪、正丹股份，汇金科技、万丰奥威、艾融软件
-    - 东方财富：显示了100只股票，前5个是海光信息、寒武纪、光启技术、润泽科技、新易盛
-
-2. 大语言模型对问题拆分能力弱，所以要能合理的提问，保证查询条件不会被改动。以下推荐第2、3种
-    - 2024年涨幅最大的100只股票按2024年12月31日总市值排名
-      > 大语言模型非常有可能拆分这句，导致一步查询被分成了多步查询
-    - 向东方财富查询“2024年涨幅最大的100只股票按2024年12月31日总市值排名”
-      > 用引号括起来，避免被拆分
-    - 向东方财富板块查询 “去年涨的最差的行业板块”，再查询此板块中去年涨的最好的5只股票
-      > 分成两步查询，先查询板块，再查询股票。但最好不要全自动，因为第一步的结果它不理解“今日涨幅”和“区间涨幅”,需要交互修正
-
-## 支持`Streamlit`
-
-实现在同一页面中查询金融数据，并手工输入到`AI`中进行深度分析。参考`streamlit`目录下的`README.md`文件。
-
-![streamlit](docs/img/streamlit.png)
-
-## 参考
-
-- [Playwright](https://playwright.dev/python/docs/intro)
-- [Selenium webdriver无法附加到edge实例，edge的--remote-debugging-port选项无效](https://blog.csdn.net/qq_30576521/article/details/142370538)

mcp_query_table-0.3.6/mcp_query_table.egg-info/SOURCES.txt

@@ -1,22 +0,0 @@
-LICENSE
-README.md
-pyproject.toml
-mcp_query_table/__init__.py
-mcp_query_table/__main__.py
-mcp_query_table/_version.py
-mcp_query_table/enums.py
-mcp_query_table/server.py
-mcp_query_table/tool.py
-mcp_query_table.egg-info/PKG-INFO
-mcp_query_table.egg-info/SOURCES.txt
-mcp_query_table.egg-info/dependency_links.txt
-mcp_query_table.egg-info/requires.txt
-mcp_query_table.egg-info/top_level.txt
-mcp_query_table/providers/__init__.py
-mcp_query_table/providers/baidu.py
-mcp_query_table/providers/n.py
-mcp_query_table/providers/yuanbao.py
-mcp_query_table/sites/__init__.py
-mcp_query_table/sites/eastmoney.py
-mcp_query_table/sites/iwencai.py
-mcp_query_table/sites/tdx.py

mcp_query_table-0.3.6/mcp_query_table.egg-info/dependency_links.txt

@@ -1 +0,0 @@
-

mcp_query_table-0.3.6/mcp_query_table.egg-info/requires.txt

@@ -1,4 +0,0 @@
-pandas
-loguru
-playwright
-mcp

mcp_query_table-0.3.6/mcp_query_table.egg-info/top_level.txt

@@ -1 +0,0 @@
-mcp_query_table

mcp_query_table-0.3.6/setup.cfg

@@ -1,4 +0,0 @@
-[egg_info]
-tag_build = 
-tag_date = 0
-