funboost 45.3__tar.gz → 45.6__tar.gz

{funboost-45.3/funboost.egg-info → funboost-45.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: funboost
-Version: 45.3
+Version: 45.6
 Summary: pip install funboost，python全功能分布式函数调度框架,。支持python所有类型的并发模式和一切知名消息队列中间件，支持如 celery dramatiq等框架整体作为funboost中间件，python函数加速器，框架包罗万象，用户能想到的控制功能全都有。一统编程思维，兼容50% python业务场景，适用范围广。只需要一行代码即可分布式执行python一切函数，99%用过funboost的pythoner 感受是　简易 方便 强劲 强大，相见恨晚 
 Home-page: https://github.com/ydf0509/funboost
 Author: bfzs
@@ -47,16 +47,25 @@ License-File: LICENSE
 
 
 <pre style="color: greenyellow;background-color: #0c1119; font-size: medium;">
-pip install funboost ,python全功能分布式函数调度框架,。  用法例子见文档1.3
-支持python所有类型的并发模式和全球一切知名消息队列中间件，
+pip install funboost ,python全功能分布式函数调度框架。  demo用法例子见文档1.3
+
+只需要一行@boost代码即可分布式执行python一切任意函数，99%用过funboost的pythoner 感受是 方便 快速 强大。
+支持python所有类型的并发模式,消息队列方面支持全球一切知名消息队列中间件和模拟的实现消息队列，
 同时funboost支持celery整个框架作为核心来发布和消费消息，使用funboost的极简api方式来自动化配置和利用celery调度,
 也支持huey dramatiq rq等任务队列框架作为funboost的broker。 
+
 python函数加速器，框架包罗万象，一统编程思维，兼容50% python编程业务场景，适用范围广。
-只需要一行代码即可分布式执行python一切函数，99%用过funboost的pythoner 感受是 方便 快速 强大。
 python万能分布式函数调度框架，支持5种并发模式，30+种消息队列中间件(或任务队列框架)，
 30种任务控制功能。给任意python函数赋能。
 用途概念就是常规经典的 生产者 + 消息队列中间件 + 消费者 编程思想。
+
 框架只需要学习@boost这一个装饰器的入参就可以，所有用法几乎和1.3例子一摸一样，非常简化简单。
+框架对代码没有入侵,可以加到任意已有项目而对项目python文件目录结构0要求,
+不像 celery django scrapy 这样的框架,要从一开始就开始规划好项目目录结构,如果不想用框架了,
+或者想改变使用其他框架框架,那么已经所写的代码组织形式就几乎成了废物,需要大改特改. 
+但是funboost完全不会这样,加上或去掉@boost装饰器,对你的项目影响为0,用户照常使用,
+所以用户可以对任意项目,任意时候,引入使用funboost或者去掉使用funboost,代码组织形式不需要发生变化.
+
 </pre>
 
 ### 框架评价
@@ -73,6 +82,7 @@ python万能分布式函数调度框架，支持5种并发模式，30+种消息
 ，连celery命令行运行起来都要反复猜测尝试。
 正因为如此用户从心理已近十分惧怕学习一种叫python框架的东西了，用户顶多愿意学习一个python包或者模块，
 学习一个框架会非常害怕觉得难度高且耗时，所以非常反感尝试新的框架。
+用过的99%都说funboost比celery简单方便太多,看都不看的人第一秒就是开始质疑重复造轮子.
 
 funboost只有一个@boost装饰器，@boost入参能自动补全，更重要的是被@boost装饰的函数，
 有哪些方法，每个方法入参是什么都能自动补全。funboost的中间件配置文件自当生成在用户当前项目根目录，
@@ -211,10 +221,10 @@ if __name__ == '__main__':
 2)funboost 使用内存队列,设置10线程并发
 ```python
 import time
-from funboost import boost, BrokerEnum
+from funboost import BoosterParams, BrokerEnum
 
 
-@boost("test_insteda_thread_queue", broker_kind=BrokerEnum.MEMORY_QUEUE, concurrent_num=10)
+@BoosterParams(queue_name="test_insteda_thread_queue", broker_kind=BrokerEnum.MEMORY_QUEUE, concurrent_num=10, auto_start_consuming_message=True)
 def f(x):
     time.sleep(3)
     print(x)
@@ -223,7 +233,8 @@ def f(x):
 if __name__ == '__main__':
     for i in range(100):
         f.push(i)
-    f.consume()
+
+
 ```
 
 
@@ -478,7 +489,7 @@ python比其他语言更需要分布式函数调度框架来执行函数，有
 funboost通过支持celery作为broker_kind,使celer框架变成了funboost的一个子集
 ```
 
-[查看分布式函数调度框架完整文档](https://funboost.readthedocs.io/zh-cn/latest/index.html)
+[查看分布式函数调度框架完整文档](https://funboost.readthedocs.io/)
 
 
 

{funboost-45.3 → funboost-45.6}/README.md

@@ -15,16 +15,25 @@
 
 
 <pre style="color: greenyellow;background-color: #0c1119; font-size: medium;">
-pip install funboost ,python全功能分布式函数调度框架,。  用法例子见文档1.3
-支持python所有类型的并发模式和全球一切知名消息队列中间件，
+pip install funboost ,python全功能分布式函数调度框架。  demo用法例子见文档1.3
+
+只需要一行@boost代码即可分布式执行python一切任意函数，99%用过funboost的pythoner 感受是 方便 快速 强大。
+支持python所有类型的并发模式,消息队列方面支持全球一切知名消息队列中间件和模拟的实现消息队列，
 同时funboost支持celery整个框架作为核心来发布和消费消息，使用funboost的极简api方式来自动化配置和利用celery调度,
 也支持huey dramatiq rq等任务队列框架作为funboost的broker。 
+
 python函数加速器，框架包罗万象，一统编程思维，兼容50% python编程业务场景，适用范围广。
-只需要一行代码即可分布式执行python一切函数，99%用过funboost的pythoner 感受是 方便 快速 强大。
 python万能分布式函数调度框架，支持5种并发模式，30+种消息队列中间件(或任务队列框架)，
 30种任务控制功能。给任意python函数赋能。
 用途概念就是常规经典的 生产者 + 消息队列中间件 + 消费者 编程思想。
+
 框架只需要学习@boost这一个装饰器的入参就可以，所有用法几乎和1.3例子一摸一样，非常简化简单。
+框架对代码没有入侵,可以加到任意已有项目而对项目python文件目录结构0要求,
+不像 celery django scrapy 这样的框架,要从一开始就开始规划好项目目录结构,如果不想用框架了,
+或者想改变使用其他框架框架,那么已经所写的代码组织形式就几乎成了废物,需要大改特改. 
+但是funboost完全不会这样,加上或去掉@boost装饰器,对你的项目影响为0,用户照常使用,
+所以用户可以对任意项目,任意时候,引入使用funboost或者去掉使用funboost,代码组织形式不需要发生变化.
+
 </pre>
 
 ### 框架评价
@@ -41,6 +50,7 @@ python万能分布式函数调度框架，支持5种并发模式，30+种消息
 ，连celery命令行运行起来都要反复猜测尝试。
 正因为如此用户从心理已近十分惧怕学习一种叫python框架的东西了，用户顶多愿意学习一个python包或者模块，
 学习一个框架会非常害怕觉得难度高且耗时，所以非常反感尝试新的框架。
+用过的99%都说funboost比celery简单方便太多,看都不看的人第一秒就是开始质疑重复造轮子.
 
 funboost只有一个@boost装饰器，@boost入参能自动补全，更重要的是被@boost装饰的函数，
 有哪些方法，每个方法入参是什么都能自动补全。funboost的中间件配置文件自当生成在用户当前项目根目录，
@@ -179,10 +189,10 @@ if __name__ == '__main__':
 2)funboost 使用内存队列,设置10线程并发
 ```python
 import time
-from funboost import boost, BrokerEnum
+from funboost import BoosterParams, BrokerEnum
 
 
-@boost("test_insteda_thread_queue", broker_kind=BrokerEnum.MEMORY_QUEUE, concurrent_num=10)
+@BoosterParams(queue_name="test_insteda_thread_queue", broker_kind=BrokerEnum.MEMORY_QUEUE, concurrent_num=10, auto_start_consuming_message=True)
 def f(x):
     time.sleep(3)
     print(x)
@@ -191,7 +201,8 @@ def f(x):
 if __name__ == '__main__':
     for i in range(100):
         f.push(i)
-    f.consume()
+
+
 ```
 
 
@@ -446,7 +457,7 @@ python比其他语言更需要分布式函数调度框架来执行函数，有
 funboost通过支持celery作为broker_kind,使celer框架变成了funboost的一个子集
 ```
 
-[查看分布式函数调度框架完整文档](https://funboost.readthedocs.io/zh-cn/latest/index.html)
+[查看分布式函数调度框架完整文档](https://funboost.readthedocs.io/)
 
 
 

{funboost-45.3 → funboost-45.6}/funboost/__init__.py

@@ -13,7 +13,7 @@ set_frame_config这个模块的 use_config_form_funboost_config_module() 是核
 这段注释说明和使用的用户无关,只和框架开发人员有关.
 '''
 
-__version__ = "45.3"
+__version__ = "45.6"
 
 from funboost.set_frame_config import show_frame_config
 
@@ -65,5 +65,5 @@ from funboost.core.current_task import funboost_current_task
 # set_interrupt_signal_handler()
 
 # 有的包默认没加handlers，原始的日志不漂亮且不可跳转不知道哪里发生的。这里把warnning级别以上的日志默认加上handlers。
-# nb_log.get_logger(name='', log_level_int=30, _log_filename='pywarning.log')
+# nb_log.get_logger(name='', log_level_int=30, log_filename='pywarning.log')
 

funboost-45.6/funboost/assist/faststream_helper.py

@@ -0,0 +1,18 @@
+import asyncio
+
+from faststream import FastStream
+from faststream.rabbit import RabbitBroker
+from funboost.funboost_config_deafult import BrokerConnConfig
+from funboost.core.loggers import get_funboost_file_logger
+
+get_funboost_file_logger('faststream.access.rabbit')
+
+broker = RabbitBroker(BrokerConnConfig.RABBITMQ_URL, max_consumers=20)
+app = FastStream(broker)
+
+# asyncio.get_event_loop().run_until_complete(broker.connect())
+#
+# asyncio.get_event_loop().run_until_complete(broker.start())
+
+def get_broker(max_consumers=None):
+    return RabbitBroker(BrokerConnConfig.RABBITMQ_URL, max_consumers=max_consumers)

funboost-45.6/funboost/assist/taskiq_helper.py

@@ -0,0 +1 @@
+

{funboost-45.3 → funboost-45.6}/funboost/consumers/base_consumer.py

@@ -35,9 +35,10 @@ from funboost.core.current_task import funboost_current_task, FctContext
 from funboost.core.loggers import develop_logger
 
 from funboost.core.func_params_model import BoosterParams, PublisherParams, BaseJsonAbleModel
+from funboost.core.serialization import Serialization
 from funboost.core.task_id_logger import TaskIdLogger
 from funboost.constant import FunctionKind
-from funboost.utils.json_helper import JsonUtils
+
 from nb_libs.path_helper import PathHelper
 from nb_log import (get_logger, LoggerLevelSetterMixin, LogManager, is_main_process,
                     nb_log_config_default)
@@ -139,7 +140,7 @@ class AbstractConsumer(LoggerLevelSetterMixin, metaclass=abc.ABCMeta, ):
         if consumer_params.consuming_function is None:
             raise ValueError('必须传 consuming_function 参数')
 
-        self._msg_schedule_time_intercal = 0 if consumer_params.qps is None else 1.0 / consumer_params.qps
+        self._msg_schedule_time_intercal = 0 if consumer_params.qps in (None, 0) else 1.0 / consumer_params.qps
 
         self._concurrent_mode_dispatcher = ConcurrentModeDispatcher(self)
         if consumer_params.concurrent_mode == ConcurrentModeEnum.ASYNC:
@@ -232,6 +233,10 @@ class AbstractConsumer(LoggerLevelSetterMixin, metaclass=abc.ABCMeta, ):
             self.logger.info(f'{self.queue_name} consumer 的消费者配置:\n {self.consumer_params.json_str_value()}')
         atexit.register(self.join_shedual_task_thread)
 
+        if self.consumer_params.auto_start_consuming_message:
+            self.publisher_of_same_queue
+            self.start_consuming_message()
+
     def _build_logger(self):
         logger_prefix = self.consumer_params.logger_prefix
         if logger_prefix != '':
@@ -408,8 +413,7 @@ class AbstractConsumer(LoggerLevelSetterMixin, metaclass=abc.ABCMeta, ):
         extra_params = {'task_id': task_id, 'publish_time': round(time.time(), 4),
                         'publish_time_format': time.strftime('%Y-%m-%d %H:%M:%S')}
         """
-        if isinstance(msg, str):
-            msg = json.loads(msg)
+        msg = Serialization.to_dict(msg)
         # 以下是清洗补全字段.
         if 'extra' not in msg:
             msg['extra'] = {'is_auto_fill_extra': True}
@@ -536,7 +540,7 @@ class AbstractConsumer(LoggerLevelSetterMixin, metaclass=abc.ABCMeta, ):
         # print(999)
         if self.consumer_params.is_show_message_get_from_broker:
             # self.logger.debug(f'从 {broker_name} 中间件 的 {self._queue_name} 中取出的消息是 {msg}')
-            self.logger.debug(f'从 {broker_name or self.consumer_params.broker_kind} 中间件 的 {self._queue_name} 中取出的消息是 {JsonUtils.to_json_str(msg)}')
+            self.logger.debug(f'从 {broker_name or self.consumer_params.broker_kind} 中间件 的 {self._queue_name} 中取出的消息是 {Serialization.to_json_str(msg)}')
 
     def _get_priority_conf(self, kw: dict, broker_task_config_key: str):
         broker_task_config = kw['body'].get('extra', {}).get(broker_task_config_key, None)
@@ -585,10 +589,12 @@ class AbstractConsumer(LoggerLevelSetterMixin, metaclass=abc.ABCMeta, ):
             # method_cls = getattr(sys.modules[self.consumer_params.consuming_function_class_module],
             #                      self.consumer_params.consuming_function_class_name)
             if self.publisher_params.consuming_function_kind == FunctionKind.CLASS_METHOD:
-                method_cls = getattr(PathHelper(method_first_param_value[ConstStrForClassMethod.CLS_FILE]).import_as_module(),method_first_param_value[ConstStrForClassMethod.CLS_NAME])
+                method_cls = getattr(PathHelper.import_module(method_first_param_value[ConstStrForClassMethod.CLS_MODULE]),
+                                     method_first_param_value[ConstStrForClassMethod.CLS_NAME])
                 real_function_only_params[method_first_param_name] = method_cls
             elif self.publisher_params.consuming_function_kind == FunctionKind.INSTANCE_METHOD:
-                method_cls = getattr(PathHelper(method_first_param_value[ConstStrForClassMethod.CLS_FILE]).import_as_module(), method_first_param_value[ConstStrForClassMethod.CLS_NAME])
+                method_cls = getattr(PathHelper.import_module(method_first_param_value[ConstStrForClassMethod.CLS_MODULE]),
+                                     method_first_param_value[ConstStrForClassMethod.CLS_NAME])
                 obj = method_cls(**method_first_param_value[ConstStrForClassMethod.OBJ_INIT_PARAMS])
                 real_function_only_params[method_first_param_name] = obj
             # print(real_function_only_params)
@@ -642,7 +648,7 @@ class AbstractConsumer(LoggerLevelSetterMixin, metaclass=abc.ABCMeta, ):
                         # RedisMixin().redis_db_frame.expire(kw['body']['extra']['task_id'], 600)
                         current_function_result_status.rpc_result_expire_seconds = self.consumer_params.rpc_result_expire_seconds
                         p.lpush(kw['body']['extra']['task_id'],
-                                json.dumps(current_function_result_status.get_status_dict(without_datetime_obj=True)))
+                                Serialization.to_json_str(current_function_result_status.get_status_dict(without_datetime_obj=True)))
                         p.expire(kw['body']['extra']['task_id'], self.consumer_params.rpc_result_expire_seconds)
                         p.execute()
 
@@ -812,7 +818,7 @@ class AbstractConsumer(LoggerLevelSetterMixin, metaclass=abc.ABCMeta, ):
                     with RedisMixin().redis_db_filter_and_rpc_result.pipeline() as p:
                         current_function_result_status.rpc_result_expire_seconds = self.consumer_params.rpc_result_expire_seconds
                         p.lpush(kw['body']['extra']['task_id'],
-                                json.dumps(current_function_result_status.get_status_dict(without_datetime_obj=True)))
+                                Serialization.to_json_str(current_function_result_status.get_status_dict(without_datetime_obj=True)))
                         p.expire(kw['body']['extra']['task_id'], self.consumer_params.rpc_result_expire_seconds)
                         p.execute()
 
@@ -852,7 +858,6 @@ class AbstractConsumer(LoggerLevelSetterMixin, metaclass=abc.ABCMeta, ):
         fct = funboost_current_task()
         fct.set_fct_context(None)
 
-
     # noinspection PyProtectedMember
     async def _async_run_consuming_function_with_confirm_and_retry(self, kw: dict, current_retry_times,
                                                                    function_result_status: FunctionResultStatus, ):
@@ -1161,14 +1166,14 @@ class DistributedConsumerStatistics(RedisMixin, FunboostFileLoggerMixin):
         results = self.redis_db_frame.smembers(redis_key)
         with self.redis_db_frame.pipeline() as p:
             for result in results:
-                result_dict = json.loads(result)
+                result_dict = Serialization.to_dict(result)
                 if self.timestamp() - result_dict['hearbeat_timestamp'] > 15 \
                         or self._consumer_identification_map['consumer_uuid'] == result_dict['consumer_uuid']:
                     # 因为这个是10秒钟运行一次，15秒还没更新，那肯定是掉线了。如果消费者本身是自己也先删除。
                     p.srem(redis_key, result)
             self._consumer_identification_map['hearbeat_datetime_str'] = time_util.DatetimeConverter().datetime_str
             self._consumer_identification_map['hearbeat_timestamp'] = self.timestamp()
-            value = json.dumps(self._consumer_identification_map, sort_keys=True)
+            value = Serialization.to_json_str(self._consumer_identification_map, )
             p.sadd(redis_key, value)
             p.execute()
 

{funboost-45.3 → funboost-45.6}/funboost/consumers/confirm_mixin.py

@@ -5,7 +5,7 @@ import json
 import time
 from funboost.utils.redis_manager import RedisMixin
 from funboost.utils import decorators
-
+from funboost.core.serialization import Serialization
 """
 此模块是依赖redis的确认消费，所以比较复杂。
 """
@@ -46,7 +46,7 @@ class ConsumerConfirmMixinWithTheHelpOfRedis(RedisMixin):
                 time_max = time.time() - self.UNCONFIRMED_TIMEOUT
                 for value in self.redis_db_frame.zrangebyscore(self._unack_zset_name, 0, time_max):
                     self.logger.warning(f'向 {self._queue_name} 重新放入未消费确认的任务 {value}')
-                    self._requeue({'body': json.loads(value)})
+                    self._requeue({'body': Serialization.to_dict(value)})
                     self.redis_db_frame.zrem(self._unack_zset_name, value)
                 self.logger.info(f'{self._unack_zset_name} 中有待确认消费任务的数量是'
                                  f' {self.redis_db_frame.zcard(self._unack_zset_name)}')

funboost-45.6/funboost/consumers/faststream_consumer.py

@@ -0,0 +1,54 @@
+import asyncio
+import json
+import threading
+import time
+
+from funboost import EmptyConsumer
+from funboost.assist.faststream_helper import broker,app,get_broker
+from faststream import FastStream,Context
+from faststream.annotations import Logger
+
+from funboost.concurrent_pool.async_helper import simple_run_in_executor
+from funboost.core.helper_funs import delete_keys_and_return_new_dict
+
+
+class FastStreamConsumer(EmptyConsumer):
+    def custom_init(self):
+        self.broker = get_broker(max_consumers=self.consumer_params.concurrent_num)
+        subc = self.broker.subscriber(self.queue_name)
+        # @broker.subscriber(self.queue_name)
+        async def f(msg:str, logger: Logger,message=Context(),broker=Context(),context=Context(),):
+            self.logger.debug(f' 这条消息是 faststream 从 {self.queue_name} 队列中取出 ,是由 faststream 框架调度 {self.consuming_function.__name__} 函数处理,msg:{message} {context}')
+            # print(logger.name)
+            # return self.consuming_function(*args, **kwargs) # 如果没有声明 autoretry_for ，那么消费函数出错了就不会自动重试了。
+            # print(msg)
+            function_only_params = delete_keys_and_return_new_dict(json.loads(msg))
+            if self._consuming_function_is_asyncio:
+                result = await self.consuming_function(**function_only_params)
+            else:
+                result = await simple_run_in_executor(self.consuming_function,**function_only_params)
+            # print(result)
+            return result
+        subc(f)
+        self.faststream_subscriber = subc
+
+    def _shedual_task(self):
+        """ 完全由faststream框架接管控制消费，不使用funboost的AbstractConsumer的_run"""
+        while 1:
+            time.sleep(100)
+
+
+    def _confirm_consume(self, kw):
+        pass
+
+    def start_consuming_message(self):
+        def _f():
+            loop = asyncio.new_event_loop()
+            loop.run_until_complete(self.broker.connect())
+            loop.run_until_complete(self.faststream_subscriber.start())
+            loop.run_forever()
+        self.keep_circulating(10, block=False)(_f)()
+        # threading.Thread(target=_f).start()
+
+    def _requeue(self, kw):
+        pass

{funboost-45.3 → funboost-45.6}/funboost/consumers/local_python_queue_consumer.py

@@ -5,7 +5,7 @@ import json
 from queue import Queue,SimpleQueue
 from funboost.constant import BrokerEnum
 from funboost.consumers.base_consumer import AbstractConsumer
-from funboost.publishers import local_python_queue_publisher
+from funboost.queues.memory_queues_map import PythonQueues
 
 
 class LocalPythonQueueConsumer(AbstractConsumer):
@@ -15,7 +15,7 @@ class LocalPythonQueueConsumer(AbstractConsumer):
 
     @property
     def local_python_queue(self) -> Queue:
-        return local_python_queue_publisher.local_pyhton_queue_name__local_pyhton_queue_obj_map[self._queue_name]
+        return PythonQueues.get_queue(self._queue_name)
 
     def _shedual_task(self):
         while True:

{funboost-45.3 → funboost-45.6}/funboost/consumers/redis_consumer.py

@@ -10,7 +10,7 @@ import time
 from funboost.constant import BrokerEnum
 from funboost.consumers.base_consumer import AbstractConsumer
 from funboost.utils.redis_manager import RedisMixin
-
+from funboost.core.serialization import Serialization
 
 class RedisConsumer(AbstractConsumer, RedisMixin):
     """
@@ -57,4 +57,4 @@ class RedisConsumer(AbstractConsumer, RedisMixin):
         pass  # redis没有确认消费的功能。
 
     def _requeue(self, kw):
-        self.redis_db_frame.rpush(self._queue_name, json.dumps(kw['body']))
+        self.redis_db_frame.rpush(self._queue_name,Serialization.to_json_str(kw['body']))

{funboost-45.3 → funboost-45.6}/funboost/consumers/redis_consumer_ack_using_timeout.py

@@ -4,8 +4,8 @@
 import json
 import time
 from funboost.consumers.base_consumer import AbstractConsumer
+from funboost.core.serialization import Serialization
 from funboost.utils.decorators import RedisDistributedLockContextManager
-from funboost.utils.json_helper import JsonUtils
 from funboost.utils.redis_manager import RedisMixin
 
 
@@ -40,7 +40,7 @@ class RedisConsumerAckUsingTimeout(AbstractConsumer, RedisMixin):
         self.redis_db_frame.zrem(self._unack_zset_name, kw['task_str'])
 
     def _requeue(self, kw):
-        self.redis_db_frame.rpush(self._queue_name, JsonUtils.to_json_str(kw['body']))
+        self.redis_db_frame.rpush(self._queue_name, Serialization.to_json_str(kw['body']))
 
     def _shedual_task(self):
         lua = '''

{funboost-45.3 → funboost-45.6}/funboost/consumers/redis_consumer_simple.py

@@ -4,6 +4,7 @@
 import json
 from funboost.constant import BrokerEnum
 from funboost.consumers.base_consumer import AbstractConsumer
+from funboost.core.serialization import Serialization
 from funboost.utils.redis_manager import RedisMixin
 
 
@@ -24,6 +25,6 @@ class RedisConsumer(AbstractConsumer, RedisMixin):
         pass  # redis没有确认消费的功能。
 
     def _requeue(self, kw):
-        self.redis_db_frame.rpush(self._queue_name, json.dumps(kw['body']))
+        self.redis_db_frame.rpush(self._queue_name, Serialization.to_json_str(kw['body']))
 
 

{funboost-45.3 → funboost-45.6}/funboost/consumers/redis_filter.py

@@ -11,6 +11,7 @@ import time
 from collections import OrderedDict
 import typing
 
+from funboost.core.serialization import Serialization
 from funboost.utils import  decorators
 from funboost.core.loggers import FunboostFileLoggerMixin
 
@@ -34,8 +35,7 @@ class RedisFilter(RedisMixin, FunboostFileLoggerMixin):
     @staticmethod
     def _get_ordered_str(value):
         """对json的键值对在redis中进行过滤，需要先把键值对排序，否则过滤会不准确如 {"a":1,"b":2} 和 {"b":2,"a":1}"""
-        if isinstance(value, str):
-            value = json.loads(value)
+        value = Serialization.to_dict(value)
         ordered_dict = OrderedDict()
         for k in sorted(value):
             ordered_dict[k] = value[k]

{funboost-45.3 → funboost-45.6}/funboost/core/current_task.py

@@ -163,7 +163,7 @@ def get_current_taskid():
     # return fct.function_result_status.task_id
     try:
         fct = funboost_current_task()
-        return fct.task_id  # 不在funboost的消费函数里面就获取不到上下文了
+        return fct.task_id  # 不在funboost的消费函数里面或者同个线程、协程就获取不到上下文了
     except (AttributeError, LookupError) as e:
         # print(e,type(e))
         return 'no_task_id'

{funboost-45.3 → funboost-45.6}/funboost/core/func_params_model.py

@@ -177,6 +177,7 @@ class BoosterParams(BaseJsonAbleModel):
     do_not_run_by_specify_time: tuple = ('10:00:00', '22:00:00')  # 不运行的时间段,在这个时间段自动不运行函数.
 
     schedule_tasks_on_main_thread: bool = False  # 直接在主线程调度任务，意味着不能直接在当前主线程同时开启两个消费者。
+    auto_start_consuming_message: bool = False  # 是否在定义后就自动启动消费，无需用户手动写 .consume() 来启动消息消费。
 
     consuming_function: typing.Callable = None  # 消费函数,在@boost时候不用指定,因为装饰器知道下面的函数.
     consuming_function_raw: typing.Callable = None
@@ -195,8 +196,7 @@ class BoosterParams(BaseJsonAbleModel):
 
     auto_generate_info: dict = {}  # 自动生成的信息,不需要用户主动传参.
 
-    consuming_function_kind :typing.Optional[str]= None  #自动生成的信息,不需要用户主动传参.
-
+    consuming_function_kind: typing.Optional[str] = None  # 自动生成的信息,不需要用户主动传参,如果自动判断失误就传递。
 
     @root_validator(skip_on_failure=True)
     def check_values(cls, values: dict):
@@ -269,7 +269,6 @@ class PriorityConsumingControlConfig(BaseJsonAbleModel):
     misfire_grace_time: typing.Union[int, None] = None
     other_extra_params: dict = None  # 其他参数, 例如消息优先级 , priority_control_config=PriorityConsumingControlConfig(other_extra_params={'priroty': priorityxx})，
 
-
     @root_validator(skip_on_failure=True)
     def cehck_values(cls, values: dict):
         if values['countdown'] and values['eta']:
@@ -297,7 +296,6 @@ class PublisherParams(BaseJsonAbleModel):
     consuming_function_kind: typing.Optional[str] = None  # 自动生成的信息,不需要用户主动传参.
 
 
-
 if __name__ == '__main__':
     from funboost.concurrent_pool import FlexibleThreadPool
 

{funboost-45.3 → funboost-45.6}/funboost/core/function_result_status_saver.py

@@ -15,6 +15,7 @@ from pymongo import IndexModel, ReplaceOne
 
 from funboost.core.func_params_model import FunctionResultStatusPersistanceConfig
 from funboost.core.helper_funs import get_publish_time, delete_keys_and_return_new_dict
+from funboost.core.serialization import Serialization
 from funboost.utils import time_util, decorators
 from funboost.utils.mongo_util import MongoMixin
 # from nb_log import LoggerMixin
@@ -46,7 +47,7 @@ class FunctionResultStatus():
             self.publish_time_str = time_util.DatetimeConverter(publish_time).datetime_str
         function_params = delete_keys_and_return_new_dict(msg_dict, )
         self.params = function_params
-        self.params_str = json.dumps(function_params, ensure_ascii=False)
+        self.params_str = Serialization.to_json_str(function_params)
         self.result = None
         self.run_times = 0
         self.exception = None
@@ -78,7 +79,8 @@ class FunctionResultStatus():
         # item.pop('time_start')
         datetime_str = time_util.DatetimeConverter().datetime_str
         try:
-            json.dumps(item['result'])  # 不希望存不可json序列化的复杂类型。麻烦。存这种类型的结果是伪需求。
+            Serialization.to_json_str(item['result'])
+            # json.dumps(item['result'])  # 不希望存不可json序列化的复杂类型。麻烦。存这种类型的结果是伪需求。
         except TypeError:
             item['result'] = str(item['result'])[:1000]
         item.update({'insert_time_str': datetime_str,
@@ -98,7 +100,7 @@ class FunctionResultStatus():
         return item
 
     def __str__(self):
-        return f'''{self.__class__}   {json.dumps(self.get_status_dict(), ensure_ascii=False)}'''
+        return f'''{self.__class__}   {Serialization.to_json_str(self.get_status_dict())}'''
 
 
 class ResultPersistenceHelper(MongoMixin, FunboostFileLoggerMixin):

{funboost-45.3 → funboost-45.6}/funboost/core/msg_result_getter.py

@@ -10,7 +10,7 @@ from funboost.concurrent_pool import CustomThreadPoolExecutor
 from funboost.concurrent_pool.flexible_thread_pool import FlexibleThreadPoolMinWorkers0
 from funboost.utils.redis_manager import RedisMixin
 from funboost.utils.redis_manager import AioRedisMixin
-
+from funboost.core.serialization import Serialization
 
 class HasNotAsyncResult(Exception):
     pass
@@ -42,7 +42,7 @@ class AsyncResult(RedisMixin):
             self._has_pop = True
             if redis_value is not None:
                 status_and_result_str = redis_value[1]
-                self._status_and_result = json.loads(status_and_result_str)
+                self._status_and_result = Serialization.to_dict(status_and_result_str)
                 self.redis_db_filter_and_rpc_result.lpush(self.task_id, status_and_result_str)
                 self.redis_db_filter_and_rpc_result.expire(self.task_id, self._status_and_result['rpc_result_expire_seconds'])
                 return self._status_and_result
@@ -147,7 +147,7 @@ if __name__ == '__main__':
             self._has_pop = True
             if redis_value is not None:
                 status_and_result_str = redis_value[1]
-                self._status_and_result = json.loads(status_and_result_str)
+                self._status_and_result = Serialization.to_dict(status_and_result_str)
                 await self.aioredis_db_filter_and_rpc_result.lpush(self.task_id, status_and_result_str)
                 await self.aioredis_db_filter_and_rpc_result.expire(self.task_id, self._status_and_result['rpc_result_expire_seconds'])
                 return self._status_and_result

funboost-45.6/funboost/core/serialization.py

@@ -0,0 +1,16 @@
+import typing
+
+import orjson
+class Serialization:
+    @staticmethod
+    def to_json_str(dic:typing.Union[dict,str]):
+        if isinstance(dic,str):
+            return dic
+        str1 =orjson.dumps(dic)
+        return str1.decode('utf8')
+
+    @staticmethod
+    def to_dict(strx:typing.Union[str,dict]):
+        if isinstance(strx,dict):
+            return strx
+        return orjson.loads(strx)

{funboost-45.3 → funboost-45.6}/funboost/factories/broker_kind__publsiher_consumer_type_map.py

@@ -16,9 +16,9 @@ from funboost.publishers.local_python_queue_publisher import LocalPythonQueuePub
 from funboost.publishers.mongomq_publisher import MongoMqPublisher
 
 from funboost.publishers.persist_queue_publisher import PersistQueuePublisher
-from funboost.publishers.rabbitmq_amqpstorm_publisher import RabbitmqPublisherUsingAmqpStorm
+
 from funboost.publishers.rabbitmq_pika_publisher import RabbitmqPublisher
-from funboost.publishers.rabbitmq_rabbitpy_publisher import RabbitmqPublisherUsingRabbitpy
+
 from funboost.publishers.redis_publisher import RedisPublisher
 from funboost.publishers.rocketmq_publisher import RocketmqPublisher
 from funboost.publishers.redis_stream_publisher import RedisStreamPublisher
@@ -36,9 +36,8 @@ from funboost.consumers.nats_consumer import NatsConsumer
 
 from funboost.consumers.peewee_conusmer import PeeweeConsumer
 from funboost.consumers.persist_queue_consumer import PersistQueueConsumer
-from funboost.consumers.rabbitmq_amqpstorm_consumer import RabbitmqConsumerAmqpStorm
 from funboost.consumers.rabbitmq_pika_consumer import RabbitmqConsumer
-from funboost.consumers.rabbitmq_rabbitpy_consumer import RabbitmqConsumerRabbitpy
+
 from funboost.consumers.redis_brpoplpush_consumer import RedisBrpopLpushConsumer
 from funboost.consumers.redis_consumer import RedisConsumer
 from funboost.consumers.redis_consumer_ack_able import RedisConsumerAckAble
@@ -57,8 +56,8 @@ from funboost.consumers.base_consumer import AbstractConsumer
 from funboost.constant import BrokerEnum
 
 broker_kind__publsiher_consumer_type_map = {
-    BrokerEnum.RABBITMQ_AMQPSTORM: (RabbitmqPublisherUsingAmqpStorm, RabbitmqConsumerAmqpStorm),
-    BrokerEnum.RABBITMQ_RABBITPY: (RabbitmqPublisherUsingRabbitpy, RabbitmqConsumerRabbitpy),
+
+
     BrokerEnum.REDIS: (RedisPublisher, RedisConsumer),
     BrokerEnum.MEMORY_QUEUE: (LocalPythonQueuePublisher, LocalPythonQueueConsumer),
     BrokerEnum.RABBITMQ_PIKA: (RabbitmqPublisher, RabbitmqConsumer),
@@ -111,6 +110,15 @@ def regist_to_funboost(broker_kind: str):
     这样当用户需要使用某些三方包中间件作为消息队列时候，按照import报错信息，用户自己去pip安装好。或者 pip install funboost[all] 一次性安装所有中间件。
     建议按照 https://github.com/ydf0509/funboost/blob/master/setup.py 中的 extra_brokers 和 install_requires 里面的版本号来安装三方包版本.
     """
+    if broker_kind == BrokerEnum.RABBITMQ_AMQPSTORM:
+        from funboost.publishers.rabbitmq_amqpstorm_publisher import RabbitmqPublisherUsingAmqpStorm
+        from funboost.consumers.rabbitmq_amqpstorm_consumer import RabbitmqConsumerAmqpStorm
+        register_custom_broker(BrokerEnum.RABBITMQ_AMQPSTORM, RabbitmqPublisherUsingAmqpStorm, RabbitmqConsumerAmqpStorm)
+
+    if broker_kind == BrokerEnum.RABBITMQ_RABBITPY:
+        from funboost.publishers.rabbitmq_rabbitpy_publisher import RabbitmqPublisherUsingRabbitpy
+        from funboost.consumers.rabbitmq_rabbitpy_consumer import RabbitmqConsumerRabbitpy
+        register_custom_broker(BrokerEnum.RABBITMQ_RABBITPY, RabbitmqPublisherUsingRabbitpy, RabbitmqConsumerRabbitpy)
 
     if broker_kind == BrokerEnum.PULSAR:
         from funboost.consumers.pulsar_consumer import PulsarConsumer