buildz 0.6.19__tar.gz → 0.6.20__tar.gz

{buildz-0.6.19/buildz.egg-info → buildz-0.6.20}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: buildz
-Version: 0.6.19
+Version: 0.6.20
 Summary: 配置读写（基于json格式进行简化）、ioc、以及其他工具代码。a json-base file format's read and write code by python, and codes to read and product object from configure file in such format(ioc), and other tool codes
 Home-page: https://github.com/buildCodeZ/buildz
 Author: Zzz
@@ -22,6 +22,9 @@ License-File: LICENSE
     buildz.tz: 加些工具，目前只有myerse diff字符串比较算法
     buildz.demo: 使用参考，运行"python -m buildz"会用这个模块
     buildz.db: sql集成工具，自用，里面import了其他sql库，使用运行"python -m buildz.db 配置文件路径"
+    buildz.base: 封装了一个基础类，继承它可以少写一些代码
+    buildz.html: xml（html）内容读取和解析
+    buildz.auto: 自动化操作（主要是做自动化测试方便些，如果不怕写一堆配置文件的话）
 代码关系:
     buildz.xf, buildz.pyz, buildz.argx, buildz.fz, buildz.tz都是独立的模块
     buildz.ioc需要buildz.xf和buildz.pyz
@@ -31,6 +34,8 @@ License-File: LICENSE
 运行python -m buildz查看帮助
 
 持续更新中。。。
+2024/10/08:
+增强auto配置功能，在auto增加数据库使用的封装，后续考虑出文档。。。如果有时间
 2024/09/25:
 修复bug，增强html的搜索功能，增强auto配置功能
 auto里加了个request的demo，里面用的requests库，可以直接配置来进行http调用
@@ -43,7 +48,7 @@ ioc:
     加修饰器
     加配置refs
 
-PS: 对比了下json.loads（修改了下json的scanner.py，让它在纯python下运行，不然json.loads会更快）和目前的xf.loads(buildz.xf.readz.loads)的速度，xf.loads比json.loads慢7倍，可能是读字符串更频繁，方法调用更多（为了代码更结构化和容易修改），其实有一版更慢(buildz.xf.read.loads，废弃代码，后面看情况删掉)，慢100倍，因为只考虑结构化，没考虑列表增减开销
+PS: 对比了下json.loads（修改了下json的scanner.py，让它在纯python下运行，不然json.loads会更快）和目前的xf.loads(buildz.xf.readz.loads)的速度，xf.loads比json.loads慢7倍，可能是读字符串更频繁，方法调用更多（为了代码更结构化和容易修改），其实有一版更慢(buildz.xf.read.loads，废弃代码，后面看情况删掉)，慢100倍，因为只考虑结构化，没考虑列表增减开销（获得的经验教训是别直接用python的列表list当堆栈做append和pop，特别慢！）
 
 1, a profile file format base on json, make it easy to write profile file, module is in buildz.xf
 2, a ioc profile file read function base on xf format, module is in buildz.ioc

{buildz-0.6.19 → buildz-0.6.20}/README.md

@@ -9,6 +9,9 @@
     buildz.tz: 加些工具，目前只有myerse diff字符串比较算法
     buildz.demo: 使用参考，运行"python -m buildz"会用这个模块
     buildz.db: sql集成工具，自用，里面import了其他sql库，使用运行"python -m buildz.db 配置文件路径"
+    buildz.base: 封装了一个基础类，继承它可以少写一些代码
+    buildz.html: xml（html）内容读取和解析
+    buildz.auto: 自动化操作（主要是做自动化测试方便些，如果不怕写一堆配置文件的话）
 代码关系:
     buildz.xf, buildz.pyz, buildz.argx, buildz.fz, buildz.tz都是独立的模块
     buildz.ioc需要buildz.xf和buildz.pyz
@@ -18,6 +21,8 @@
 运行python -m buildz查看帮助
 
 持续更新中。。。
+2024/10/08:
+增强auto配置功能，在auto增加数据库使用的封装，后续考虑出文档。。。如果有时间
 2024/09/25:
 修复bug，增强html的搜索功能，增强auto配置功能
 auto里加了个request的demo，里面用的requests库，可以直接配置来进行http调用
@@ -30,7 +35,7 @@ ioc:
     加修饰器
     加配置refs
 
-PS: 对比了下json.loads（修改了下json的scanner.py，让它在纯python下运行，不然json.loads会更快）和目前的xf.loads(buildz.xf.readz.loads)的速度，xf.loads比json.loads慢7倍，可能是读字符串更频繁，方法调用更多（为了代码更结构化和容易修改），其实有一版更慢(buildz.xf.read.loads，废弃代码，后面看情况删掉)，慢100倍，因为只考虑结构化，没考虑列表增减开销
+PS: 对比了下json.loads（修改了下json的scanner.py，让它在纯python下运行，不然json.loads会更快）和目前的xf.loads(buildz.xf.readz.loads)的速度，xf.loads比json.loads慢7倍，可能是读字符串更频繁，方法调用更多（为了代码更结构化和容易修改），其实有一版更慢(buildz.xf.read.loads，废弃代码，后面看情况删掉)，慢100倍，因为只考虑结构化，没考虑列表增减开销（获得的经验教训是别直接用python的列表list当堆栈做append和pop，特别慢！）
 
 1, a profile file format base on json, make it easy to write profile file, module is in buildz.xf
 2, a ioc profile file read function base on xf format, module is in buildz.ioc

{buildz-0.6.19 → buildz-0.6.20}/buildz/auto/cache.py

@@ -49,13 +49,14 @@ pass
 
 
 @wrap.obj(id="cache.save")
-@wrap.obj_args("ref, cache", "ref, log")
+@wrap.obj_args("ref, cache.file", "ref, log")
 class Save(Base):
-    def init(self, cache, log):
+    def init(self, cache, log, fkey = "cache.save"):
+        self.fkey = fkey
         self.cache = cache
         self.log = log
     def call(self, maps, fp):
-        fp = xf.get(maps, "cache.save", None)
+        fp = xf.get(maps, self.fkey, None)
         if fp is None:
             self.log.warn(f"cache not save cause 'cache.save' is None")
             return
@@ -66,7 +67,7 @@ class Save(Base):
         return True
 
 pass
-@wrap.obj(id="cache")
+@wrap.obj(id="cache.file")
 @wrap.obj_args("ref, log", "env, cache.rfp.current.first, false")
 class Cache(Base):
     def get(self, key):
@@ -76,7 +77,8 @@ class Cache(Base):
         xf.sets(self.data, key.split("."), val)
     def remove(self, key):
         xf.removes(self.data, key.split("."))
-    def init(self, log, current_first=False):
+    def init(self, log, current_first=False, fkey = "cache"):
+        self.fkey = fkey
         self.current_first = current_first
         self.log = log
         self.data = {}
@@ -113,12 +115,60 @@ class Cache(Base):
                 return _fp
         return fp
     def call(self, maps, fp):
-        fp = xf.g(maps, cache="cache.js")
+        fp = xf.get(maps, self.fkey, "cache.js")
+        if type(fp)!=list:
+            fp = [fp]
+        fps=fp
         data = {}
-        if os.path.isfile(fp):
-            self.log.info(f"load cache from {fp}")
-            data = xf.flush_maps(xf.loadf(fp),visit_list=True)
+        for fp in fps:
+            fp = self.rfp(fp)
+            if os.path.isfile(fp):
+                self.log.info(f"load cache from {fp}")
+                xdata = xf.flush_maps(xf.loadf(fp),visit_list=True)
+                xf.fill(xdata, data, replace=1)
         xf.fill(data, self.data, replace=0)
         return True
 
 pass
+
+@wrap.obj(id="cache.mem")
+@wrap.obj_args("ref, log")
+class Mem(Cache):
+    def init(self, log, current_first=False, fkey = "mem"):
+        super().init(log)
+
+pass
+@wrap.obj(id="cache")
+@wrap.obj_args("ref, cache.file", "ref, cache.mem")
+class Caches(Base):
+    def init(self, cache, mem):
+        self.cache = cache
+        self.mem = mem
+        self.caches = [cache, mem]
+        self.set = cache.set
+        self.remove = cache.remove
+        self.call=cache.call
+        self.rfp = cache.rfp
+        self.get_current = cache.get_current
+        self.add_current = cache.add_current
+        self.set_current = cache.set_current
+    def get_file(self, key):
+        return self.cache.get(key)
+    def get_mem(self, key):
+        return self.mem.get(key)
+    def set_file(self, key, val):
+        self.cache.set(key,val)
+    def set_mem(self, key, val):
+        self.mem.set(key, val)
+    def remove_file(self, key):
+        self.cache.remove(key)
+    def remove_mem(self, key):
+        self.mem.remove(key)
+    def get(self, key):
+        for cache in self.caches:
+            v = cache.get(key)
+            if v is not None:
+                return v
+        return None
+
+pass

buildz-0.6.20/buildz/auto/dbs.py

@@ -0,0 +1,24 @@
+#
+from ..tools import *
+from buildz.ioc import wrap
+import os
+from buildz.db.dv import build
+@wrap.obj(id="dbs")
+@wrap.obj_args("ref, cache", "ref, log")
+class Dbs(Base):
+    def init(self, cache, log):
+        self.cache = cache
+        self.log = log
+        self.dbs = {}
+    def call(self, maps, fp):
+        confs = xf.g(maps, dbs={})
+        for key,conf in confs.items():
+            url,user,pwd,dv = xf.g(conf, url=None, user=None, pwd=None, device=key)
+            dv = build(dv, [url, user, pwd], val)
+            self.dbs[key] = dv
+        self.cache.set_mem("dbs", self.dbs)
+        return True
+
+pass
+
+

{buildz-0.6.19 → buildz-0.6.20}/buildz/auto/deal.py

@@ -9,7 +9,10 @@ from ..ioc import wrap
 class List(Base):
     def init(self, deal, mg):
         self.deal = deal
+        self.curr_deal = deal
         self.mg = mg
+    def curr(self):
+        return self.curr_deal
     def call(self, maps, fp):
         datas = xf.g(maps, datas = [])
         sdeal = xf.g(maps, deal = None)
@@ -19,6 +22,7 @@ class List(Base):
             deal = deal_obj
         elif sdeal is not None:
             deal = self.mg.get(sdeal)
+        self.curr_deal = deal
         for data in datas:
             if not deal(data):
                 return False

buildz-0.6.20/buildz/auto/deal_list.py

@@ -0,0 +1,32 @@
+#
+from ..tools import *
+from buildz.ioc import wrap
+import os
+@wrap.obj(id="deal.list")
+@wrap.obj_args("ref, cache", "ref, log", "ref, cache.modify", "ref, list")
+class List(Base):
+    def init(self, cache, log, upd, lst):
+        self.cache = cache
+        self.log = log
+        self.upd = upd
+        self.lst = lst
+    def call(self, data, fc):
+        data = self.upd(data)
+        fp = xf.g(data, file = None)
+        if fp is not None:
+            fp = self.cache.rfp(fp)
+        datas = xf.g(data, datas=[])
+        if fp is not None and os.path.isfile(fp):
+            datas = xf.loadf(fp)
+        if type(datas)==dict:
+            datas = xf.g(datas, datas=[])
+        deal = self.lst.curr()
+        print(f"List.datas: {xf.dumps(datas,format=1,deep=1)}")
+        for data in datas:
+            if not deal(data):
+                return False
+        return True
+
+pass
+
+

{buildz-0.6.19 → buildz-0.6.20}/buildz/auto/deal_type.py

@@ -20,6 +20,7 @@ pass
 
 @wrap.obj(id = "def.deal.type")
 @wrap.obj_args("ioc, confs")
+@wrap.obj_set(cache="ref, cache")
 class DefDeal(Base):
     def init(self, mg):
         self.mg = mg
@@ -30,7 +31,7 @@ class DefDeal(Base):
         data = xf.g(conf, types={})
         rst = {}
         for _type, calls in data.items():
-            rst[_type] = factory(calls)
+            rst[_type] = factory(calls, True)
         obj = DealType(rst, self.mg.get("cache"))
         maps['deal_obj'] = obj
         return True

buildz-0.6.20/buildz/auto/factory.py

@@ -0,0 +1,31 @@
+#
+
+from .. import xf
+from .. import ioc
+from ..base import Base
+from ..ioc import wrap
+
+class DeepFc(Base):
+    def init(self, fcs, default = None):
+        self.default = default
+        self.fc = None
+        self.next=None
+        if len(fcs)>0:
+            self.fc = fcs[0]
+            self.next = DeepFc(fcs[1:], default)
+    def call(self, data):
+        if self.fc is None:
+            return self.default
+        return self.fc(data, self.next)
+
+pass
+@wrap.obj(id = "buildz.auto.deal.fill")
+@wrap.obj_args("ioc, confs")
+class Fill(Base):
+    def init(self, mg):
+        self.mg = mg
+    def call(self, orders, default=None):
+        fcs = [self.mg.get(id) for id in orders]
+        return DeepFc(fcs,default)
+
+pass

buildz-0.6.20/buildz/auto/init.py

@@ -0,0 +1,13 @@
+#
+
+
+
+
+
+
+
+
+
+
+from . import deal, deal_type, factory, cache, log, config, dbs
+from . import verify, save, request, defs, deal_list

buildz-0.6.20/buildz/auto/log.py

@@ -0,0 +1,22 @@
+#
+
+
+from .. import xf
+from .. import ioc
+from ..base import Base
+from ..ioc import wrap
+from ..tools import *
+import time, sys
+from ..tz.log import FpLog
+@wrap.obj(id="log")
+class AutoLog(FpLog):
+    def call(self, maps, fp):
+        fp = xf.g(maps, log = None)
+        self.fp = fp
+        shows = xf.get(maps, "log.shows")
+        if shows is None:
+            shows = ["info", "warn", "error"]
+        self.shows = shows
+        return True
+
+pass

{buildz-0.6.19 → buildz-0.6.20}/buildz/auto/request.py

@@ -58,10 +58,13 @@ class Request(Base):
         xf.s(data, result_code = rp.status_code)
         if self.debug:
             self.log.debug(f"request url '{url}' return code: {rp.status_code}")
+        show_obj = None
         try:
+            show_obj = rp.content
             xf.s(data, result_content=rp.content)
             debug_ct = rp.content
             s = xf.decode(rp.content, "utf-8")
+            show_obj = s
             if self.debug:
                 self.log.debug(f"request url '{url}' return msg:{s}")
             xf.s(data, result_text=s)
@@ -69,10 +72,13 @@ class Request(Base):
             self.log.warn(f"exp in deal response on '{url}': {exp}")
         try:
             obj = json.loads(s)
+            show_obj = xf.dumps(obj,format=1,deepp=1)
             xf.s(data, result=obj)
         except Exception as exp:
             #self.log.warn(f"exp in deal response on '{url}': {exp}")
             pass
+        if self.debug:
+            self.log.debug(f"request '{url}' response: {show_obj}")
         try:
             xf.s(data, result_cookies=dict(rp.cookies))
         except Exception as exp:

{buildz-0.6.19 → buildz-0.6.20}/buildz/auto/test/cache/cache.js

@@ -1,11 +1,11 @@
-{
-    request: {
-        debug: false
-    },
-    host: www.baidu.com,
-    test: {
-        url: tmpurl,
-        val: 1,
-        code: 1.99
-    },
+{
+    request: {
+        debug: false
+    },
+    host: www.baidu.com,
+    test: {
+        url: tmpurl,
+        val: 1,
+        code: 1.99
+    },
 }

buildz-0.6.20/buildz/auto/test/cache/save.js

@@ -0,0 +1,27 @@
+{
+    cache: {
+        path: {
+            current: [
+                null,
+                "D:\\rootz\\python\\gits\\buildz_upd\\buildz\\auto\\test\\data"
+            ]
+        }
+    },
+    request: {
+        debug: false
+    },
+    host: www.baidu.com,
+    test: {
+        url: tmpurl,
+        val: 1,
+        code: 1.99
+    },
+    result: {
+        code: 1.99
+    },
+    test2: {
+        headers: {
+            content_type: "text/html"
+        }
+    }
+}

{buildz-0.6.19 → buildz-0.6.20}/buildz/auto/test/config/config.js

@@ -1,11 +1,12 @@
-calls: [log, cache, def.deal.type, list, cache.save]
-cache: cache/cache.js
-cache.save: cache/cache.js
+calls: [log, cache, dbs, def.deal.type, list, cache.save]
+cache: [cache/cache.js,cache/save.js]
+cache.save: cache/save.js
 log: log/%Y%m%d_log.txt
 log.shows: [info, warn, debug, error]
 def.deal: {
     types: {
         http.get: [defs, request, verify, save]
         get: [defs, request.get, verify, save]
+        list: [defs, deal.list]
     }
 }

buildz-0.6.19/buildz/auto/test/data/test.js → buildz-0.6.20/buildz/auto/test/data/fp.js

@@ -24,4 +24,7 @@ datas: [
     // 或者配置所在的文件
     item1.js
     item2.js
+    {
+        type: list
+    }
 ]

buildz-0.6.20/buildz/auto/test/data/test.js

@@ -0,0 +1,31 @@
+configs: config/config.js
+datas: [
+    //写一个配置
+    {
+        note: test
+        url: test
+        type: http.get
+        data: {
+            url: "#{test.url}"
+        }
+        save: {
+            test.url: data.url
+        }
+        result: {
+            code: 1.99
+        }
+        verify: [
+            "result.code, [>, 1]"
+        ]
+        save: {
+            result.code: result.code
+        }
+    }
+    // 或者配置所在的文件
+    item1.js
+    item2.js
+    {
+        type: list
+        file: fp.js
+    }
+]

buildz-0.6.20/buildz/auto/test/log/20240930_log.txt

@@ -0,0 +1,24 @@
+[INFO] 2024-09-30 12:18:42 load cache from cache/cache.js
+[DEBUG] 2024-09-30 12:18:42 test data: {'note': 'test', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl'}, 'save': {'result.code': 'result.code'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[DEBUG] 2024-09-30 12:18:42 test data: {'note': 'test1', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl', 'val': 1}, 'save': {'test.url': 'data.url'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[INFO] 2024-09-30 12:19:00 load cache from cache/cache.js
+[INFO] 2024-09-30 12:19:00 load cache from cache/save.js
+[DEBUG] 2024-09-30 12:19:00 test data: {'note': 'test', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl'}, 'save': {'result.code': 'result.code'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[DEBUG] 2024-09-30 12:19:00 test data: {'note': 'test1', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl', 'val': 1}, 'save': {'test.url': 'data.url'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[INFO] 2024-09-30 20:22:25 load cache from cache/cache.js
+[INFO] 2024-09-30 20:22:25 load cache from cache/save.js
+[DEBUG] 2024-09-30 20:22:25 test data: {'note': 'test', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl'}, 'save': {'result.code': 'result.code'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[DEBUG] 2024-09-30 20:22:25 test data: {'note': 'test1', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl', 'val': 1}, 'save': {'test.url': 'data.url'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[ERROR] 2024-09-30 20:22:25 error in request 'https://www.baidu.com/' with method get: HTTPSConnectionPool(host='www.baidu.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002247FE3AF10>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
+[INFO] 2024-09-30 20:22:38 load cache from cache/cache.js
+[INFO] 2024-09-30 20:22:38 load cache from cache/save.js
+[DEBUG] 2024-09-30 20:22:38 test data: {'note': 'test', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl'}, 'save': {'result.code': 'result.code'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[DEBUG] 2024-09-30 20:22:38 test data: {'note': 'test1', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl', 'val': 1}, 'save': {'test.url': 'data.url'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[INFO] 2024-09-30 20:26:22 load cache from cache/cache.js
+[INFO] 2024-09-30 20:26:22 load cache from cache/save.js
+[DEBUG] 2024-09-30 20:26:23 test data: {'note': 'test', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl'}, 'save': {'result.code': 'result.code'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[DEBUG] 2024-09-30 20:26:23 test data: {'note': 'test1', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl', 'val': 1}, 'save': {'test.url': 'data.url'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[INFO] 2024-09-30 20:40:14 load cache from cache/cache.js
+[INFO] 2024-09-30 20:40:14 load cache from cache/save.js
+[DEBUG] 2024-09-30 20:40:14 test data: {'note': 'test', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl'}, 'save': {'result.code': 'result.code'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}
+[DEBUG] 2024-09-30 20:40:14 test data: {'note': 'test1', 'url': 'test', 'type': 'http.get', 'data': {'url': 'tmpurl', 'val': 1}, 'save': {'test.url': 'data.url'}, 'result': {'code': 1.99}, 'verify': ['result.code, [>, 1]']}