dv-flow-mgr 0.0.1.12750690879a1__tar.gz → 0.0.1.12822558956a1__tar.gz

{dv_flow_mgr-0.0.1.12750690879a1 → dv_flow_mgr-0.0.1.12822558956a1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: dv-flow-mgr
-Version: 0.0.1.12750690879a1
+Version: 0.0.1.12822558956a1
 Summary: DV Flow Manager is a build system for silicon design
 Author-email: Matthew Ballance <matt.ballance@gmail.com>
 License:                                  Apache License

{dv_flow_mgr-0.0.1.12750690879a1 → dv_flow_mgr-0.0.1.12822558956a1}/docs/Roadmap.md

@@ -29,6 +29,11 @@ are evaluated.
 - JQ-based data extraction
 - YAML task templates / expansions
 - Support for annotating job requirements 
+- Mark tasks as producing and accepting certain data
+  - FileSet task `produces` fileset of `type`
+  - SimImage task `accepts` systemVerilogSource, verilogSource, verilogPreCompLib, etc
+  => Mostly useful for checking and suggestion
+  => As more are marked, can treat as more binding
 
 # Library
 

{dv_flow_mgr-0.0.1.12750690879a1 → dv_flow_mgr-0.0.1.12822558956a1}/docs/index.rst

@@ -6,9 +6,9 @@
 DV Flow Manager
 ===============
 
-DV Flow Manager helps users capture the tasks and dataflow in their
-design and verification (DV) flows. You can think of DV Flow Manager as a 
-sort of "make for silicon engineering".
+DV Flow Manager provides a "make for silicon engineering": a specification
+for capturing design and verification (DV) tasks and dataflow in a way that
+enables concurrent execution and efficient avoidance of redundant work.
 
 .. mermaid::
 
@@ -28,4 +28,5 @@ sort of "make for silicon engineering".
    :caption: Contents:
 
    quickstart
+   intro
    reference

dv_flow_mgr-0.0.1.12822558956a1/docs/intro.rst

@@ -0,0 +1,185 @@
+############
+Introduction
+############
+
+
+
+
+Many software languages have co-evolved with a build system. For example, C/C++ 
+has Make and CMake. Java has ANT, Maven, and Gradle. All of these build systems
+provide features that cater to specific ways that a given language is processed,
+and provide built-in notions to make setting up simple cases as easy as possible.
+
+One simple example is Make and a single-file C program. Take the code below:
+
+.. code-block:: C
+    
+    #include <stdio.h>
+
+    int main() {
+        printf("Hello, world!\n");
+        return 0;
+    }
+
+
+Make provides enough built-in features that are C/C++-specific that we can create
+an executable from this source file (assume it's named hello.c) simply by running:
+
+.. code-block:: bash
+
+    make hello
+
+Make knows about C files, it knows about the existance of a C compiler, and it knows
+that an executable can be created from a C file of the same name.
+
+Meanwhile, in Silicon Engineering Land...
+=========================================
+
+Much like software languages, the languages, tools, and flows used in silicon engineering
+have their own unique characteristics. For example, in a silicon-design environment, many 
+flows are run over the same source files -- possibly with different configurations.
+
+* We compile our design with a UVM testbench to run dynamic (simulation-based) verification
+* We compile our design with different testbenches to run formal verification
+* We likely use slightly different subset when targeting synthesis
+
+In addition, we also need to be flexible when it comes to tooling. Over time, we'll likely
+use different tools from different providers, and want our projects to adapt as easily as 
+possible to a change of tool. It's also likely that we will either want to add new tools
+to our environment over time, or adapt our environment to take advantage of new 
+productivity-enhancing tool features.
+
+DV Flow Manager is designed to be the 'make' for silicon engineering. There are three
+aspects to the tool:
+
+* **Flow Specification** - Processing steps for a given project are captured in a hierarchy
+  of YAML files. The flow-specification schema is designed to be tool-independent, such 
+  that multiple tools can be implemented that comprehend a flow specification.
+* **Task Library** - Processing steps are implemented as `tasks`. A library of common tasks
+  is defined to cover common cases, such as creating a simulation image. External libraries
+  of tasks are supported, such that tools can bundle a task library along with the tool installation.
+* **Tools** - The Python implementation of DV Flow Manager is one example of a tool. Other tools
+  may be added in the future to provide visualization, simplify development, etc.
+
+
+
+Key Concepts
+============
+
+DV Flow Manager has three key concepts:
+* **Package** - A packages is parameterizd namespace that contain tasks. 
+* **Task** - A task is a processing step in a flow. Tasks represent a data-processing step, which
+  might be as simple as building a list of files, or might be a complex as creating a hardened macro
+  from multiple source collections. 
+* **Dataflow Dependencies** - Tasks are related by dataflow dependencies. In order for a task to 
+  execute, the data from all of its dependencies must be available. Each task also produces a 
+  dataflow object that can be consumed by other tasks. 
+
+Let's look at an example to better-understand these concepts.
+
+.. code-block:: YAML
+
+    package:
+      name: my_ip
+
+      imports:
+      - name: hdl.sim.vlt
+        as: hdl.sim
+
+      tasks:
+        - name: rtl
+          uses: std.Fileset
+          with:
+            base: "rtl"
+            include: "*.sv"
+
+        - name: tb
+          uses: std.Fileset
+          needs: [rtl]
+          with:
+            base: "tb"
+            include: "*.sv"
+
+        - name: sim
+          uses: hdl.sim.SimImage
+          needs: [rtl, tb]
+
+        -name: test1
+          uses: hdl.sim.RunSim
+          needs: [sim]
+
+The code above specifies two collections of source code --
+one for the design and one for the testbench. This source
+code is compiled into as simulation image using the 
+pre-defined task named `hdl.sim.SimImage`. After,
+we execute the simulation image.
+
+
+.. mermaid::
+
+    flowchart TD
+      A[rtl] --> B[tb]
+      B[tb] --> E[sim]
+      E --> F[test1]
+
+The task graph for this flow is shown above. Each step depends on the
+prior step, so there is no opportunity for concurrent execution.
+
+Now, let's say that we want to run a series of tests. We can add 
+a new task per tests, where we customize the activity that is run
+by passing arguments to the simulation.
+
+.. code-block:: YAML
+
+    # ...
+        -name: test1
+          uses: hdl.sim.RunSim
+          needs: [sim]
+        -name: test2
+          uses: hdl.sim.RunSim
+          needs: [sim]
+        -name: test3
+          uses: hdl.sim.RunSim
+          needs: [sim]
+
+.. mermaid::
+
+    flowchart TD
+      A[rtl] --> B[tb]
+      B[tb] --> E[sim]
+      E --> F[test1]
+      E --> G[test2]
+      E --> H[test3]
+
+Our task graph now looks like the above. Our build tasks are sequential,
+while our test tasks only depend on the simulation image being
+up-to-date, and and can execute concurrently.
+
+## Dataflow
+
+What ties all the tasks above together is dependency-based dataflow.
+
+.. code-block:: YAML
+
+        - name: tb
+          uses: std.Fileset
+          needs: [rtl]
+          with:
+            base: "tb"
+            include: "*.sv"
+
+        - name: sim
+          uses: hdl.sim.SimImage
+          needs: [rtl, tb]
+
+When the `sim` task places dependencies on the `rtl` and `tb`
+tasks, it receives the output from those tasks as input. In 
+this case, that means that the simulation-image compilation
+task has a list of all of the source files that it needs to
+compile. The `sim` task also produces an output, which contains 
+a reference to the directory where the simulation image resides.
+The `test` tasks use this input to locate the simulation image.
+
+
+
+

{dv_flow_mgr-0.0.1.12750690879a1 → dv_flow_mgr-0.0.1.12822558956a1}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "dv-flow-mgr"
-version = "0.0.1.12750690879a1"
+version = "0.0.1.12822558956a1"
 dependencies = [
     "pyyaml",
     "toposort",

{dv_flow_mgr-0.0.1.12750690879a1 → dv_flow_mgr-0.0.1.12822558956a1}/src/dv_flow_mgr/fileset.py

@@ -24,8 +24,9 @@ from pydantic import BaseModel
 from typing import Any, Dict, List, Tuple
 
 class FileSet(BaseModel):
-    src : str
     type : str
     basedir : str
+    name : str = ""
+    src : str = None
     files : List[str] = dc.Field(default_factory=list)
     params : Dict[str,str] = dc.Field(default_factory=dict)

{dv_flow_mgr-0.0.1.12750690879a1 → dv_flow_mgr-0.0.1.12822558956a1}/src/dv_flow_mgr/task.py

@@ -155,14 +155,6 @@ class Task(object):
             # Merge filesets. A fileset with the same
             print("deps_o: %s" % str(deps_o))
 
-            # First, merge the dep maps of all the inputs
-            deps_m = self.depends[0].output.deps.copy()
-            for deps in map(lambda d: d.deps, self.depends[1:]):
-                for k,v in deps.items():
-                    if k in deps_m:
-                        deps_m[k].add(v)
-                    else:
-                        deps_m[k] = set(v)
 
             print("deps_m: %s" % str(deps_m))
 
@@ -182,8 +174,7 @@ class Task(object):
         # Mark the source of this data as being this task
         input.src = self.name
 
-        if not os.path.isdir(self.rundir):
-            os.makedirs(self.rundir)
+        self.init_rundir()
 
         result = await self.run(input)
 
@@ -202,9 +193,7 @@ class Task(object):
             result = self.getOutput()
 
         # Write-back the memento, if specified
-        if self.memento is not None:
-            with open(os.path.join(self.rundir, "memento.json"), "w") as fp:
-                fp.write(self.memento.model_dump_json(indent=2))
+        self.save_memento()
 
         self.running = False
 
@@ -214,6 +203,15 @@ class Task(object):
     async def run(self, input : TaskData) -> TaskData:
         raise NotImplementedError("TaskImpl.run() not implemented")
     
+    def init_rundir(self):
+        if not os.path.isdir(self.rundir):
+            os.makedirs(self.rundir)
+
+    def save_memento(self):
+        if self.memento is not None:
+            with open(os.path.join(self.rundir, "memento.json"), "w") as fp:
+                fp.write(self.memento.model_dump_json(indent=2))
+    
     def setOutput(self, output : TaskData):
         self.output_set = True
         output.src = self.name

dv_flow_mgr-0.0.1.12822558956a1/src/dv_flow_mgr/task_data.py

@@ -0,0 +1,269 @@
+#****************************************************************************
+#* task_data.py
+#*
+#* Copyright 2023 Matthew Ballance and Contributors
+#*
+#* Licensed under the Apache License, Version 2.0 (the "License"); you may 
+#* not use this file except in compliance with the License.  
+#* You may obtain a copy of the License at:
+#*
+#*   http://www.apache.org/licenses/LICENSE-2.0
+#*
+#* Unless required by applicable law or agreed to in writing, software 
+#* distributed under the License is distributed on an "AS IS" BASIS, 
+#* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  
+#* See the License for the specific language governing permissions and 
+#* limitations under the License.
+#*
+#* Created on:
+#*     Author: 
+#*
+#****************************************************************************
+import enum
+import pydantic.dataclasses as dc
+from pydantic import BaseModel
+from typing import Any, Dict, Set, List, Tuple
+from .fileset import FileSet
+from toposort import toposort
+
+class TaskDataParamOpE(enum.Enum):
+    Set = enum.auto()
+    Append = enum.auto()
+    Prepend = enum.auto()
+    PathAppend = enum.auto()
+    PathPrepend = enum.auto()
+
+class TaskDataParamKindE(enum.Enum):
+    String = enum.auto()
+    FilePath = enum.auto()
+    SearchPath = enum.auto()
+    List = enum.auto()
+
+class TaskDataParamOp(BaseModel):
+    op : TaskDataParamOpE
+    value : Any
+
+class TaskDataParam(BaseModel):
+    kind : TaskDataParamKindE
+    ops : List[TaskDataParamOp] = dc.Field(default_factory=list)
+
+class TaskData(BaseModel):
+    src : str = None
+    params : Dict[str,TaskDataParam] = dc.Field(default_factory=dict)
+    deps : Dict[str,Set[str]] = dc.Field(default_factory=dict)
+    filesets : List[FileSet] = dc.Field(default_factory=list)
+    changed : bool = False
+
+    def hasParam(self, name: str) -> bool:
+        return name in self.params
+    
+    def getParam(self, name: str) -> Any:
+        return self.params[name]
+    
+    def setParam(self, name: str, value: Any):
+        self.params[name] = value
+
+    def addFileSet(self, fs : FileSet):
+        self.filesets.append(fs)
+
+    def getFileSets(self, type=None, order=True) -> List[FileSet]:
+        ret = []
+
+        if order:
+            # The deps map specifies task dependencies
+
+            candidate_fs = []
+            for fs in self.filesets:
+                if type is None or fs.type in type:
+                    candidate_fs.append(fs)
+
+            order = toposort(self.deps)
+
+            for order_s in order:
+                for fs in candidate_fs:
+                    if fs.src in order_s:
+                            ret.append(fs)
+        else:
+            for fs in self.filesets:
+                if type is None or fs.type in type:
+                    ret.append(fs)
+        
+        return ret
+
+    def copy(self) -> 'TaskData':
+        ret = TaskData()
+        ret.src = self.src
+        ret.params = self.params.copy()
+        for d in self.deps:
+            ret.deps.append(d.clone())
+        ret.changed = self.changed
+        return ret
+    
+    def setParamVal(self, name: str, kind : TaskDataParamKindE, value: Any):
+        if name not in self.params:
+            self.params[name] = TaskDataParam(kind=kind)
+        self.params[name].ops.append(TaskDataParamOp(op=TaskDataParamOpE.Set, value=value))
+    
+    def getParamVal(self, name: str) -> Any:
+        if name not in self.params.keys():
+            raise Exception("No such parameter: %s" % name)
+        param = self.params[name]
+        value = param.ops[0].value
+
+        if len(param.ops) > 1:
+            for op in param.ops[1:]:
+                if op.op == TaskDataParamOpE.Append:
+                    if isinstance(value, list):
+                        value.extend(op.value)
+                    else:
+                        value += op.value
+                elif op.op == TaskDataParamOpE.Prepend:
+                    if isinstance(value, list):
+                        for nv in op.value:
+                            value.insert(0, nv)
+                    else:
+                        value = op.value + value
+                elif op.op == TaskDataParamOpE.PathAppend:
+                    if isinstance(value, list):
+                        value = ":".join(value)
+                    value = value + ":" + op.value
+                elif op.op == TaskDataParamOpE.PathPrepend:
+                    if isinstance(value, list):
+                        value = ":".join(value)
+                    value = op.value + ":" + value
+
+        return value
+
+    @staticmethod    
+    def merge(incoming : List['TaskData'], local : 'TaskData' = None) -> 'TaskData':
+        """Merges incoming data with local settings and produces an output"""
+
+        # Deal with the dependency trees first
+        output = TaskData()
+
+        # First, merge the dep maps of all the inputs
+        output.deps = incoming[0].deps.copy()
+        for deps in map(lambda i: i.deps, incoming[1:]):
+            for k,v in deps.items():
+                if k not in output.deps:
+                    output.deps[k] = []
+                for vi in v:
+                    if vi not in output.deps[k]:
+                        output.deps[k].append(v)
+
+        # Process filesets
+        for inp in incoming:
+            for fs in inp.filesets:
+                exists = False
+                for fs_o in output.filesets:
+                    if fs_o.name == fs.name and fs_o.src == fs.src:
+                        exists = True
+                        break
+                if not exists:
+                    output.addFileSet(fs.model_copy())
+
+        # Now, deal with parameters
+        # Find collisions first
+        colliding_keys = set()
+        passthrough_keys = set()
+
+        for i in incoming:
+            for k in i.params.keys():
+                if k in passthrough_keys:
+                    colliding_keys.add(k)
+                else:
+                    passthrough_keys.add(k)
+
+        # Now, removes those that are locally set
+        local_set_params = set()
+        if local is not None:
+            for k,v in local.params.items():
+                if len(v.ops) == 1 and v.ops[0].op == TaskDataParamOpE.Set:
+                    local_set_params.add(k)
+                    # If are setting locally, it's not passthrough
+                    passthrough_keys.remove(k)
+                    if k in colliding_keys:
+                        colliding_keys.remove(k)
+        
+        # Construct the passthrough set by removing 
+        # colliding entries and those that we will set locally
+        for k in colliding_keys:
+            if k in passthrough_keys:
+                passthrough_keys.remove(k)
+
+        # For the remaining keys, check for conflicts by
+        # confirming that the last 'set' in each incoming parameter
+        # are equal
+        for k in colliding_keys:
+            value = None
+            for i,inp in enumerate(incoming):
+                value_i = None
+                param = inp.params[k]
+                if len(param.ops) == 1:
+                    value_i = param.ops[0].value
+                else:
+                    # Iterate in reverse over the operations
+                    for op in param.ops[::-1]:
+                        if op.op == TaskDataParamOpE.Set:
+                            value_i = op.value
+                            break
+                if not i:
+                    value = value_i
+                else:
+                    if value != value_i:
+                        raise Exception("Parameter %s has conflicting values (%s %s)" % (
+                            k,
+                            str(value),
+                            value(value_i)))
+
+
+        # Now, we need to construct the result
+        # - copy over passthrough parameters
+        # - add locally-set parameters
+        # - for others
+        #   - Apply full list for first input
+        #   - Apply all beyond the last 'set' operation for others
+        for k in passthrough_keys:
+            # Find an input that has the parameter
+            for inp in incoming:
+                if k in inp.params:
+                    break
+            # Find the value of the param
+            param = inp.params[k]
+
+            if len(param.ops) == 1:
+                output.params[k] = TaskDataParam(kind=param.kind)
+                output.params[k].ops.append(param.ops[0])
+            else:
+                for op in param.ops[::-1]:
+                    if op.op == TaskDataParamOpE.Set:
+                        output.params[k] = TaskDataParam(kind=param.kind)
+                        output.params[k].ops.append(op)
+                        break
+        for k in local_set_params:
+            output.params[k] = local.params[k].model_copy()
+
+        for k in colliding_keys:
+            value = None
+            for i,inp in enumerate(incoming):
+                # Find the last location that performs a 'set' 
+                last_set_i = -1
+                param = inp.params[k]
+                if len(param.ops) == 1:
+                    last_set_i = 0
+                else:
+                    # Iterate in reverse over the operations
+                    for j,op in enumerate(param.ops[::-1]):
+                        if op.op == TaskDataParamOpE.Set:
+                            last_set_i = j
+                            break
+                
+                if not i:
+                    # Copy the full list, including the last 'set'
+                    output.params[k].ops = param.param[last_set_i:].copy()
+                else:
+                    # append any additional directives
+                    if last_set_i+1 < len(param.ops):
+                        output.params[k].extend(param.ops[last_set_i+1:])
+
+        return output

dv_flow_mgr-0.0.1.12822558956a1/src/dv_flow_mgr/tasklib/builtin_pkg.py

@@ -0,0 +1,61 @@
+import os
+import sys
+import glob
+import fnmatch
+import importlib
+import pydantic.dataclasses as dc
+from ..package import TaskCtor
+from ..task import Task, TaskParams, TaskCtorT
+from ..task_data import TaskData
+from ..task_memento import TaskMemento
+from typing import List, Tuple
+import dataclasses as dc
+from ..package_def import Package
+
+class TaskPyClass(Task):
+
+    async def run(self, input : TaskData) -> TaskData:
+
+        if self.srcdir not in sys.path:
+            sys.path.insert(0, self.srcdir)
+
+        print("sys.path: %s" % str(sys.path), flush=True)
+        idx = self.params.pyclass.rfind('.')
+        modname = self.params.pyclass[:idx]
+        clsname = self.params.pyclass[idx+1:]
+
+        if os.path.isfile(os.path.join(self.basedir, "my_module.py")):
+            print("my_module.py exists", flush=True)
+        else:
+            print("my_module.py does not exist", flush=True)
+
+        try:
+            print("modname=%s" % modname, flush=True)
+            module = importlib.import_module(modname)
+        except ModuleNotFoundError as e:
+            print("Module not found: %s syspath=%s" % (str(e), str(sys.path)), flush=True)
+            raise e
+
+        cls = getattr(module, clsname)
+
+        obj = cls(self.name, self.task_id, self.session, self.basedir, srcdir=self.srcdir)
+
+        return await obj.run(input)
+
+
+class TaskPyClassParams(TaskParams):
+    pyclass : str
+
+class TaskPyClassMemento(TaskMemento):
+    pass
+
+class TaskPyClassCtor(TaskCtorT):
+    def __init__(self):
+        super().__init__(TaskPyClassParams, TaskPyClass)
+
+@dc.dataclass
+class PackageBuiltin(Package):
+
+    def __post_init__(self):
+        print("PackageBuiltin::__post_init__", flush=True)
+        self.tasks["PyClass"] = TaskPyClass()

dv_flow_mgr-0.0.1.12822558956a1/src/dv_flow_mgr/tasklib/std/flow.dv

@@ -0,0 +1,12 @@
+
+package:
+  name: std
+
+  tasks:
+  - name: Exec
+    uses: builtin.PyClass
+    params:
+    - name: cmd
+      type: string
+    - name: pyclass
+      type: string

{dv_flow_mgr-0.0.1.12750690879a1 → dv_flow_mgr-0.0.1.12822558956a1}/src/dv_flow_mgr.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: dv-flow-mgr
-Version: 0.0.1.12750690879a1
+Version: 0.0.1.12822558956a1
 Summary: DV Flow Manager is a build system for silicon design
 Author-email: Matthew Ballance <matt.ballance@gmail.com>
 License:                                  Apache License

{dv_flow_mgr-0.0.1.12750690879a1 → dv_flow_mgr-0.0.1.12822558956a1}/src/dv_flow_mgr.egg-info/SOURCES.txt

@@ -12,6 +12,7 @@ docs/Stages.md
 docs/TypesAndDefs.md
 docs/conf.py
 docs/index.rst
+docs/intro.rst
 docs/quickstart.rst
 docs/reference.rst
 src/dv_flow_mgr/__init__.py
@@ -36,6 +37,7 @@ src/dv_flow_mgr.egg-info/requires.txt
 src/dv_flow_mgr.egg-info/top_level.txt
 src/dv_flow_mgr/cmds/cmd_run.py
 src/dv_flow_mgr/share/flow.json
+src/dv_flow_mgr/tasklib/builtin_pkg.py
 src/dv_flow_mgr/tasklib/hdl/sim/mti_pkg.py
 src/dv_flow_mgr/tasklib/hdl/sim/mti_task_sim_image.py
 src/dv_flow_mgr/tasklib/hdl/sim/mti_task_sim_run.py
@@ -49,12 +51,16 @@ src/dv_flow_mgr/tasklib/hdl/sim/vlt_pkg.py
 src/dv_flow_mgr/tasklib/hdl/sim/vlt_task_sim_image.py
 src/dv_flow_mgr/tasklib/hdl/sim/vlt_task_sim_run.py
 src/dv_flow_mgr/tasklib/std/fileset.py
+src/dv_flow_mgr/tasklib/std/flow.dv
 src/dv_flow_mgr/tasklib/std/pkg_std.py
 src/dv_flow_mgr/tasklib/std/std.dfs
 src/dv_flow_mgr/tasklib/std/task_fileset.py
 src/dv_flow_mgr/tasklib/std/task_null.py
 tests/examples/example1/example1.flow
+tests/unit/__init__.py
+tests/unit/test_data_merge.py
 tests/unit/test_fileset.py
+tests/unit/test_pyclass.py
 tests/unit/test_smoke copy.sav
 tests/unit/test_smoke.py
 tests/unit/data/fileset/test1/test1.dfs

dv_flow_mgr-0.0.1.12822558956a1/tests/unit/__init__.py

@@ -0,0 +1 @@
+

dv_flow_mgr-0.0.1.12822558956a1/tests/unit/test_data_merge.py

@@ -0,0 +1,121 @@
+import asyncio
+import io
+import os
+import dataclasses as dc
+import pytest
+from typing import List
+import yaml
+from dv_flow_mgr import TaskData, FileSet, Session, TaskData, TaskDataParamKindE
+from pydantic import BaseModel
+from shutil import copytree
+from .tasklib import TaskNull
+
+def test_empty_in():
+
+    in1 = TaskData()
+    in2 = TaskData()
+
+    out = TaskData.merge([in1, in2])
+
+    assert len(out.params) == 0
+
+def test_empty_combine_nonoverlap_in():
+
+    in1 = TaskData()
+    in1.setParamVal("v1", "1")
+    in2 = TaskData()
+    in2.setParamVal("v2", "2")
+
+    out = TaskData.merge([in1, in2])
+
+    assert len(out.params) != 0
+    assert "v1" in out.params.keys()
+    assert out.getParamVal("v1") == "1"
+    assert "v2" in out.params.keys()
+    assert out.getParamVal("v2") == "2"
+
+def test_empty_combine_nonoverlap_in():
+
+    in1 = TaskData()
+    in1.setParamVal("v1", TaskDataParamKindE.String, "1")
+    in2 = TaskData()
+    in2.setParamVal("v2", TaskDataParamKindE.String, "2")
+
+    out = TaskData.merge([in1, in2])
+
+    assert len(out.params) != 0
+    assert "v1" in out.params.keys()
+    assert out.getParamVal("v1") == "1"
+    assert "v2" in out.params.keys()
+    assert out.getParamVal("v2") == "2"
+
+def test_conflict_1():
+
+    in1 = TaskData()
+    in1.setParamVal("v1", TaskDataParamKindE.String, "1")
+    in2 = TaskData()
+    in2.setParamVal("v1", TaskDataParamKindE.String, "2")
+
+    with pytest.raises(Exception):
+        out = TaskData.merge([in1, in2])
+
+def test_fileset_merge_1():
+    in1 = TaskData(src="in1")
+    in1.addFileSet(FileSet(
+        src="in1",
+        type="systemVerilogSource",
+        basedir="."))
+
+    in2 = TaskData(src="in2")
+    in2.addFileSet(FileSet(
+        src="in2",
+        type="systemVerilogSource",
+        basedir="."))
+
+    out = TaskData.merge([in1, in2])
+
+    assert len(out.filesets) == 2
+
+def test_fileset_merge_common_dep_1():
+    in1 = TaskData(src="in1")
+    in1.addFileSet(FileSet(
+        src="in1",
+        type="systemVerilogSource",
+        basedir="."))
+    in1.addFileSet(FileSet(
+        src="in0",
+        type="systemVerilogSource",
+        basedir="."))
+    in1.deps = {
+        "in1": ["in0"]
+    }
+
+    in2 = TaskData(src="in2")
+    in2.addFileSet(FileSet(
+        src="in1",
+        type="systemVerilogSource",
+        basedir="."))
+    in2.addFileSet(FileSet(
+        src="in2",
+        type="systemVerilogSource",
+        basedir="."))
+    in2.addFileSet(FileSet(
+        src="in0",
+        type="systemVerilogSource",
+        basedir="."))
+    in2.deps = {
+        "in1": ["in0"],
+        "in2": ["in1"]
+    }
+
+    out = TaskData.merge([in2, in1])
+
+    assert len(out.filesets) == 3
+    fs = out.getFileSets(type=["systemVerilogSource"], order=True)
+    assert len(fs) == 3
+
+    assert fs[0].src == "in0"
+    assert fs[1].src == "in1"
+    assert fs[2].src == "in2"
+
+

dv_flow_mgr-0.0.1.12822558956a1/tests/unit/test_pyclass.py

@@ -0,0 +1,29 @@
+
+import os
+import asyncio
+import pytest
+from dv_flow_mgr import TaskData
+from dv_flow_mgr.tasklib.builtin_pkg import TaskPyClass, TaskPyClassParams
+
+def test_smoke(tmpdir):
+    module = """
+from dv_flow_mgr import Task, TaskData
+
+class foo(Task):
+
+    async def run(self, input : TaskData) -> TaskData:
+        print("foo::run", flush=True)
+        return input
+"""
+    print("test_smoke")
+
+    with open(os.path.join(tmpdir, "my_module.py"), "w") as f:
+        f.write(module)
+
+    params = TaskPyClassParams(pyclass="my_module.foo")
+    basedir = os.path.join(tmpdir)
+    task = TaskPyClass("t1", -1, None, params, basedir, srcdir=basedir)
+
+    in_data = TaskData()
+    asyncio.run(task.run(in_data))
+    pass

{dv_flow_mgr-0.0.1.12750690879a1 → dv_flow_mgr-0.0.1.12822558956a1}/tests/unit/test_smoke.py

@@ -5,7 +5,7 @@ import dataclasses as dc
 import pytest
 from typing import List
 import yaml
-from dv_flow_mgr import FileSet, PackageDef, Session, TaskData, TaskImpl
+from dv_flow_mgr import FileSet, PackageDef, Session, TaskData
 from pydantic import BaseModel
 from shutil import copytree
 

dv_flow_mgr-0.0.1.12750690879a1/src/dv_flow_mgr/task_data.py

@@ -1,94 +0,0 @@
-#****************************************************************************
-#* task_data.py
-#*
-#* Copyright 2023 Matthew Ballance and Contributors
-#*
-#* Licensed under the Apache License, Version 2.0 (the "License"); you may 
-#* not use this file except in compliance with the License.  
-#* You may obtain a copy of the License at:
-#*
-#*   http://www.apache.org/licenses/LICENSE-2.0
-#*
-#* Unless required by applicable law or agreed to in writing, software 
-#* distributed under the License is distributed on an "AS IS" BASIS, 
-#* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  
-#* See the License for the specific language governing permissions and 
-#* limitations under the License.
-#*
-#* Created on:
-#*     Author: 
-#*
-#****************************************************************************
-import enum
-import pydantic.dataclasses as dc
-from pydantic import BaseModel
-from typing import Any, Dict, Set, List, Tuple
-from .fileset import FileSet
-
-class TaskDataParamOpE(enum.Enum):
-    Set = enum.auto()
-    Append = enum.auto()
-    Prepend = enum.auto()
-    PathAppend = enum.auto()
-    PathPrepend = enum.auto()
-
-class TaskDataParamOp(BaseModel):
-    op : TaskDataParamOpE
-    value : Any
-
-class TaskDataParam(BaseModel):
-    value : Any
-    ops : List[TaskDataParamOp] = dc.Field(default_factory=list)
-
-class TaskData(BaseModel):
-    src : str = None
-    params : Dict[str,Any] = dc.Field(default_factory=dict)
-    deps : Dict[str,Set[str]] = dc.Field(default_factory=dict)
-    changed : bool = False
-
-    def hasParam(self, name: str) -> bool:
-        return name in self.params
-    
-    def getParam(self, name: str) -> Any:
-        return self.params[name]
-    
-    def setParam(self, name: str, value: Any):
-        self.params[name] = value
-
-    def addFileSet(self, fs : FileSet):
-        fs.src = self.src
-        if "filesets" not in self.params:
-            self.params["filesets"] = []
-        self.params["filesets"].append(fs)
-
-    def getFileSets(self, type=None) -> List[FileSet]:
-        ret = []
-
-        if "filesets" in self.params:
-            for fs in self.params["filesets"]:
-                if type is None or fs.type in type:
-                    ret.append(fs)
-        
-        return ret
-
-    def copy(self) -> 'TaskData':
-        ret = TaskData()
-        ret.src = self.src
-        ret.params = self.params.copy()
-        for d in self.deps:
-            ret.deps.append(d.clone())
-        ret.changed = self.changed
-        return ret
-    
-    def merge(self, other):
-        for k,v in other.params.items():
-            if k not in self.params:
-                if hasattr(v, "copy"):
-                    self.params[k] = v.copy()
-                else:
-                    self.params[k] = v
-            elif hasattr(self.params[k], "merge"):
-                self.params[k].merge(v)
-            elif self.params[k] != v:
-                raise Exception("Parameter %s has conflicting values" % k)
-