dv-flow-mgr 0.0.1.12992722023a1__tar.gz → 0.0.1.13335470340a1__tar.gz

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: dv-flow-mgr
-Version: 0.0.1.12992722023a1
+Version: 0.0.1.13335470340a1
 Summary: DV Flow Manager is a build system for silicon design
 Author-email: Matthew Ballance <matt.ballance@gmail.com>
 License:                                  Apache License

dv_flow_mgr-0.0.1.13335470340a1/docs/KeyArchitecture.md

@@ -0,0 +1,65 @@
+
+# Tasks
+- A task has 0..N dependencies (references to tasks whose output
+  data must be available prior to execution of the task)
+- The input data available to a task includes
+  - Output data from its dependencies
+  - Memento data from its prior execution (a memento is a record of task execution)
+    - Makes sense to include markers here
+  - Locally-specified data (ie task parameters)
+- The core implementation of a task is invoked with a
+  consolidated set of data that is specified in the task definition.
+  - Example: select all FileSet that have kind systemVerilog, verilog, or VHDL 
+- The output of a task is comprised of (think of this as a filter?)
+  - Output data from its dependencies
+  - Memento data from the implementation, including marker information
+
+# Types
+- Types are named structures
+
+# Task Parameters
+- A task may have parameters. Parameters are singletons, and are
+  not extensible from outside. 
+- If a task needs to accept outside input, it must select that
+  by type from input parameter sets
+- Direct parameters
+- Task parameters are exposed to task implementation via a type with the task name
+- Do we really need this?
+- Using explicit types results in one extra layer of indirection
+- Still may have to deal with ambiguity...
+
+- name: RunSim
+  uses: SimRun
+  with:
+  - name: simargs
+    type: 
+      list:
+        item: SimArgs
+    value: ${{ jq in.[]}}
+  - uses: SimArgs
+    with:
+    - append-list
+
+So... Tasks do not have parameters. 
+- Tasks can inject parameter sets into the input set
+- Tasks can control what emerges from the task
+
+plusargs: .[] select(uses==SimArgs) | 
+
+# Cross-Task Dataflow
+- Cross-task dataflow is comprised of a collection of typed objects
+  containing typed fields.
+- Augmenting data is done by injecting new object instances into
+  the dataset. For example:
+  - The dataset contains an object for specifying simulation runtime
+    arguments. It contains a directive to append "+UVM_TEST=my_test" 
+    to the arguments
+  - I add a new runtime-arguments object that contains a directive
+    to append "+debug" to the arguments
+  - The consumer will see [+UVM_TEST=my_test +debug]
+
+
+# Data-Combination Rules
+- In most cases, data must be flattened in order to use it. For example,
+  
+- 

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/docs/Roadmap.md

@@ -123,4 +123,9 @@ are evaluated.
 - Dependency strategy for Exec/Make
   - Specify dependency file created by task
   - When the timestamp is updated, task is known to have been rebuilt
-- 
+- 
+
+- Allow each task to emit a series of markers as part of the result
+  - Enables users to easily click on first N warnings/errors in each step
+    without needing to find and open a logfile
+- Allow files to be attached as part of the result

dv_flow_mgr-0.0.1.13335470340a1/docs/Steps.md

@@ -0,0 +1,50 @@
+
+# Tasks and Types
+- Load type definitions from YAML
+- Load task definitions from YAML
+
+# Data Selection and Extraction
+- Sort and select data
+- Need to support expressions
+- 
+
+# Task Implementation Source/Result Data
+
+Source Data
+- Task parameters
+- Upstream-change indication
+- Memento
+
+Result Data
+- List of output parameter sets
+- Change indication
+- Memento
+
+# Data Combination
+-
+
+# Day in the Life
+- Task Runner receives outputs from dependent tasks
+- ORs input changed' input flags to determine 'changed' flag to pass to task
+- Task Runner orders parameters using dependency information
+- Task Runner evaluates task-parameter creation code. Uses input data in this process
+  - Creates object of appropriate type
+  - Evalutes base-up to assign and augment parameter values
+- Retrieves memento (if available)
+- Passes accumulated data to task
+  - changed
+  - parameters
+  - 
+- Receives output from task
+  - list of parameter sets
+  - changed
+  - memento
+  - list of markers
+- Saves memento and markers for later inspection (central dir?)
+- Sets 'self' as source of parameter sets
+- Forms output data from
+  - changed
+  - list of parameter sets
+
+# Creating Task Parameters
+- 

dv_flow_mgr-0.0.1.13335470340a1/docs/Usecases.md

@@ -0,0 +1,49 @@
+
+# Simple Initial Project Setup
+
+
+# Reconfigure Project Build
+
+# Global Configuration Data
+Some configuration data is global in nature -- in other words,
+it is relevant to multiple tools. Debug is one example. Enabling
+a debug flow impacts multiple simulation tools, and may impact
+how user-specified input behaves as well (eg adding +UVM_VERBOSITY)
+
+# Configure Package Usage
+I have a package that depends on UVM, as well as other packages (UVCs)
+that depend on UVM. By default, the UVM library is provided as a
+pre-compiled library (n simulators that support it). For "reasons",
+I want to switch to using the pure-source version. My choice must
+apply to all packages within my orbit -- in other words, having
+my package depend on source UVM while the UVC depends on pre-compiled
+will result in a conflict.
+
+# Configure Project Toolchain
+My package has external dependencies that may specify processing
+in addition to sources (eg the UVM library specifies how to 
+precompile it). I need a way to configure the toolchain that the
+external packages use the toolchain defined by my package.
+
+# Inline Specification of Inputs
+The simulation tool receives its inputs and controls via task data.
+This provides flexibility, by allowing multiple tasks to contribute 
+command-line arguments. While I could create a new task to output
+data that would add a command-line argument, I'd prefer to have
+a more-direct way to do so -- a sort of inline injection of task
+data that is combined with the task data of dependencies prior to
+being presented to the implementation.
+
+
+# Static Composition of Task Graphs (Introspection?)
+I have a collection of test definitions. I want to take a selection
+of them, connect my preferred simulation image, and run them.
+
+
+# Static Composition of Task Graphs (Introspection)
+I have collections of test definitions. I want to select tests that
+have a specific attribute / tag value, connect my preferred
+simulation image, and run them.
+
+
+

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/ivpm.yaml

@@ -25,5 +25,7 @@ package:
       src: pypi
     - name: sphinxcontrib-mermaid
       src: pypi
+    - name: jq
+      src: pypi
 
 

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "dv-flow-mgr"
-version = "0.0.1.12992722023a1"
+version = "0.0.1.13335470340a1"
 dependencies = [
     'pydantic',
     'pyyaml',

dv_flow_mgr-0.0.1.13335470340a1/src/dv_flow/mgr/expr_parser.py

@@ -0,0 +1,38 @@
+import dataclasses as dc
+import enum
+
+class TokenKind(enum.Enum):
+    ID = enum.auto()
+
+@dc.dataclass
+class Expr(object):
+    kind : TokenKind = TokenKind.ID
+
+class ExprId(Expr):
+    id : str
+
+    def __post_init__(self):
+        self.kind = TokenKind.ID
+
+class ExprPipe(Expr):
+    lhs : Expr
+    rhs : Expr
+
+    def __post_init__(self):
+        self.kind = TokenKind.ID
+
+
+
+class ExprParser(object):
+
+    def __init__(self, input):
+        self.input = input
+
+    def __iter__(self):
+        return self
+    
+    def __next__(self):
+
+        self.expr = expr
+        self.tokens = []
+        self.pos = 0

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/src/dv_flow/mgr/fragment_def.py

@@ -26,11 +26,13 @@ from typing import Any, Dict, List, Union
 from .package import Package
 from .package_import_spec import PackageImportSpec
 from .task_def import TaskDef
+from .type_def import TypeDef
 
 class FragmentDef(BaseModel):
     tasks : List[TaskDef] = dc.Field(default_factory=list)
     imports : List[Union[str,PackageImportSpec]] = dc.Field(default_factory=list, alias="imports")
     fragments: List[str] = dc.Field(default_factory=list)
+    types : List[TypeDef] = dc.Field(default_factory=list)
 
     basedir : str = None
 

dv_flow_mgr-0.0.1.13335470340a1/src/dv_flow/mgr/out

@@ -0,0 +1,12 @@
+	   __main__.py
+	    flow.py
+	   package.py
+	   package_def.py
+	   package_import_spec.py
+	    parameters.py
+	   pkg_rgy.py
+	   task.py
+	   task_graph_builder.py
+	   task_graph_runner.py
+	   util.py
+

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/src/dv_flow/mgr/package.py

@@ -31,6 +31,7 @@ class Package(object):
     # Package holds constructors for tasks
     # - Dict holds the default parameters for the task
     tasks : Dict[str,TaskCtor] = dc.field(default_factory=dict)
+    types : Dict[str,Any] = dc.field(default_factory=dict)
     _log : ClassVar = logging.getLogger("Package")
 
     def getTaskCtor(self, name : str) -> TaskCtor:

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/src/dv_flow/mgr/package_def.py

@@ -19,6 +19,7 @@
 #*     Author: 
 #*
 #****************************************************************************
+import io
 import os
 import json
 import yaml
@@ -35,6 +36,7 @@ from .package_import_spec import PackageImportSpec, PackageSpec
 from .task import TaskCtor, TaskCtorProxy, TaskCtorCls, TaskCtorParam, TaskCtorParamCls
 from .task_def import TaskDef, TaskSpec
 from .std.task_null import TaskNull
+from .type_def import TypeDef
 
 
 class PackageDef(BaseModel):
@@ -44,6 +46,7 @@ class PackageDef(BaseModel):
     tasks : List[TaskDef] = dc.Field(default_factory=list)
     imports : List[PackageImportSpec] = dc.Field(default_factory=list)
     fragments: List[str] = dc.Field(default_factory=list)
+    types : List[TypeDef] = dc.Field(default_factory=list)
 
     fragment_l : List['FragmentDef'] = dc.Field(default_factory=list, exclude=True)
 
@@ -314,6 +317,32 @@ class PackageDef(BaseModel):
         file_s.pop()
 
         return pkg
+
+    @staticmethod
+    def loads(data, exp_pkg_name=None):
+        return PackageDef._loadPkgDefS(data, exp_pkg_name)
+        pass
+
+    @staticmethod
+    def _loadPkgDefS(data, exp_pkg_name):
+        ret = None
+        doc = yaml.load(io.StringIO(data), Loader=yaml.FullLoader)
+        if "package" not in doc.keys():
+            raise Exception("Missing 'package' key in %s" % root)
+        pkg = PackageDef(**(doc["package"]))
+        pkg.basedir = None
+
+#            for t in pkg.tasks:
+#                t.basedir = os.path.dirname(root)
+
+        if exp_pkg_name is not None:
+            if exp_pkg_name != pkg.name:
+                raise Exception("Package name mismatch: %s != %s" % (exp_pkg_name, pkg.name))
+
+        if len(pkg.fragments) > 0:
+            raise Exception("Cannot load a package-def with fragments from a string")
+
+        return pkg
     
     @staticmethod
     def _loadFragmentSpec(pkg, spec, file_s):

dv_flow_mgr-0.0.1.13335470340a1/src/dv_flow/mgr/param_def.py

@@ -0,0 +1,25 @@
+
+from typing import Any, List, Union
+from pydantic import BaseModel, Field
+
+class ListType(BaseModel):
+    item : Union[str, 'ComplexType']
+
+class MapType(BaseModel):
+    key : Union[str, 'ComplexType']
+    item : Union[str, 'ComplexType']
+
+class ComplexType(BaseModel):
+    list : Union[ListType, None] = None
+    map : Union[MapType, None] = None
+
+class ParamDef(BaseModel):
+    name : str
+    doc : str = None
+    type : Union[str, 'ComplexType'] = None
+    value : Union[Any, None] = None
+    append : Union[Any, None] = None
+    prepend : Union[Any, None] = None
+    path_append : Union[Any, None] = Field(alias="path-append", default=None)
+    path_prepend : Union[Any, None] = Field(alias="path-prepend", default=None)
+

dv_flow_mgr-0.0.1.13335470340a1/src/dv_flow/mgr/std/flow.dv

@@ -0,0 +1,78 @@
+
+package:
+  name: std
+
+  tasks:
+  - name: Message
+    pyclass: dv_flow.mgr.std.message.Message
+    with:
+      msg:
+        type: str
+        value: ""
+  - name: FileSet
+    pyclass: dv_flow.mgr.std.fileset.FileSet
+    with:
+      base:
+        type: str
+        value: ""
+      type:
+        type: str
+        value: ""
+      include:
+        type: str
+        value: ""
+      exclude:
+        type: str
+        value: ""
+#  - name: Exec
+#    pyclass: dv_flow.mgr.std.exec.Exec
+#    with: {}
+  types:
+  # - name: TaskDataItem
+  #   doc: |
+  #     Base type for a data element produced by a task.
+  #     TaskDataItem is never used directly
+  #   with:
+  #   - name: kind
+  #     type: str
+  #     kind: metadata
+  #     doc: |
+  #       Kind of data item
+  #   - name: src
+  #     type: str
+  #     kind: metadata
+  #     doc: |
+  #       Name of the task that produced this item
+  #   - name: id
+  #     type: str
+  #     kind: metadata
+  #     doc: |
+  #       Unique identiifer 
+
+  - name: Env
+    doc: |
+      Environment variables
+    with:
+    - name: doc
+
+  - name: FileSet
+    uses: TaskDataItem
+    with:
+    - name: kind
+      value: "FileSet"
+    - name: base
+      type: str
+    - name: fileType
+      type: str
+    - name: files
+      type: 
+        list:
+          item:
+            type: str
+
+    # type:
+    #   list: (str,int)
+    #     - complex type
+    # type:
+    #   object:
+    #     

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/src/dv_flow/mgr/task_data.py

@@ -48,26 +48,22 @@ class TaskDataParam(BaseModel):
     kind : TaskDataParamKindE
     ops : List[TaskDataParamOp] = dc.Field(default_factory=list)
 
+class TaskDataItem(BaseModel):
+    type : str
+    src : str
+    id : str
+
 class TaskData(BaseModel):
     src : str = None
-    params : Dict[str,TaskDataParam] = dc.Field(default_factory=dict)
     deps : Dict[str,Set[str]] = dc.Field(default_factory=dict)
-    filesets : List[FileSet] = dc.Field(default_factory=list)
     changed : bool = False
+    data : List[TaskDataItem] = dc.Field(default_factory=list)
 
     _log : ClassVar = logging.getLogger("TaskData")
 
-    def hasParam(self, name: str) -> bool:
-        return name in self.params
-    
-    def getParam(self, name: str) -> Any:
-        return self.params[name]
-    
-    def setParam(self, name: str, value: Any):
-        self.params[name] = value
-
-    def addFileSet(self, fs : FileSet):
-        self.filesets.append(fs)
+    def addData(self, item : TaskDataItem):
+        item.src = self.src
+        self.data.append(item)
 
     def getFileSets(self, type=None, order=True) -> List[FileSet]:
         ret = []
@@ -77,8 +73,10 @@ class TaskData(BaseModel):
         if order:
             # The deps map specifies task dependencies
 
+            filesets = filter(lambda x: x.type == "std.FileSet", self.data)
+
             candidate_fs = []
-            for fs in self.filesets:
+            for fs in filesets:
                 self._log.debug("fs: %s" % str(fs))
                 if type is None or fs.type in type:
                     candidate_fs.append(fs)

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/src/dv_flow/mgr/task_def.py

@@ -22,7 +22,9 @@
 import pydantic.dataclasses as dc
 from pydantic import BaseModel
 from typing import Any, Dict, List, Union, Tuple
+from .param_def import ParamDef
 from .task import Task
+from .task_output import TaskOutput
 
 @dc.dataclass
 class TaskSpec(object):
@@ -37,7 +39,8 @@ class TaskDef(BaseModel):
     desc : str = dc.Field(default="")
     doc : str = dc.Field(default="")
     depends : List[Union[str,TaskSpec]] = dc.Field(default_factory=list, alias="needs")
-    params: Dict[str,Any] = dc.Field(default_factory=dict, alias="with")
+    params: List[ParamDef] = dc.Field(default_factory=list, alias="with")
+    out: List[TaskOutput] = dc.Field(default_factory=list)
 
     def copy(self) -> 'TaskDef':
         ret = TaskDef(

dv_flow_mgr-0.0.1.13335470340a1/src/dv_flow/mgr/task_impl_data.py

@@ -0,0 +1,17 @@
+
+from pydantic import BaseModel
+from typing import Any, ClassVar, Dict, Set, List, Tuple
+
+class TaskImplParams(BaseModel):
+    pass
+
+class TaskImplSourceData(BaseModel):
+    params : Any
+    changed : bool
+    memento : Any
+
+class TaskImplResultData(BaseModel):
+    data : List[Any]
+    changed : bool
+    memento : Any
+

dv_flow_mgr-0.0.1.13335470340a1/src/dv_flow/mgr/task_output.py

@@ -0,0 +1,6 @@
+from pydantic import BaseModel, Field
+from typing import Any, List
+
+class TaskOutput(BaseModel):
+    type : str
+    params : List[Any] = Field(default_factory=list, alias="with")

dv_flow_mgr-0.0.1.13335470340a1/src/dv_flow/mgr/type_def.py

@@ -0,0 +1,10 @@
+
+from typing import List, Union
+from pydantic import BaseModel, Field
+from .param_def import ParamDef
+
+class TypeDef(BaseModel):
+    name : str
+    uses : str = None
+    doc : str = None
+    fields : List[ParamDef] = Field(alias="with", default_factory=list)

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/src/dv_flow_mgr.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: dv-flow-mgr
-Version: 0.0.1.12992722023a1
+Version: 0.0.1.13335470340a1
 Summary: DV Flow Manager is a build system for silicon design
 Author-email: Matthew Ballance <matt.ballance@gmail.com>
 License:                                  Apache License

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/src/dv_flow_mgr.egg-info/SOURCES.txt

@@ -5,11 +5,14 @@ ivpm.yaml
 pyproject.toml
 .github/workflows/ci.yml
 .vscode/settings.json
+docs/KeyArchitecture.md
 docs/Makefile
 docs/Notes.md
 docs/Roadmap.md
 docs/Stages.md
+docs/Steps.md
 docs/TypesAndDefs.md
+docs/Usecases.md
 docs/conf.py
 docs/index.rst
 docs/intro.rst
@@ -17,11 +20,14 @@ docs/quickstart.rst
 docs/reference.rst
 src/dv_flow/mgr/__init__.py
 src/dv_flow/mgr/__main__.py
+src/dv_flow/mgr/expr_parser.py
 src/dv_flow/mgr/fileset.py
 src/dv_flow/mgr/fragment_def.py
+src/dv_flow/mgr/out
 src/dv_flow/mgr/package.py
 src/dv_flow/mgr/package_def.py
 src/dv_flow/mgr/package_import_spec.py
+src/dv_flow/mgr/param_def.py
 src/dv_flow/mgr/pkg_rgy.py
 src/dv_flow/mgr/task.py
 src/dv_flow/mgr/task_data.py
@@ -29,7 +35,11 @@ src/dv_flow/mgr/task_def.py
 src/dv_flow/mgr/task_graph_builder.py
 src/dv_flow/mgr/task_graph_runner.py
 src/dv_flow/mgr/task_graph_runner_local.py
+src/dv_flow/mgr/task_impl_data.py
 src/dv_flow/mgr/task_memento.py
+src/dv_flow/mgr/task_output.py
+src/dv_flow/mgr/type.py
+src/dv_flow/mgr/type_def.py
 src/dv_flow/mgr/util.py
 src/dv_flow/mgr/cmds/cmd_run.py
 src/dv_flow/mgr/share/flow.json
@@ -50,6 +60,7 @@ tests/system/test_stdlib.py
 tests/unit/__init__.py
 tests/unit/test_data_merge.py
 tests/unit/test_fileset.py
+tests/unit/test_parse.py
 tests/unit/test_pyclass.py
 tests/unit/test_smoke copy.sav
 tests/unit/test_smoke.py

dv_flow_mgr-0.0.1.13335470340a1/tests/unit/test_parse.py

@@ -0,0 +1,38 @@
+import io
+import pytest
+import yaml
+
+from dv_flow.mgr.type_def import TypeDef
+
+def test_typedef_1():
+    flow_dv = """
+package:
+  name: abc
+  types:
+  - name: T1
+    with:
+    - name: F1
+      doc: "Field 1"
+      type: str
+    - name: F2
+      type:
+        list:
+          item: str
+    - name: F3
+      type:
+        map:
+          key: str
+          item:
+            list:
+              item: str
+"""
+
+    data = yaml.load(io.StringIO(flow_dv), yaml.FullLoader)
+
+    typedef = data["package"]["types"][0]
+    td = TypeDef(**typedef)
+
+    print("td=" + str(td))
+
+
+

{dv_flow_mgr-0.0.1.12992722023a1 → dv_flow_mgr-0.0.1.13335470340a1}/tests/unit/test_smoke.py

@@ -3,12 +3,66 @@ import io
 import os
 import dataclasses as dc
 import pytest
+import jq
 from typing import List
 import yaml
-from dv_flow.mgr import FileSet, PackageDef, TaskData
+from dv_flow.mgr import FileSet, PackageDef, TaskData, TaskGraphBuilder
 from pydantic import BaseModel
 from shutil import copytree
 
+def test_smoke_1():
+    flowdv = """
+package:
+    name: my_pkg
+
+    tasks:
+    - name: entry
+      with:
+      - name: p1
+        type: int
+      - name: p2
+        type: int
+"""
+
+    pkg_def = PackageDef.loads(flowdv)
+    builder = TaskGraphBuilder(pkg_def, os.getcwd())
+    task = builder.mkTaskGraph("my_pkg.entry")
+
+def test_jq():
+    data = [
+        {
+            "type": "FileSet",
+            "kind": "systemVerilogSource"
+        },
+        {
+            "type": "FileSet",
+            "kind": "vhdlSource"
+        },
+        {
+            "type": "FileSet",
+            "kind": "verilogSource"
+        }
+    ]
+
+    result = jq.compile(""".[] | 
+                        select(
+                            .kind == "systemVerilogSource" 
+                            or .kind == "verilogSource")
+                        """).input(data).all()
+    value = """
+    with:
+    - name: files
+      list-append: |
+      ${{ 
+      in | jq('.[] | select(
+        .kind == "systemVerilogSource" 
+        or .kind == "verilogSource")')
+      }}
+"""
+    print("result: %s" % str(result))
+
+
+
 # def test_smoke():
 #     file = """
 # package:

dv_flow_mgr-0.0.1.12992722023a1/src/dv_flow/mgr/std/flow.dv

@@ -1,30 +0,0 @@
-
-package:
-  name: std
-
-  tasks:
-  - name: Message
-    pyclass: dv_flow.mgr.std.message.Message
-    with:
-      msg:
-        type: str
-        value: ""
-  - name: FileSet
-    pyclass: dv_flow.mgr.std.fileset.FileSet
-    with:
-      base:
-        type: str
-        value: ""
-      type:
-        type: str
-        value: ""
-      include:
-        type: str
-        value: ""
-      exclude:
-        type: str
-        value: ""
-#  - name: Exec
-#    pyclass: dv_flow.mgr.std.exec.Exec
-#    with: {}
-