dftracer 2.2.0.post8.dev0__tar.gz → 2.2.0.post10.dev0__tar.gz

{dftracer-2.2.0.post8.dev0/python/dftracer.egg-info → dftracer-2.2.0.post10.dev0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dftracer
-Version: 2.2.0.post8.dev0
+Version: 2.2.0.post10.dev0
 Summary: I/O profiler for deep learning python apps. Specifically for dlio_benchmark.
 Author-email: "Hariharan Devarajan (Hari)" <hariharandev1@llnl.gov>
 License-Expression: MIT

dftracer-2.2.0.post10.dev0/docs/provenance.rst

@@ -0,0 +1,367 @@
+================================
+Entities, Relations & Provenance
+================================
+
+DFTracer can record **what data an application works with and how each piece
+of data came to be**, alongside its usual timing and I/O events. The same API
+serves provenance (lineage of scientific artifacts) and performance analysis
+(which events touched which data instances).
+
+.. contents::
+   :local:
+   :depth: 2
+
+----------------------------------------
+What it captures
+----------------------------------------
+
+**Entity**
+   A typed instance of something the application works with: an input file, a
+   dataset shard, a checkpoint, a model, a predicted protein structure, a GPU
+   buffer. Identified by ``(type, key)``: the *type* is an application-defined
+   class name, the *key* is any application identifier for the instance (a
+   path, a sample id, ``step=120``). The key is hashed into a 64-bit
+   ``EntityID``; it is never stored, so keys may be long or sensitive.
+
+**Entity type**
+   Described once per process with a **role** (input, output, intermediate,
+   parameter, reference) and a short description of what it represents.
+   Roles let users ask "what went in", "what came out", "what happened
+   between".
+
+**Event relation**
+   An event (a traced function or region) relates to entities: it **used**
+   (consumed), **generated** (produced), **invalidated** (deleted or ended) or
+   **updated** (modified in place) them. Any event may carry relations, in any
+   category, such as a C++ function, a Python region or an MPI call.
+
+**Entity relation**
+   Two entities relate directly: **derived_from**, **revision_of**,
+   **contains** (has-a), **part_of** (is-in), **specialization_of**
+   (inherits / is-a), **alternate_of**, **depends_on**.
+
+Together these form a graph: entities and events as nodes, relations as edges.
+Walking ``generated`` edges backwards from an artifact gives its lineage;
+walking ``used`` edges forwards gives the impact of an input.
+
+Design rules
+   * Enums wherever possible; strings only where unavoidable, and then
+     fixed-length and JSON-safe.
+   * Identity is computed by the core (FNV-1a-64 of ``type`` 0x1f ``key``), so
+     the same ``(type, key)`` gets the same ``EntityID`` in C, C++ and Python,
+     in every process and on every node.
+   * Each entity and each entity type is recorded **once per process**; later
+     declarations cost a set lookup. The first declaration's store and uri win.
+   * Relations are the event's content and are written even when
+     ``DFTRACER_INC_METADATA`` is off (other optional event args still need it).
+
+----------------------------------------
+Types and enums
+----------------------------------------
+
+Defined in ``dftracer/core/common/entity.h`` (C-compatible) and
+``dftracer/core/common/typedef.h``.
+
+=========================== ==========================================================================
+Type                        Meaning
+=========================== ==========================================================================
+``EntityID``                ``uint64_t``; 16 lowercase hex characters in traces
+``ConstEntityTypeName``     type name, at most ``DFT_ENTITY_TYPE_LEN - 1`` (31)
+``ConstEntityKey``          instance key, hashed into the id, not stored
+``ConstEntityURI``          locator unique within the app, at most 255 characters
+``ConstEntityDescription``  type description, at most 255 characters
+``EntityStore``             where the instance lives (enum below)
+``EntityRole``              role of an entity type (enum below)
+``EntityRelation``          event and entity relations (enum below)
+=========================== ==========================================================================
+
+Strings are truncated to their capacity and restricted to ``A-Z a-z 0-9 _ - .
+: / @ + = , ~`` and space; any other character (quotes, backslash, ``|``,
+control characters) becomes ``_``.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Enum
+     - Values (as written in traces)
+   * - ``EntityStore``
+     - 0 memory, 1 gpu_memory, 2 local_disk, 3 parallel_fs, 4 burst_buffer,
+       5 object_store, 6 database, 7 network, 8 other
+   * - ``EntityRole``
+     - 0 unknown, 1 input, 2 output, 3 intermediate, 4 parameter, 5 reference
+   * - ``EntityRelation`` (event)
+     - 0 used, 1 generated, 2 invalidated, 3 updated
+   * - ``EntityRelation`` (entity)
+     - 16 derived_from, 17 revision_of, 18 contains, 19 part_of,
+       20 specialization_of, 21 alternate_of, 22 depends_on
+
+Enum values are part of the trace format and are append-only.
+
+----------------------------------------
+C API
+----------------------------------------
+
+.. code-block:: c
+
+    #include <dftracer/dftracer.h>
+
+    DFTRACER_C_ENTITY_TYPE("sequence", DFT_ROLE_INPUT, "A query protein sequence");
+
+    void fold(const char* seq_id, const char* out_path) {
+      DFTRACER_C_FUNCTION_START();
+      DFTRACER_C_FUNCTION_USES("sequence", seq_id);            /* store: memory */
+      EntityID s = update_relation_entity(data_fn, DFT_REL_GENERATED,
+                                          "protein_structure", seq_id,
+                                          DFT_STORE_PARALLEL_FS, out_path);
+      DFTRACER_C_FUNCTION_END();
+    }
+
+    EntityID tar = DFTRACER_C_ENTITY("structure_tar", seq_id,
+                                     DFT_STORE_PARALLEL_FS, tar_path);
+    DFTRACER_C_ENTITY_RELATE(DFT_REL_CONTAINS, tar, s);
+
+Functions: ``dftracer_declare_entity``, ``dftracer_declare_entity_type``,
+``dftracer_relate_entities``, ``update_relation``, ``update_relation_entity``.
+Macros: ``DFTRACER_C_ENTITY``, ``DFTRACER_C_ENTITY_TYPE``,
+``DFTRACER_C_ENTITY_RELATE``, ``DFTRACER_C_FUNCTION_RELATE``,
+``DFTRACER_C_FUNCTION_USES``, ``DFTRACER_C_FUNCTION_GENERATES``,
+``DFTRACER_C_REGION_RELATE``.
+
+----------------------------------------
+C++ API
+----------------------------------------
+
+.. code-block:: cpp
+
+    #include <dftracer/dftracer.h>
+
+    void relax(const std::string& seq, const std::string& path) {
+      DFTRACER_CPP_FUNCTION();
+      EntityID raw = profiler_dft_fn.uses("unrelaxed_structure", seq.c_str());
+      EntityID out = profiler_dft_fn.generates("relaxed_structure", seq.c_str(),
+                                               DFT_STORE_BURST_BUFFER, path.c_str());
+      DFTRACER_CPP_ENTITY_RELATE(DFT_REL_REVISION_OF, out, raw);
+    }
+
+``DFTracer`` gains ``relate(EntityRelation, EntityID)``,
+``relate(EntityRelation, type, key, store, uri)``, ``uses(...)`` and
+``generates(...)``; macros ``DFTRACER_CPP_ENTITY``, ``DFTRACER_CPP_ENTITY_TYPE``,
+``DFTRACER_CPP_ENTITY_RELATE``, ``DFTRACER_CPP_FUNCTION_RELATE``,
+``DFTRACER_CPP_FUNCTION_USES``, ``DFTRACER_CPP_FUNCTION_GENERATES``,
+``DFTRACER_CPP_REGION_RELATE``.
+
+----------------------------------------
+Python API
+----------------------------------------
+
+.. code-block:: python
+
+    from dftracer.python import (dftracer, dft_fn, EntityStore, EntityRole,
+                                 EntityRelation)
+
+    log = dftracer.initialize_log(logfile=None, data_dir=None, process_id=-1)
+    log.declare_entity_type("filtered_msa", EntityRole.OUTPUT,
+                            "Diversity-filtered multiple sequence alignment")
+
+    with dft_fn("APP", name="filter_msa") as f:
+        merged = f.uses("merged_msa", seq_id, EntityStore.BURST_BUFFER, merged_path)
+        filtered = f.generates("filtered_msa", seq_id, EntityStore.BURST_BUFFER, out_path)
+        f.relate_entity(EntityRelation.INVALIDATED, "merged_msa", seq_id)  # deleted
+
+    log.relate_entities(EntityRelation.DERIVED_FROM, filtered, merged)
+
+``dftracer`` gains ``declare_entity`` (returns the id as an ``int``, 0 when
+tracing is off), ``declare_entity_type`` and ``relate_entities``; ``dft_fn``
+gains ``uses``, ``generates``, ``relate_entity`` and ``relate``. The native
+module also exposes ``entity_hash(type, key)`` and ``entity_hex(id)`` to compute
+ids without tracing.
+
+----------------------------------------
+Trace records
+----------------------------------------
+
+Entities, entity types and entity relations are metadata records
+(``"ph":4``, ``"cat":"dftracer"``, ``"type":1``) with named fields:
+
+.. code-block:: json
+
+    {"name":"EH","cat":"dftracer","type":1,"pid":1,"tid":1,"ph":4,
+     "args":{"hhash":"…","id":"8f0de68c04eea0bf","type":"protein_structure",
+             "store":3,"uri":"/p/out/a.pdb"}}
+    {"name":"ET","cat":"dftracer","type":1,"pid":1,"tid":1,"ph":4,
+     "args":{"hhash":"…","type":"protein_structure","role":2,
+             "description":"Predicted 3D structure"}}
+    {"name":"ER","cat":"dftracer","type":1,"pid":1,"tid":1,"ph":4,
+     "args":{"hhash":"…","relation":18,"subject":"6e6d650dd131e83b",
+             "object":"8f0de68c04eea0bf"}}
+
+An event's relations are one ``relations`` object in its ``args``: relation
+name to an array of entity ids. Nesting them keeps relation names from
+colliding with other event args (the lifecycle ``end`` event has its own
+``used`` object, for example):
+
+.. code-block:: json
+
+    {"id":2,"name":"fold","cat":"CPP_APP","type":9,"pid":1,"tid":1,
+     "ts":1790877632269434,"dur":32,"ph":1,
+     "args":{"hhash":"…","relations":{"used":["d824b0bb91487885"],
+                                       "generated":["8f0de68c04eea0bf"]}}}
+
+Every record is one JSON line.
+The ``hhash`` field names the host, so records from different nodes stay apart.
+An ``EH`` record says what an id stands for.
+An ``ET`` record says what a type means and which role it plays.
+An ``ER`` record is one edge between two entities, with the ``relation`` code from the table above.
+An event record says which entities it used, generated, invalidated or updated.
+
+Each process writes its own ``EH`` and ``ET`` records, once per entity and once per type.
+A trace directory therefore holds the same entity many times, once per process that touched it.
+Readers keep the first copy and drop the rest.
+The ``store`` is a number from the ``EntityStore`` table and ``uri`` is empty when the application gave none.
+Event relations hold only ids.
+The entity behind an id is the ``EH`` record with the same ``id``, which may sit in another file.
+
+A reader joins events to entities by id; ids from different processes and
+languages join without coordination. ``dftracer-utils`` builds the graph
+(``trace::provenance::extract_provenance_graph``, ``GET /api/prov/graph`` and
+the *Provenance* tab of ``dftracer_server``).
+
+----------------------------------------
+Reading a trace
+----------------------------------------
+
+The traces of one run sit in a directory with one ``.pfw.gz`` file per process.
+Read them with the ``dftracer-utils`` dataframe API, which scans all files in one pass and joins them by id.
+
+.. code-block:: python
+
+    from dftracer.utils import TraceViewer, col
+
+    view = TraceViewer("traces/", "/tmp/traces-index").phase("any")
+
+    # One scan feeds both reads. Entity records are metadata, so the
+    # viewer must read every record family.
+    with view.session() as s:
+        eh = s.collect(view.filter('name == "EH"'))
+        prov = s.collect(view.filter('cat == "PROV"'))
+
+    entities = (
+        eh.result()
+        .select("args.id", "args.type")
+        .rename({"args.id": "entity", "args.type": "type"})
+        .distinct("entity")
+    )
+
+    # One row per (event, relation, entity), labeled with the entity type.
+    events = prov.result()
+    edges = (
+        events.unpivot(
+            id_vars=["name", "pid", "ts"],
+            value_vars=[c for c in events.columns if c.startswith("args.relations.")],
+        )
+        .drop_nulls()
+        .join(entities, left_on="value", right_on="entity")
+    )
+    edges = edges.assign(relation=edges["variable"].str.extract(r"relations\.(\w+)\."))
+
+    made = edges.filter(col("relation") == "generated")
+    print(made.group_by(["name", "type"]).agg("count:pid").sort(["name", "type"]).to_pandas())
+
+The second argument of ``TraceViewer`` is where the index goes.
+Without it the viewer writes a ``.dftindex`` directory next to the traces, which is not what you want in a shared trace directory.
+
+Two details of the dataframe shape the code.
+Metadata records have ``ph`` 4 and are left out of a default query, so ``phase("any")`` is required to see ``EH``, ``ET`` and ``ER``.
+The session registers both reads first and runs them together, so the trace is decompressed and parsed once.
+Nested fields become columns named by their path, and an array becomes one column per position, such as ``args.relations.used.0`` and ``args.relations.used.1``.
+The ``unpivot`` call turns those columns back into one row per relation, and ``distinct`` keeps one ``EH`` record for each entity, since every process writes its own copy.
+
+On a protein structure prediction run of 10 sequences the last line prints which step made which kind of entity, here cut to the first rows:
+
+.. code-block:: text
+
+                      name           type  count_pid
+    0           a3m_to_sto  stockholm_msa         10
+    1           filter_msa   filtered_msa         10
+    2  load_and_merge_msas     merged_msa         10
+    3          load_inputs       sequence         10
+    4          msa_featgen   msa_features         10
+    ...
+
+The same query in duql is one text with no dataframe code:
+
+.. code-block:: python
+
+    from dftracer.utils import TraceViewer
+
+    q = """
+    let entities = from all
+      | where name == "EH"
+      | select entity = args.id, etype = args.type
+      | distinct;
+    where cat == "PROV"
+    | unpivot args.relations.*.* as variable, entity
+    | where entity is not null
+    | derive etype = entity -> entities.etype
+    | derive relation = extract(variable, "relations[.]([a-z]+)[.]", 1)
+    | where relation == "generated"
+    | group name, etype { n = count() }
+    """
+    print(TraceViewer("traces/", "/tmp/traces-index").duql(q).collect().to_pandas())
+
+The ``let`` names a row set of the entity records, and ``from all`` reads them although they are metadata, so the viewer needs no ``phase("any")``.
+The ``*.*`` pattern in ``unpivot`` expands to every ``args.relations.<name>.<position>`` field in the trace, which does the job of the column list in the dataframe code.
+``entity -> entities.etype`` looks up the type of each entity id in the row set.
+The result is the same 16 rows, one for each step and entity type.
+
+The ``edges`` table answers the common questions with a filter.
+``edges.filter(col("relation") == "used")`` lists what each step consumed, and ``"invalidated"`` lists what each step deleted.
+Filtering on ``col("value")`` with one id gives every event that touched that entity, whichever process wrote it.
+For a ``revision_of`` or ``contains`` edge, read the ``ER`` records the same way, with ``name == "ER"`` and the columns ``args.relation``, ``args.subject`` and ``args.object``.
+
+----------------------------------------
+Walking the graph
+----------------------------------------
+
+The lineage of an entity is every entity that the steps before it used, and its impact is every entity that the steps after it made.
+Both are a walk over the ``edges`` table above, one join per hop.
+
+.. code-block:: python
+
+    from dftracer.utils import DataFrame
+
+    ACT = ["pid", "ts"]  # an activity has no id; its process and start time name it
+    generated = edges.filter(col("relation") == "generated").select(*ACT, "value")
+    used = edges.filter(col("relation") == "used").select(*ACT, "value")
+
+
+    def walk(start, backward=True):
+        into, out = (generated, used) if backward else (used, generated)
+        seen = DataFrame.from_dict({"value": [start]})
+        frontier = seen
+        while len(frontier):
+            steps = frontier.join(into, on="value").select(*ACT).distinct()
+            reached = steps.join(out, on=ACT).select("value").distinct()
+            frontier = reached.join(seen, on="value", how="anti")
+            seen = seen.concat(frontier)
+        return seen.join(entities, left_on="value", right_on="entity")
+
+
+    best = entities.filter(col("type") == "best_structure").sort("entity")["entity"].to_list()[0]
+    lineage = walk(best)
+    model = entities.filter(col("type") == "model_params")["entity"].to_list()[0]
+    impact = walk(model, backward=False)
+    print(len(lineage), len(impact.filter(col("type") == "best_structure")))
+
+.. code-block:: text
+
+    28 10
+
+A backward hop finds the activities that generated the entities in hand and then the entities those activities used.
+A forward hop goes the other way, and the anti join drops the entities already seen, so the walk ends when a hop finds nothing new.
+On the run above, each best structure comes from 28 entities, from the input chunk and the sequence to the alignments, the reference databases, both model weights and the seed.
+Each set of model weights reaches all 10 best structures, since every prediction runs both models and picks the better one.
+
+An activity record has no id field, so the walk names an activity by its ``pid`` and ``ts``.
+That is unique as long as one process does not start two activities in the same microsecond.
+The ``ER`` edges, such as ``revision_of`` and ``contains``, are not part of the walk, since no activity reads or writes through them.

{dftracer-2.2.0.post8.dev0 → dftracer-2.2.0.post10.dev0}/examples/dfanalyzer/dfanalyzer-ideas.ipynb

@@ -1,4 +1,4 @@
 [diffend] Oversized file quarantined before diffing.
-name: dftracer-2.2.0.post8.dev0/examples/dfanalyzer/dfanalyzer-ideas.ipynb
+name: dftracer-2.2.0.post10.dev0/examples/dfanalyzer/dfanalyzer-ideas.ipynb
 size: 52418033 bytes
 sha256: e4b283c4e2e7dd17e9f5452b2fb101d1d9f32d4dd13fa56742629faea6700b55

{dftracer-2.2.0.post8.dev0 → dftracer-2.2.0.post10.dev0}/include/dftracer/core/common/enumeration.h

@@ -4,7 +4,9 @@
 
 #ifndef DFTRACER_ENUMERATION_H
 #define DFTRACER_ENUMERATION_H
-#include <cpp-logger/logger.h>
+#include <cstdint>
+#include <string>
+
 enum WriterType : uint8_t { CHROME = 0 };
 enum ProfilerStage : uint8_t {
   PROFILER_INIT = 0,
@@ -179,17 +181,6 @@ inline void convert(const std::string& s, ProfileInitType& type) {
     type = ProfileInitType::PROFILER_INIT_NONE;
   }
 }
-inline void convert(const std::string& s, cpplogger::LoggerType& type) {
-  if (s == "DEBUG") {
-    type = cpplogger::LoggerType::CPP_LOGGER_DEBUG;
-  } else if (s == "INFO") {
-    type = cpplogger::LoggerType::CPP_LOGGER_INFO;
-  } else if (s == "WARN") {
-    type = cpplogger::LoggerType::CPP_LOGGER_WARN;
-  } else {
-    type = cpplogger::LoggerType::CPP_LOGGER_ERROR;
-  }
-}
 inline void convert(const std::string& s, AggregationType& type) {
   if (s == "FULL") {
     type = AggregationType::AGGREGATION_TYPE_FULL;

{dftracer-2.2.0.post8.dev0 → dftracer-2.2.0.post10.dev0}/python/dftracer/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '2.2.0.post8.dev0'
-__version_tuple__ = version_tuple = (2, 2, 0, 'post8', 'dev0')
+__version__ = version = '2.2.0.post10.dev0'
+__version_tuple__ = version_tuple = (2, 2, 0, 'post10', 'dev0')
 
 __commit_id__ = commit_id = None

{dftracer-2.2.0.post8.dev0 → dftracer-2.2.0.post10.dev0/python/dftracer.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dftracer
-Version: 2.2.0.post8.dev0
+Version: 2.2.0.post10.dev0
 Summary: I/O profiler for deep learning python apps. Specifically for dlio_benchmark.
 Author-email: "Hariharan Devarajan (Hari)" <hariharandev1@llnl.gov>
 License-Expression: MIT

{dftracer-2.2.0.post8.dev0 → dftracer-2.2.0.post10.dev0}/python/dftracer.egg-info/SOURCES.txt

@@ -364,6 +364,7 @@ test/check_events_present.sh
 test/check_file.sh
 test/check_file_at_least.sh
 test/check_file_not.sh
+test/check_public_headers.cmake
 test/ignore_result.h
 test/util.h
 test/c/test.c

{dftracer-2.2.0.post8.dev0 → dftracer-2.2.0.post10.dev0}/python/dftracer.egg-info/scm_file_list.json

@@ -368,6 +368,7 @@
     "test/check_file.sh",
     "test/check_file_at_least.sh",
     "test/check_file_not.sh",
+    "test/check_public_headers.cmake",
     "test/cpp/test.cpp",
     "test/cpp/test_cpp_api_coverage.cpp",
     "test/cpp/test_entity.cpp",

{dftracer-2.2.0.post8.dev0 → dftracer-2.2.0.post10.dev0}/scripts/wheel/test_wheel.py

@@ -14,9 +14,13 @@ Checks, in order:
   3. every bundled shared library dlopen()s
   4. every native executable in dftracer/bin starts (catches ELF corruption
      from the auditwheel/patchelf rewrite, which segfaults before main)
-  5. a real trace is produced: dftracer_service + an LD_PRELOAD'd workload,
+  5. every shipped dftracer/ public header, except the bundled
+     dftracer-utils ones under dftracer/utils/, compiles against the wheel's
+     own include/ directory alone (catches a header that includes a
+     dependency whose headers the wheel does not ship, e.g. cpp-logger)
+  6. a real trace is produced: dftracer_service + an LD_PRELOAD'd workload,
      and the resulting .pfw trace contains the I/O events
-  6. the pydftracer Python API layer, if installed, can log through the wheel
+  7. the pydftracer Python API layer, if installed, can log through the wheel
 """
 
 from __future__ import annotations
@@ -27,6 +31,7 @@ import glob
 import gzip
 import json
 import os
+import shutil
 import subprocess
 import sys
 import tempfile
@@ -214,6 +219,41 @@ def main() -> int:
         if broken:
             raise AssertionError("executables crash on startup: " + ", ".join(broken))
 
+    @check("public headers compile standalone")
+    def _headers():
+        include_dir = pkg_dir / "include"
+        # dftracer/utils/ holds the bundled dftracer-utils headers, which are
+        # C++20 and owned by that package, so only dftracer's own are checked.
+        utils_dir = include_dir / "dftracer" / "utils"
+        headers = sorted(
+            h
+            for h in (include_dir / "dftracer").rglob("*.h")
+            if utils_dir not in h.parents
+        )
+        if not headers:
+            raise AssertionError("no dftracer/ headers found in the wheel")
+        cxx = shutil.which(os.environ.get("CXX", "c++"))
+        if cxx is None:
+            raise Skip("no C++ compiler on PATH")
+        broken = []
+        with tempfile.TemporaryDirectory() as tmp:
+            for header in headers:
+                rel = header.relative_to(include_dir).as_posix()
+                tu = Path(tmp) / "tu.cpp"
+                tu.write_text(f"#include <{rel}>\nint main() {{ return 0; }}\n")
+                # Only the wheel's include/ on the path: a user compiling
+                # against the installed wheel has nothing else.
+                proc = run(
+                    [cxx, "-std=c++17", "-fsyntax-only", f"-I{include_dir}", str(tu)]
+                )
+                if proc.returncode != 0:
+                    broken.append(rel)
+                    print(f"BROKEN     : {rel}\n{proc.stdout.strip()}")
+                else:
+                    print(f"compiles   : {rel}")
+        if broken:
+            raise AssertionError("headers do not compile: " + ", ".join(broken))
+
     if args.no_trace:
         print("\nskipping runtime tracing checks (--no-trace)", flush=True)
         return report()

{dftracer-2.2.0.post8.dev0 → dftracer-2.2.0.post10.dev0}/src/dftracer/core/utils/configuration_manager.h

@@ -9,6 +9,21 @@
 
 #include <string>
 #include <vector>
+
+// Kept out of the public enumeration.h so installed headers do not
+// depend on cpp-logger, which is a private dependency.
+inline void convert(const std::string& s, cpplogger::LoggerType& type) {
+  if (s == "DEBUG") {
+    type = cpplogger::LoggerType::CPP_LOGGER_DEBUG;
+  } else if (s == "INFO") {
+    type = cpplogger::LoggerType::CPP_LOGGER_INFO;
+  } else if (s == "WARN") {
+    type = cpplogger::LoggerType::CPP_LOGGER_WARN;
+  } else {
+    type = cpplogger::LoggerType::CPP_LOGGER_ERROR;
+  }
+}
+
 namespace dftracer {
 class Metadata;
 struct ExecInfo {

{dftracer-2.2.0.post8.dev0 → dftracer-2.2.0.post10.dev0}/test/CMakeLists.txt

@@ -846,6 +846,20 @@ set_tests_properties(check_file_exists_${test_name} PROPERTIES DEPENDS ${test_na
 # ########################################################################
 
 # This test verifies the CMake package config works correctly
+# Installed public headers must compile without any private dependency
+# (cpp-logger, brahma, ...), since wheels and installs do not ship those.
+set(test_name test_public_headers_self_contained)
+add_test(NAME ${test_name}
+    COMMAND ${CMAKE_COMMAND}
+    "-DHEADERS=${DFTRACER_CORE_PUBLIC_INCLUDE}"
+    -DSOURCE_ROOT=${PROJECT_SOURCE_DIR}/include
+    -DSTAGE_DIR=${CMAKE_CURRENT_BINARY_DIR}/public_headers
+    -DCXX_COMPILER=${CMAKE_CXX_COMPILER}
+    -DC_COMPILER=${CMAKE_C_COMPILER}
+    "-DC_HEADERS=dftracer/dftracer.h;dftracer/core/common/constants.h;dftracer/core/common/typedef.h;dftracer/core/common/entity.h"
+    -P ${CMAKE_CURRENT_SOURCE_DIR}/check_public_headers.cmake
+)
+
 set(test_name test_package_find_build_tree)
 find_program(SYSTEM_CMAKE cmake PATHS /usr/bin /usr/local/bin NO_DEFAULT_PATH)
 set(_dftracer_package_find_cmake_args

dftracer-2.2.0.post10.dev0/test/check_public_headers.cmake

@@ -0,0 +1,71 @@
+# Checks that every installed public header compiles on its own.
+#
+# A wheel or `make install` ships only the headers in
+# DFTRACER_CORE_PUBLIC_INCLUDE. Private dependencies (cpp-logger, brahma,
+# gotcha, yaml-cpp, ...) are not shipped, so a public header that includes
+# one breaks every downstream C/C++ build. This script copies the public
+# headers into an empty include tree and compiles each one with only that
+# tree on the include path.
+#
+# Inputs (-D):
+#   HEADERS      ;-list of absolute public header paths
+#   SOURCE_ROOT  directory the include paths are relative to (<src>/include)
+#   STAGE_DIR    scratch directory, wiped on every run
+#   CXX_COMPILER C++ compiler
+#   C_COMPILER   C compiler
+#   C_HEADERS    ;-list of headers (relative to SOURCE_ROOT) that must also
+#                compile as C
+
+foreach(var HEADERS SOURCE_ROOT STAGE_DIR CXX_COMPILER C_COMPILER)
+  if(NOT DEFINED ${var} OR "${${var}}" STREQUAL "")
+    message(FATAL_ERROR "check_public_headers: ${var} is not set")
+  endif()
+endforeach()
+
+file(REMOVE_RECURSE "${STAGE_DIR}")
+set(stage_include "${STAGE_DIR}/include")
+file(MAKE_DIRECTORY "${stage_include}")
+
+set(relative_headers)
+foreach(header ${HEADERS})
+  file(RELATIVE_PATH rel "${SOURCE_ROOT}" "${header}")
+  if(rel MATCHES "^\\.\\./")
+    message(FATAL_ERROR "check_public_headers: ${header} is not under ${SOURCE_ROOT}")
+  endif()
+  get_filename_component(rel_dir "${rel}" DIRECTORY)
+  file(COPY "${header}" DESTINATION "${stage_include}/${rel_dir}")
+  list(APPEND relative_headers "${rel}")
+endforeach()
+
+set(failures 0)
+
+function(compile_header compiler lang std rel)
+  string(MAKE_C_IDENTIFIER "${rel}_${lang}" stem)
+  set(tu "${STAGE_DIR}/${stem}.${lang}")
+  file(WRITE "${tu}" "#include <${rel}>\nint main(void) { return 0; }\n")
+  execute_process(
+    COMMAND "${compiler}" ${std} -fsyntax-only -I "${stage_include}" "${tu}"
+    RESULT_VARIABLE rc
+    OUTPUT_VARIABLE out
+    ERROR_VARIABLE out)
+  if(rc EQUAL 0)
+    message(STATUS "PASS [${lang}] ${rel}")
+  else()
+    message(STATUS "FAIL [${lang}] ${rel}\n${out}")
+    math(EXPR n "${failures} + 1")
+    set(failures ${n} PARENT_SCOPE)
+  endif()
+endfunction()
+
+foreach(rel ${relative_headers})
+  compile_header("${CXX_COMPILER}" cpp -std=c++17 "${rel}")
+endforeach()
+foreach(rel ${C_HEADERS})
+  compile_header("${C_COMPILER}" c -std=c99 "${rel}")
+endforeach()
+
+if(failures GREATER 0)
+  message(FATAL_ERROR
+    "${failures} public header check(s) failed: an installed header includes "
+    "something that is not installed with it.")
+endif()