easylink 0.1.14__tar.gz → 0.1.15__tar.gz

{easylink-0.1.14 → easylink-0.1.15}/CHANGELOG.rst

@@ -1,3 +1,7 @@
+**0.1.15 - 5/5/25**
+
+ - Fix SyntaxWarning for unescaped backslashes
+
 **0.1.14 - 5/1/25**
 
  - Add support for EmbarrassinglyParallelSteps to accept sections (i.e. non-leaf steps)

{easylink-0.1.14 → easylink-0.1.15}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: easylink
-Version: 0.1.14
+Version: 0.1.15
 Summary: Research repository for the EasyLink ER ecosystem project.
 Home-page: https://github.com/ihmeuw/easylink
 Author: The EasyLink developers

{easylink-0.1.14 → easylink-0.1.15}/docs/source/user_guide/tutorials/getting_started.rst

@@ -227,9 +227,11 @@ of the pipeline, we can view intermediary outputs as well::
 
 Environments
 ============
-The ``-e`` argument to ``easylink run`` accepts a YAML file specifying information about the computing environment which will execute the steps of the 
-pipeline. When we ran our first pipeline, ``common/pipeline.yaml``, above, we passed ``specifications/common/environment_local.yaml`` to this argument.
-The contents of this YAML file are shown below.
+The ``--computing-environment`` (``-e``) argument to ``easylink run`` accepts a YAML file specifying 
+information about the computing environment which will execute the steps of the 
+pipeline. When we ran our first pipeline, ``tests/specifications/common/pipeline.yaml`` above, we passed 
+``tests/specifications/common/environment_local.yaml`` 
+to this argument. The contents of this YAML file are shown below.
 
 .. code-block:: yaml
 
@@ -246,8 +248,9 @@ Let's run this same pipeline with the ``slurm`` computing environment. `Slurm <h
 cluster management system which EasyLink can interface with to schedule and run the steps of a pipeline using the resources of a computing cluster. This means that instead of 
 running all pipeline steps in your local computing environment, each step can be run with the additional resources of a separate compute node.
 
-To run the pipeline using slurm, we will pass ``specifications/examples/environment_slurm.yaml`` 
-to the ``environment`` command line parameter, which looks like this:
+To run the pipeline using slurm, we will pass :download:`environment_slurm.yaml <environment_slurm.yaml>` 
+to the ``--computing-environment`` command line parameter. Download the file to the directory you will run ``easylink`` 
+in -- I downloaded it to the root ``easylink`` directory. The YAML looks like this:
 
 .. code-block:: yaml
 
@@ -277,15 +280,15 @@ section, but using the ``slurm`` environment rather than ``local``.
 
 .. code-block:: console
 
-   $ easylink run -p specifications/common/pipeline.yaml -i specifications/common/input_data.yaml -e specifications/examples/environment_slurm.yaml
-   2025-04-23 08:39:45.486 | 0:00:02.489631 | run:158 - Running pipeline
-   2025-04-23 08:39:45.486 | 0:00:02.489962 | run:160 - Results directory: /mnt/share/homes/tylerdy/easylink/tests/results/2025_04_23_08_39_45
-   2025-04-23 08:39:48.973 | 0:00:05.976983 | main:115 - Running Snakemake
-   [Wed Apr 23 08:39:49 2025]
+   $ easylink run -p tests/specifications/common/pipeline.yaml -i tests/specifications/common/input_data.yaml -e environment_slurm.yaml
+   2025-05-01 08:24:01.901 | 0:00:02.805179 | run:158 - Running pipeline
+   2025-05-01 08:24:01.901 | 0:00:02.805621 | run:160 - Results directory: /mnt/share/homes/tylerdy/easylink/results/2025_05_01_08_24_01
+   2025-05-01 08:24:05.205 | 0:00:06.109547 | main:115 - Running Snakemake
+   [Thu May  1 08:24:06 2025]
    Job 9: Validating step_4_python_pandas input slot step_4_secondary_input
    Reason: Missing output files: input_validations/step_4_python_pandas/step_4_secondary_input_validator
    ...
-   [Wed Apr 23 08:43:00 2025]
+   [Thu May  1 08:26:16 2025]
    Job 0: Grabbing final output
    Reason: Missing output files: result.parquet; Input files updated by another job: input_validations/final_validator, intermediate/step_4_python_pandas/result.parquet
 
@@ -299,6 +302,132 @@ environment. However, for a real large-scale record linkage pipeline, the additi
 faster than ``local``, or make it *possible* to run the pipeline when it wouldn't be otherwise 
 (in the case where the local environment doesn't have sufficient resources to run the pipeline).
 
+Input data
+==========
+The ``--input-data`` (``-i``) argument to ``easylink run`` accepts a YAML file specifying a list 
+of paths to files or directories containing input data to be used by the pipeline. 
+When we ran our first pipeline, ``common/pipeline.yaml``, above, we passed 
+``tests/specifications/common/input_data.yaml`` 
+as this YAML file, shown below::
+
+   input_file_1: /mnt/team/simulation_science/priv/engineering/er_ecosystem/sample_data/dummy/input_file_1.parquet
+   input_file_2: /mnt/team/simulation_science/priv/engineering/er_ecosystem/sample_data/dummy/input_file_2.parquet
+   input_file_3: /mnt/team/simulation_science/priv/engineering/er_ecosystem/sample_data/dummy/input_file_3.parquet
+
+Let's try passing a different input data specification YAML file, 
+:download:`input_data.yaml <input_data.yaml>`, which looks like this::
+
+   input_file_1: input_file_1.parquet
+   input_file_2: input_file_2.parquet
+   input_file_3: input_file_3.parquet
+
+Download the file to the directory you will run easylink in, and then download the three input 
+Parquet files, :download:`input_file_1.parquet <input_file_1.parquet>`, :download:`input_file_2.parquet <input_file_2.parquet>` 
+and :download:`input_file_3.parquet <input_file_3.parquet>` to the same directory. In this case 
+I downloaded them to the root ``easylink`` directory.
+
+These input files look a little different than the three input files we used in the pipelines we ran above, 
+where all three input files listed in the YAML specification were identical. Let's compare one of those, 
+``/mnt/team/simulation_science/priv/engineering/er_ecosystem/sample_data/dummy/input_file_1.parquet``, to 
+the three files we will use here::
+
+   $ pqprint /mnt/team/simulation_science/priv/engineering/er_ecosystem/sample_data/dummy/input_file_1.parquet
+         foo bar  counter
+   0        0   a        0
+   1        1   b        0
+   2        2   c        0
+   3        3   d        0
+   4        4   e        0
+   ...    ...  ..      ...
+   9995  9995   a        0
+   9996  9996   b        0
+   9997  9997   c        0
+   9998  9998   d        0
+   9999  9999   e        0
+   [10000 rows x 3 columns]
+   $ pqprint input_file_1.parquet 
+      foo bar  counter
+   0     0   l       10
+   1     1   m       10
+   2     2   n       10
+   3     3   o       10
+   4     4   p       10
+   ..  ...  ..      ...
+   95   95   l       10
+   96   96   m       10
+   97   97   n       10
+   98   98   o       10
+   99   99   p       10
+
+   [100 rows x 3 columns]
+   $ pqprint input_file_2.parquet 
+      foo bar  counter
+   0     0   q       20
+   1     1   r       20
+   2     2   s       20
+   3     3   t       20
+   4     4   u       20
+   ..  ...  ..      ...
+   95   95   q       20
+   96   96   r       20
+   97   97   s       20
+   98   98   t       20
+   99   99   u       20
+   [100 rows x 3 columns]
+   $ pqprint input_file_3.parquet 
+      foo bar  counter
+   0     0   v       30
+   1     1   w       30
+   2     2   x       30
+   3     3   y       30
+   4     4   z       30
+   ..  ...  ..      ...
+   95   95   v       30
+   96   96   w       30
+   97   97   x       30
+   98   98   y       30
+   99   99   z       30
+   [100 rows x 3 columns]
+
+Our three new input files look different from each other and from the previous input files.
+They have 100 rows each instead of 10000, the ``bar`` column has a different set of values 
+for each file, and the ``counter`` in each file starts at a different value.
+
+Let's run the same pipeline as before, but with this new input data YAML.
+
+.. code-block:: console
+   
+   $ easylink run -p tests/specifications/common/pipeline.yaml -i input_data.yaml -e tests/specifications/common/environment_local.yaml
+   2025-05-01 08:05:01.123 | 0:00:02.781384 | run:158 - Running pipeline
+   2025-05-01 08:05:01.123 | 0:00:02.781776 | run:160 - Results directory: /mnt/share/homes/tylerdy/easylink/results/2025_05_01_08_05_01
+   2025-05-01 08:05:04.498 | 0:00:06.156166 | main:115 - Running Snakemake
+   [Thu May  1 08:05:05 2025]
+   Job 9: Validating step_4_python_pandas input slot step_4_secondary_input
+   Reason: Missing output files: input_validations/step_4_python_pandas/step_4_secondary_input_validator
+   ...
+   [Thu May  1 08:05:32 2025]
+   Job 0: Grabbing final output
+   Reason: Missing output files: result.parquet; Input files updated by another job: intermediate/step_4_python_pandas/result.parquet, input_validations/final_validator
+   $ pqprint results/2025_05_01_08_05_01/result.parquet 
+      foo bar  counter  added_column_0  added_column_1  added_column_2  added_column_3  added_column_4
+   0      0   l       14             0.0             1.0             2.0             3.0               4
+   1      1   m       14             0.0             1.0             2.0             3.0               4
+   2      2   n       14             0.0             1.0             2.0             3.0               4
+   3      3   o       14             0.0             1.0             2.0             3.0               4
+   4      4   p       14             0.0             1.0             2.0             3.0               4
+   ..   ...  ..      ...             ...             ...             ...             ...             ...
+   595   95   v       31             0.0             0.0             0.0             0.0               4
+   596   96   w       31             0.0             0.0             0.0             0.0               4
+   597   97   x       31             0.0             0.0             0.0             0.0               4
+   598   98   y       31             0.0             0.0             0.0             0.0               4
+   599   99   z       31             0.0             0.0             0.0             0.0               4
+
+As expected, the ``results.parquet`` has 600 rows (as opposed to 60k with the old input YAML) 
+and the range of ``bar`` and ``counter``  values are consistent 
+with our new input files. As before, the transformation of the data is specific to the development schema and will 
+change.
+
+
 More Pipeline Specifications
 ============================
 The ``tests`` folder includes several other pipeline specification files (YAML files). While some are special 

easylink-0.1.15/docs/source/user_guide/tutorials/input_data.yaml

@@ -0,0 +1,3 @@
+input_file_1: input_file_1.parquet
+input_file_2: input_file_2.parquet
+input_file_3: input_file_3.parquet

easylink-0.1.15/src/easylink/_version.py

@@ -0,0 +1 @@
+__version__ = "0.1.15"

{easylink-0.1.14 → easylink-0.1.15}/src/easylink/utilities/spark.smk

@@ -70,7 +70,7 @@ rule wait_for_spark_master:
         while true; do
 
             if [[ -e {params.spark_master_log_file} ]]; then
-                found=`grep -o "\(spark://.*$\)" {params.spark_master_log_file} || true`
+                found=`grep -o "\\(spark://.*$\\)" {params.spark_master_log_file} || true`
 
                 if [[ ! -z $found ]]; then
                     echo "Spark master URL found: $found"
@@ -178,7 +178,7 @@ rule wait_for_spark_worker:
         while true; do
 
             if [[ -e {params.spark_worker_log_file} ]]; then
-                found=`grep -o "\(Worker: Successfully registered with master $MASTER_URL\)" {params.spark_worker_log_file} || true`
+                found=`grep -o "\\(Worker: Successfully registered with master $MASTER_URL\\)" {params.spark_worker_log_file} || true`
 
                 if [[ ! -z $found ]]; then
                     echo "Spark Worker {wildcards.scatteritem} registered successfully"

{easylink-0.1.14 → easylink-0.1.15}/src/easylink.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: easylink
-Version: 0.1.14
+Version: 0.1.15
 Summary: Research repository for the EasyLink ER ecosystem project.
 Home-page: https://github.com/ihmeuw/easylink
 Author: The EasyLink developers

{easylink-0.1.14 → easylink-0.1.15}/src/easylink.egg-info/SOURCES.txt

@@ -72,8 +72,13 @@ docs/source/user_guide/index.rst
 docs/source/user_guide/tutorials/DAG-common-pipeline.svg
 docs/source/user_guide/tutorials/DAG-e2e-pipeline-expanded.svg
 docs/source/user_guide/tutorials/DAG-e2e-pipeline.svg
+docs/source/user_guide/tutorials/environment_slurm.yaml
 docs/source/user_guide/tutorials/getting_started.rst
 docs/source/user_guide/tutorials/index.rst
+docs/source/user_guide/tutorials/input_data.yaml
+docs/source/user_guide/tutorials/input_file_1.parquet
+docs/source/user_guide/tutorials/input_file_2.parquet
+docs/source/user_guide/tutorials/input_file_3.parquet
 src/easylink/__about__.py
 src/easylink/__init__.py
 src/easylink/_version.py
@@ -140,7 +145,6 @@ tests/specifications/common/pipeline.yaml
 tests/specifications/e2e/environment_slurm.yaml
 tests/specifications/e2e/pipeline.yaml
 tests/specifications/e2e/pipeline_expanded.yaml
-tests/specifications/examples/environment_slurm.yaml
 tests/specifications/integration/environment_spark_slurm.yaml
 tests/specifications/integration/pipeline.yaml
 tests/specifications/integration/pipeline_spark.yaml

easylink-0.1.14/src/easylink/_version.py

@@ -1 +0,0 @@
-__version__ = "0.1.14"