idf-build-apps 1.0.0rc0__tar.gz → 1.0.1__tar.gz

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/.github/workflows/test-build-idf-apps.yml

@@ -60,6 +60,7 @@ jobs:
           export PYENV_ROOT="$HOME/.pyenv" && export PATH="$PYENV_ROOT/bin:$PATH" && eval "$(pyenv init --path)"
           pyenv global ${{ matrix.python-version }}
           rm $(which python3)
+          echo "urllib3<1.25,>=1.21.1" >> $IDF_PATH/requirements.txt
           bash $IDF_PATH/install.sh
           . $IDF_PATH/export.sh
           pip install idf_build_apps-*-py2.py3-none-any.whl
@@ -76,7 +77,7 @@ jobs:
     needs: build-python-packages
     strategy:
       matrix:
-        idf-branch: [ release-v4.3, release-v4.4, release-v5.0]
+        idf-branch: [release-v4.3, release-v4.4, release-v5.0, release-v5.1]
     runs-on: ubuntu-latest
     container:
       image: espressif/idf:${{ matrix.idf-branch }}

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/CHANGELOG.md

@@ -2,7 +2,13 @@
 
 All notable changes to this project will be documented in this file.
 
-## [Unreleased - 1.0.0]
+## [1.0.1] (2023-06-12)
+
+## Fixed
+
+- glob patterns are matched recursively
+
+## [1.0.0] (2023-05-25)
 
 ## Added
 
@@ -130,7 +136,7 @@ This is the last version to support ESP-IDF v4.1 since it's EOL on Feb. 24th, 20
 
 ### Fixed
 
-- Ralative path defined in the manifest files depend on the current work path
+- Relative path defined in the manifest files depend on the current work path
 
   Added `manifest_rootpath` argument in `find_apps()`. Will use this value instead as the root folder for calculating absolute path
 

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/CONTRIBUTING.md

@@ -6,9 +6,10 @@ Please use python 3.7+ for developing, but keep in mind that we need to support
 |-----------------|---------------------------|
 | 4.1             | 2.7, 3.4+                 |
 | 4.2             | 2.7, 3.4+                 |
-| 4.3             | 3.6+                      |
-| 4.4             | 3.6+                      |
+| 4.3             | 2.7, 3.4+                 |
+| 4.4             | 2.7, 3.4+                 |
 | 5.0             | 3.7+                      |
+| 5.1             | 3.7+                      |
 | master          | 3.7+                      |
 
 ## Setup the Dev Environment

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: idf-build-apps
-Version: 1.0.0rc0
+Version: 1.0.1
 Summary: Tools for building ESP-IDF related apps.
 Author-email: Fu Hanxi <fuhanxi@espressif.com>
 Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
@@ -17,6 +17,7 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Requires-Dist: pathlib; python_version < '3.4'
+Requires-Dist: glob2; python_version < '3.5'
 Requires-Dist: pyparsing
 Requires-Dist: pyyaml
 Requires-Dist: packaging

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/docs/find_build.md

@@ -59,6 +59,7 @@ flowchart TB
     end
 ```
 
+(config-rules)=
 ### Config Rules
 
 Config rule represents the sdkconfig file pattern and the config name. The syntax is simple: `[FILE_PATTERN]=[CONFIG_NAME]`.
@@ -96,10 +97,10 @@ For example, in project `test-1`:
       - The wildcard matches two files. Build two apps based on each sdkconfig file.
       -  - ``sdkconfig.ci.foo``
          - ``sdkconfig.ci.bar``
+```
 
-.. note::
-
-   For each config rule, only one wildcard is supported.
+```{note}
+For each config rule, only one wildcard is supported.
 ```
 
 ### Placeholders for Work Directory and Build Directory
@@ -114,7 +115,7 @@ By default, `idf-build-apps` would use the project directory as the work directo
 
 #### Build Directory
 
-Build directory is the directory where the binary files output would be generated. If it is set to a relative path, the full path would be calculated based on the work directory. If it is a absolute path, it would override the work directory settings.
+Build directory is the directory where the binary files output would be generated. If it is set to a relative path, the full path would be calculated based on the work directory. If it is an absolute path, it would override the work directory settings.
 
 By default, `idf-build-apps` would follow what ESP-IDF does, use `build` as the build directory.
 
@@ -127,6 +128,7 @@ Placeholders are a set of symbols, which could be used when setting work directo
 - `@n`: Would be replaced by the project name.
 - `@f`: Would be replaced by the escaped project path (replaced "/" to "_").
 - `@i`: Would be replaced by the build index. (only available in `build` command)
+- `@p`: Would be replaced by the parallel build index. (default to `1`, only available in `build` command)
 
 For example,
 
@@ -160,11 +162,8 @@ The output would be:
 
 Building apps is a process that build all the applications that are collected by the "finding apps" process.
 
-```{eval-rst}
-
-.. note::
-
-   Almost all CLI options that ``find`` supported are also supported in ``build`` command. You may call ``idf-build-apps find -h`` or ``idf-build-apps build -h`` for all possible CLI options.
+```{note}
+Almost all CLI options that ``find`` supported are also supported in ``build`` command. You may call ``idf-build-apps find -h`` or ``idf-build-apps build -h`` for all possible CLI options.
 ```
 
 ### Tips on `build` CLI Options

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/docs/manifest.md

@@ -30,6 +30,7 @@ One typical manifest file look like this:
   - `IDF_VERSION_MINOR`
   - `IDF_VERSION_PATCH`
   - `INCLUDE_DEFAULT` (The default value of officially supported targets is 1, otherwise is 0)
+  - `CONFIG_NAME` (config name defined in [](project:#config-rules))
   - environment variables, default to `0` if not set
 - String, must be double-quoted. e.g., `"esp32"`, `"12345"`
 - Integer, support decimal and hex. e.g., `1`, `0xAB`
@@ -138,7 +139,7 @@ One app will be built when:
 - Files that matches any of the specified `depends_filepatterns` in the corresponding manifest file are modified
 - Any of the `build_components` are modified. `build_components` are defined in the `project_description.json`, which is generated by running `idf.py reconfigure` or `idf.py build`
 
-For example, this is an app `exmaple/foo`, which depends on `comp1`, `comp2`, `comp3` and all files under `common_header_files`:
+For example, this is an app `example/foo`, which depends on `comp1`, `comp2`, `comp3` and all files under `common_header_files`:
 
 ```yaml
 examples/foo:

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/idf_build_apps/__init__.py

@@ -7,7 +7,7 @@ Tools for building ESP-IDF related apps.
 
 import logging
 
-__version__ = '1.0.0.rc0'
+__version__ = '1.0.1'
 
 LOGGER = logging.getLogger('idf_build_apps')
 

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/idf_build_apps/finder.py

@@ -17,6 +17,7 @@ from .app import (
 )
 from .utils import (
     config_rules_from_str,
+    to_absolute_path,
     to_list,
 )
 
@@ -165,15 +166,18 @@ def _find_apps(
 
     # The remaining part is for recursive == True
     apps = []
+    # handle the exclude list, since the config file might use linux style, but run in windows
+    exclude_list = [to_absolute_path(p) for p in exclude_list]
     for root, dirs, _ in os.walk(path):
         LOGGER.debug('Entering %s', root)
-        if root in exclude_list:
+        root_path = to_absolute_path(root)
+        if root_path in exclude_list:
             LOGGER.debug('=> Skipping %s (excluded)', root)
             del dirs[:]
             continue
 
-        if root.endswith('managed_components'):  # idf-component-manager
-            LOGGER.debug('=> Skipping %s (managed components)', root)
+        if root_path.parts[-1] == 'managed_components':  # idf-component-manager
+            LOGGER.debug('=> Skipping %s (managed components)', root_path)
             del dirs[:]
             continue
 

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/idf_build_apps/main.py

@@ -498,11 +498,7 @@ def get_parser():  # type: () -> argparse.ArgumentParser
         action='store_true',
         help='Look for apps in the specified paths recursively',
     )
-    common_args.add_argument(
-        '--exclude',
-        nargs='+',
-        help='Ignore specified directory (if --recursive is given)',
-    )
+    common_args.add_argument('--exclude', nargs='+', help='Ignore specified path (if --recursive is given)')
     common_args.add_argument(
         '--work-dir',
         help='If set, the app is first copied into the specified directory, and then built. '
@@ -588,10 +584,9 @@ def get_parser():  # type: () -> argparse.ArgumentParser
         '--ignore-app-dependencies-filepatterns',
         nargs='*',
         default=None,
-        help='space-separated list which specifies the file patterns used for ignoring the component dependencies. '
-        'The `depends_components` and `depends_filepatterns` set in the manifest files will be ignored when any of '
-        'the specified file patterns matches any of the modified files. Must be used together with '
-        '--modified-files',
+        help='space-separated list which specifies the file patterns used for ignoring checking the app dependencies. '
+        'The `depends_components` and `depends_filepatterns` set in the manifest files will be ignored when any of the '
+        'specified file patterns matches any of the modified files. Must be used together with --modified-files',
     )
 
     common_args.add_argument(

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/idf_build_apps/utils.py

@@ -23,6 +23,12 @@ except ImportError:
     pass
 
 
+if sys.version_info < (3, 5):
+    import glob2 as glob
+else:
+    import glob
+
+
 class ConfigRule:
     def __init__(self, file_name, config_name):  # type: (str, str) -> None
         """
@@ -213,14 +219,17 @@ def files_matches_patterns(
     patterns,  # type: list[str] | str
     rootpath=None,  # type: str
 ):  # type: (...) -> bool
-    # can't match a absolute pattern with a relative path
+    # can't match an absolute pattern with a relative path
     # change all to absolute paths
     files = [to_absolute_path(f, rootpath) for f in to_list(files)]
     patterns = [to_absolute_path(p, rootpath) for p in to_list(patterns)]
 
+    matched_paths = set()
+    for pat in patterns:
+        matched_paths.update(glob.glob(str(pat), recursive=True))
+
     for f in files:
-        for p in patterns:
-            if f.match(str(p)):
-                return True
+        if str(f) in matched_paths:
+            return True
 
     return False

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/pyproject.toml

@@ -27,6 +27,7 @@ requires-python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
 
 dependencies = [
     "pathlib; python_version < '3.4'",
+    "glob2; python_version < '3.5'",
     "pyparsing",
     "pyyaml",
     "packaging",

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/setup.py

@@ -15,6 +15,7 @@ install_requires = \
 extras_require = \
 {":python_version < '3.11'": ['toml'],
  ":python_version < '3.4'": ['pathlib'],
+ ":python_version < '3.5'": ['glob2'],
  'doc': ['sphinx',
          'sphinx-rtd-theme',
          'sphinx_copybutton',
@@ -26,7 +27,7 @@ entry_points = \
 {'console_scripts': ['idf-build-apps = idf_build_apps:main.main']}
 
 setup(name='idf-build-apps',
-      version='1.0.0rc0',
+      version='1.0.1',
       description='Tools for building ESP-IDF related apps.',
       author=None,
       author_email='Fu Hanxi <fuhanxi@espressif.com>',

{idf_build_apps-1.0.0rc0 → idf_build_apps-1.0.1}/tests/test_finder.py

@@ -3,6 +3,7 @@
 
 import logging
 import os
+import tempfile
 from pathlib import (
     Path,
 )
@@ -185,9 +186,8 @@ get-started:
     @pytest.mark.parametrize(
         'modified_components, modified_files, could_find_apps',
         [
-            ([], '/foo', True),
             ([], str(IDF_PATH / 'examples' / 'README.md'), False),
-            (None, [str(IDF_PATH / 'examples' / 'get-started' / 'hello_world' / 'a.md')], True),
+            (None, [str(IDF_PATH / 'examples' / 'get-started' / 'hello_world' / 'a.md')], False),
             (
                 [],
                 [
@@ -208,7 +208,6 @@ get-started:
             f'''
 {test_dir}:
     depends_filepatterns:
-        - /foo
         - examples/get-started/hello_world/**
         - examples/foo/**
 ''',
@@ -469,3 +468,32 @@ class TestFindWithSdkconfigFiles:
         )
         assert len(apps) == 3
         monkeypatch.delenv('TEST_ENV_VAR')
+
+
+@pytest.mark.parametrize(
+    'exclude_list, apps_count',
+    [
+        (['test1'], 3),  # not excluded
+        (['folder1/test2'], 2),
+        (['folder1'], 1),
+        (['folder2/test2'], 2),
+    ],
+)
+def test_find_apps_with_exclude(tmp_path, exclude_list, apps_count):
+    (tmp_path / 'folder1').mkdir()
+    (tmp_path / 'folder2').mkdir()
+
+    create_project('test1', tmp_path / 'folder1')
+    create_project('test2', tmp_path / 'folder1')
+    create_project('test2', tmp_path / 'folder2')
+
+    os.chdir(tmp_path)
+    apps = find_apps(str(tmp_path), 'esp32', recursive=True, exclude_list=exclude_list)
+    assert len(apps) == apps_count
+
+    # or with absolute_path
+    exclude_list = [os.path.abspath(x) for x in exclude_list]
+    tmp_path = os.path.abspath(tmp_path)
+    os.chdir(tempfile.tempdir)
+    apps = find_apps(str(tmp_path), 'esp32', recursive=True, exclude_list=exclude_list)
+    assert len(apps) == apps_count

idf_build_apps-1.0.1/tests/test_utils.py

@@ -0,0 +1,136 @@
+# SPDX-FileCopyrightText: 2022-2023 Espressif Systems (Shanghai) CO LTD
+# SPDX-License-Identifier: Apache-2.0
+import os
+from pathlib import (
+    Path,
+)
+
+import pytest
+
+from idf_build_apps.utils import (
+    files_matches_patterns,
+    get_parallel_start_stop,
+    rmdir,
+    to_absolute_path,
+)
+
+
+@pytest.mark.parametrize(
+    'patterns, expected',
+    [
+        ('*.txt', ['test/inner', 'test/inner/test.txt', 'test/test.txt']),
+        (
+            ['*.txt', '*.log'],
+            [
+                'test/inner',
+                'test/inner/build.log',
+                'test/inner/test.txt',
+                'test/test.txt',
+            ],
+        ),
+    ],
+)
+def test_rmdir(tmpdir, patterns, expected):
+    test_dir = tmpdir.mkdir('test')
+    dir1 = test_dir.mkdir('inner')
+    test_dir.mkdir('inner2')
+
+    Path(dir1 / 'test.txt').touch()
+    Path(dir1 / 'build.log').touch()
+    Path(test_dir / 'test.txt').touch()
+
+    rmdir(test_dir, exclude_file_patterns=patterns)
+
+    assert sorted(Path(test_dir).glob('**/*')) == [Path(tmpdir / i) for i in expected]
+
+
+@pytest.mark.parametrize(
+    'total, parallel_count, parallel_index, start, stop',
+    [
+        (1, 1, 1, 1, 1),
+        (1, 2, 2, 2, 1),
+        (1, 10, 1, 1, 1),
+        (6, 4, 1, 1, 2),
+        (6, 4, 2, 3, 4),
+        (6, 4, 3, 5, 6),
+        (6, 4, 4, 7, 6),
+        (10, 10, 9, 9, 9),
+        (33, 2, 1, 1, 17),
+        (33, 2, 2, 18, 33),
+    ],
+)
+def test_get_parallel_start_stop(total, parallel_count, parallel_index, start, stop):
+    assert (start, stop) == get_parallel_start_stop(total, parallel_count, parallel_index)
+
+
+def test_files_matches_patterns(tmpdir):
+    # used for testing absolute paths
+    temp_dir = tmpdir.mkdir('temp')
+    os.chdir(temp_dir)
+
+    # create real files
+    test_dir = tmpdir.mkdir('test')
+    a_dir = test_dir.mkdir('a')
+    b_dir = a_dir.mkdir('b')
+    c_dir = b_dir.mkdir('c')
+    b_py = Path(a_dir / 'b.py')
+    c_py = Path(b_dir / 'c.py')
+    d_py = Path(c_dir / 'd.py')
+    b_py.touch()
+    c_py.touch()
+    d_py.touch()
+
+    # ├── temp
+    # └── test
+    #     └── a
+    #         ├── b
+    #         │   ├── c
+    #         │   │   └── d.py
+    #         │   └── c.py
+    #         ├── .hidden
+    #         └── b.py
+    #
+
+    # in correct relative path
+    for matched_files, pat, rootpath in [
+        ([b_py], '*.py', a_dir),
+        ([b_py, c_py, d_py], '**/*.py', a_dir),
+        ([c_py], '*.py', b_dir),
+        ([c_py, d_py], '**/*.py', b_dir),
+        ([d_py], '*.py', c_dir),
+        ([d_py], '**/*.py', c_dir),
+    ]:
+        for f in matched_files:
+            assert files_matches_patterns(f, pat, rootpath)
+
+    # in None root path with relative pattern
+    for matched_files, pat in [
+        ([b_py], 'a/*.py'),
+        ([b_py, c_py, d_py], 'a/**/*.py'),
+        ([c_py], 'a/b/*.py'),
+        ([c_py, d_py], 'a/b/**/*.py'),
+        ([d_py], 'a/b/c/*.py'),
+        ([d_py], 'a/b/c/**/*.py'),
+    ]:
+        for f in matched_files:
+            # with correct pwd
+            os.chdir(test_dir)
+            assert files_matches_patterns(f, pat)
+
+            # with wrong pwd
+            os.chdir(temp_dir)
+            assert not files_matches_patterns(f, pat)
+
+    # use correct absolute path, in wrong pwd
+    for matched_files, pat in [
+        ([b_py], 'a/*.py'),
+        ([b_py, c_py, d_py], 'a/**/*.py'),
+        ([c_py], 'a/b/*.py'),
+        ([c_py, d_py], 'a/b/**/*.py'),
+        ([d_py], 'a/b/c/*.py'),
+        ([d_py], 'a/b/c/**/*.py'),
+    ]:
+        abs_pat = to_absolute_path(pat, test_dir)
+        os.chdir(temp_dir)
+        for f in matched_files:
+            assert files_matches_patterns(f, abs_pat)

idf_build_apps-1.0.0rc0/tests/test_utils.py

@@ -1,84 +0,0 @@
-# SPDX-FileCopyrightText: 2022-2023 Espressif Systems (Shanghai) CO LTD
-# SPDX-License-Identifier: Apache-2.0
-
-from pathlib import (
-    Path,
-)
-
-import pytest
-
-from idf_build_apps.utils import (
-    files_matches_patterns,
-    get_parallel_start_stop,
-    rmdir,
-)
-
-
-@pytest.mark.parametrize(
-    'patterns, expected',
-    [
-        ('*.txt', ['test/inner', 'test/inner/test.txt', 'test/test.txt']),
-        (
-            ['*.txt', '*.log'],
-            [
-                'test/inner',
-                'test/inner/build.log',
-                'test/inner/test.txt',
-                'test/test.txt',
-            ],
-        ),
-    ],
-)
-def test_rmdir(tmpdir, patterns, expected):
-    test_dir = tmpdir.mkdir('test')
-    dir1 = test_dir.mkdir('inner')
-    test_dir.mkdir('inner2')
-
-    Path(dir1 / 'test.txt').touch()
-    Path(dir1 / 'build.log').touch()
-    Path(test_dir / 'test.txt').touch()
-
-    rmdir(test_dir, exclude_file_patterns=patterns)
-
-    assert sorted(Path(test_dir).glob('**/*')) == [Path(tmpdir / i) for i in expected]
-
-
-@pytest.mark.parametrize(
-    'total, parallel_count, parallel_index, start, stop',
-    [
-        (1, 1, 1, 1, 1),
-        (1, 2, 2, 2, 1),
-        (1, 10, 1, 1, 1),
-        (6, 4, 1, 1, 2),
-        (6, 4, 2, 3, 4),
-        (6, 4, 3, 5, 6),
-        (6, 4, 4, 7, 6),
-        (10, 10, 9, 9, 9),
-        (33, 2, 1, 1, 17),
-        (33, 2, 2, 18, 33),
-    ],
-)
-def test_get_parallel_start_stop(total, parallel_count, parallel_index, start, stop):
-    assert (start, stop) == get_parallel_start_stop(total, parallel_count, parallel_index)
-
-
-@pytest.mark.parametrize(
-    'files, patterns, rootpath, result',
-    [
-        ('c.py', '*.py', None, True),
-        ('c.py', '*.py', '/a', True),
-        ('/a/b/c.py', 'c.py', None, False),
-        ('/a/b/c.py', 'c.py', '/a/b', True),
-        ('/a/b/../b/c.py', '*.py', None, False),
-        ('/a/b/../b/c.py', '*.py', '/a/b', True),
-        ('/a/b/../b/c.py', '**/*.py', None, False),
-        ('/a/b/../b/c.py', '**/*.py', '/a', True),
-        ('c.py', '/a/**/*.py', None, False),
-        ('c.py', '/a/**/*.py', '/a', False),
-        ('c.py', '/a/**/*.py', '/a/b', True),
-        ('/a/b/c.py', '/a/**/*.py', None, True),
-        ('/a/b/c.py', '/a/**/*.py', 'foo', True),
-    ],
-)
-def test_files_matches_patterns(files, patterns, rootpath, result):
-    assert files_matches_patterns(files, patterns, rootpath) == result