esp-idf/tools/test_build_system

Tools & Build System Tests

This directory contains tests for the build system and build-related tools. These tests are meant to be used both by developers and in CI. Please check the sections below for details on:

• Running the tests locally
• Debugging test failures
• Adding new tests
• Fixtures and helper functions

Running the tests locally

1. Install pytest using install.{sh,bat,ps1,fish} --enable-pytest.
2. Activate the IDF shell environment using export.{sh,bat,ps1,fish}.

3. To run all the tests, go to $IDF_PATH/tools/test_build_system directory, then run:

   pytest
   

4. To run one specific test, use -k flag of pytest, for example

   pytest -k test_compile_commands_json_updated_by_reconfigure
   

5. To speed up the builds you can install Ccache and set the following environment variables:

   export IDF_CCACHE_ENABLE=1
   export CCACHE_NOHASHDIR=1
   

Debugging test failures

If you are working on a bug fix or a feature and one of the tests starts to fail, you should try to reproduce the failure locally.

1. Find the name of the failing test in the CI job log
2. Follow the steps in the section above to run that one test
3. By default, the fixtures which create temporary directories will remove them after the test. To prevent the directories from being removed, run pytest with --work-dir /some/path flag. The temporary directories will be created under /some/path, and you will be able to inspect them once the test fails.
4. You can increase the logging level to see the commands being executed by the test by running pytest with --log-cli-level DEBUG argument.

Adding new tests

1. When adding a new test, think of the developer who might have to run this test locally.
   • Avoid adding tests which take a long time to run. Running the entire test suite should be possible!
   • Remember that developers run these tests in their IDF work directories. Be careful with destructive actions, especially removing directories recursively. Developers might have untracked files in the directory you are removing! Prefer using the idf_copy fixture to make a copy of the IDF directory, when doing some modifications to IDF source. If this is too expensive, do the modification in place but make sure to clean up the changes you perform using a try/finally block. This especially applies to the newly created files.
2. Read through the test cases and try to find a test which does something similar to what you need to test. This will usually be a good starting point. Also read through the section below, which explains fixtures and utility functions.
3. The tests need to run on Windows, Linux and macOS. Avoid calling OS-specific programs such as sed or awk in tests. If you need to perform some complex file modification in the test case, consider writing a Python helper function for that.

Fixtures and helper functions

If you aren't yet familiar with Pytest fixtures, please take a few moments and read a Pytest tutorial or watch the training, before moving on to the next section.

test_app_copy fixture

This fixture selects the app (inside IDF) to be used by the test and copies this app to a temporary directory, recursively. The working directory is set to the root of the copied app. The directory is removed once the test is finished.

def test_something(test_app_copy):
    assert test_app_copy == os.getcwd()
    # the current working directory now contains the copy of the test app

If the test case doesn't use the test_app_copy argument, pylint will typically warn about an unused argument, even if the fixture is actually used. To avoid the warning, use the following pattern:

@pytest.mark.usefixtures('test_app_copy')
def test_something(idf_py):
    idf_py('build')

By default, the source app is tools/test_build_system/build_test_app and the destination directory name is derived from the test case name. (See more about this test app here.) This can be overridden using a @pytest.mark.test_app_copy decorator, as shown below. The first argument is the path of the source app. The second argument is the name of the temporary directory to create. The second argument is optional, it is mostly useful to test handling of special characters (such as spaces) in the path.

@pytest.mark.test_app_copy('examples/get-started/blink', 'custom dir name')
@pytest.mark.usefixtures('test_app_copy')
def test_something():
    pass

idf_py fixture

This fixture runs idf.py with IDF environment set up.

@pytest.mark.usefixtures('test_app_copy')
def test_something(idf_py)
    # 1. 'test_app_copy' fixture has copied the test app into a temporary directory and
    #    set the working directory there.
    # 2. 'idf_py' fixture is a function which calls idf.py:
    idf_py('fullclean')

    # 3. It accepts multiple arguments and returns a subprocess.CompletedProcess
    #    instance. It can be used to check the process output.
    output = idf_py('-DIDF_TARGET=esp32c3', 'reconfigure')
    assert 'CONFIG_IDF_TARGET="esp32c3"' in Path('sdkconfig').read_text()
    assert 'Building ESP-IDF components for target esp32c3' in output.stdout

    # 4. Raises subprocess.CalledProcessError on failure
    with(pytest.raises(subprocess.CalledProcessError)) as exc_info:
        idf_py('unknown_command')
    assert 'command "unknown_command" is not known to idf.py' in exc_info.value.stderr

default_idf_env fixture

Returns a dictionary of environment variables required for the IDF build environment. It is similar to the output of env command after running the export script.

@pytest.mark.usefixtures('test_app_copy')
def test_cmake(default_idf_env):
    with tempfile.TemporaryDirectory() as build_dir:
        # default_idf_env can be passed to subprocess APIs to run other tools
        subprocess.run(['cmake', '-B', build_dir, '.'], env=default_idf_env, check=True)

Note, default_idf_env sets up the environment based on the IDF_PATH environment variable set before launching pytest.

idf_copy fixture

Copies IDF from IDF_PATH into a new temporary directory. @pytest.mark.idf_copy('name prefix') can be used to specify the name prefix of the temporary directory.

For the duration of the test, IDF_PATH environment variable is set to the newly created copy.

@pytest.mark.usefixtures('test_app_copy')
def test_idf_copy(idf_copy):
    # idf_copy is the temporary IDF copy.
    # For example, we can check if idf.py build can work without the .git directory:
    shutil.rmtree(os.path.join(idf_copy, '.git'), ignore_errors=True)
    # Note that we can't use idf_py fixture, since it uses the default IDF path.
    # We can use 'get_idf_build_env' with 'run_idf_py', instead:
    env = get_idf_build_env(idf_copy)
    run_idf_py('build', env=env)

Build snapshots

get_snapshot(list_of_globs) function takes a list of glob expressions, finds the files matching these expressions, and returns a Snapshot instance. Snapshot instances record file names and their modification timestamps. Two Snapshot instances can be compared using assert_same and assert_different methods:

@pytest.mark.usefixtures('test_app_copy')
def test_build_jsons_updated_by_reconfigure(idf_py):
    globs = ['build/*.json']
    
    idf_py('reconfigure')
    snapshot_1 = get_snapshot(globs)
    snapshot_2 = get_snapshot(globs)
    snapshot_2.assert_same(snapshot_1)

    idf_py('reconfigure')
    snapshot_3 = get_snapshot(globs)
    snapshot_3.assert_different(snapshot_2)

Helper functions for file modifications

A few extra functions are provided to make simple file modifications:

• append_to_file(filename: typing.Union[str, Path], what: str) -> None — appends the given string to a file.
• replace_in_file(filename: typing.Union[str, Path], search: str, replace: str) -> None — searches the file for occurrences of the string search and replaces all of them with replace, then writes the result back to the file.

Application under test

Most build system tests should use the included build_test_app, if possible.

Using other test apps and examples for the purpose of testing the build system is okay as long as you keep the following in mind:

• Don't use the build system tests to compile examples or test apps under a particular combination of sdkconfig options. Use the sdkconfig.ci.* files for that, instead.
• Examples or test apps may be changed, renamed or removed. If you add a dependency on another example or a test app, your test case might need to be rewritten if someone has to modify or remove the example or a test app your test case depends on.

For convenience, the build_test_app app included here provides several placeholders which can be modified using the replace_in_file function. You can find the placeholders by running grep -r placeholder_ build_test_app.