cmake: Generate list of components with dependent items first, use deterministic ordering

Guarantees that a component's project_include.cmake will be called after its dependent components'
project_include.cmake. Because of cycles in the dependency graph, this is less useful than you'd
think but it gives a strong guarantee for any component which is not part of a cycle.

Also applies deterministic ordering (ordering is initialised as COMPONENT_REQUIRES_COMMON then all
COMPONENTS in alphabetical order, but then the sorting by dependencies is applied.)

docs/en/api-guides/build-system-cmake.rst

@@ -407,10 +407,21 @@ Requirements in the build system implementation
 -----------------------------------------------
 
 - Very early in the CMake configuration process, the script ``expand_requirements.cmake`` is run. This script does a partial evaluation of all component CMakeLists.txt files and builds a graph of component requirements (this graph may have cycles). The graph is used to generate a file ``component_depends.cmake`` in the build directory.
-- The main CMake process then includes this file and uses it to determine the list of components to include in the build (internal ``BUILD_COMPONENTS`` variable).
+- The main CMake process then includes this file and uses it to determine the list of components to include in the build (internal ``BUILD_COMPONENTS`` variable). The ``BUILD_COMPONENTS`` variable is sorted so dependencies are listed first, however as the component dependency graph has cycles this cannot be guaranteed for all components. The order should be deterministic given the same set of components and component dependencies.
+- The value of ``BUILD_COMPONENTS`` is logged by CMake as "Component names: "
 - Configuration is then evaluated for the components included in the build.
 - Each component is included in the build normally and the CMakeLists.txt file is evaluated again to add the component libraries to the build.
 
+Component Dependency Order
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The order of components in the ``BUILD_COMPONENTS`` variable determines other orderings during the build:
+
+- Order that :ref:`project_include.cmake` files are included into the project.
+- Order that the list of header paths is generated for compilation (via ``-I`` argument). (Note that for a given component's source files, only that component's dependency's header paths are passed to the compiler.)
+- Order that component object archives are passed to the linker (note that the build system also passes ``--start-group`` and ``--end-group`` to the linker to allow cycles in linker dependencies, however the basic order is determined by ``BUILD_COMPONENTS``.
+
+
 Build Process Internals
 =======================
 
@@ -434,7 +445,7 @@ The custom ``project()`` function performs the following steps:
 - Sets the `CMAKE_TOOLCHAIN_FILE`_ variable to the ESP-IDF toolchain file with the Xtensa ESP32 toolchain.
 - Declare the actual cmake-level project by calling the `CMake project function <cmake project_>`_.
 - Load the git version. This includes some magic which will automatically re-run CMake if a new revision is checked out in git. See `File Globbing & Incremental Builds`_.
-- Include ``project_include.cmake`` files from any components which have them.
+- Include :ref:`project_include.cmake` files from any components which have them.
 - Add each component to the build. Each component CMakeLists file calls ``register_component``, calls the CMake `add_library <cmake add_library_>`_ function to add a library and then adds source files, compile options, etc.
 - Add the final app executable to the build.
 - Go back and add inter-component dependencies between components (ie adding the public header directories of each component to each other component).
@@ -462,6 +473,8 @@ If you don't want this behaviour, it can be disabled by passing ``--no-warnings`
 Overriding Parts of the Project
 -------------------------------
 
+.. _project_include.cmake:
+
 project_include.cmake
 ^^^^^^^^^^^^^^^^^^^^^
 
@@ -475,6 +488,8 @@ Unlike component ``CMakeLists.txt`` files, when including a ``project_include.cm
 
 Note that ``project_include.cmake`` isn't necessary for the most common component uses - such as adding include directories to the project, or ``LDFLAGS`` to the final linking step. These values can be customised via the ``CMakeLists.txt`` file itself. See `Optional Project Variables`_ for details.
 
+``project_include.cmake`` files are included in the order given in ``BUILD_COMPONENTS`` variable (as logged by CMake). This means that a component's ``project_include.cmake`` file will be included after it's all dependencies' ``project_include.cmake`` files, unless both components are part of a dependency cycle. This is important if a ``project_include.cmake`` file relies on variables set by another component. See also :ref:`above<component-requirements-implementation>`.
+
 Take great care when setting variables or targets in a ``project_include.cmake`` file. As the values are included into the top-level project CMake pass, they can influence or break functionality across all components!
 
 KConfig.projbuild

tools/cmake/project.cmake

@@ -47,15 +47,13 @@ macro(project name)
     # Set global variables used by rest of the build
     idf_set_global_variables()
 
-    # Establish dependencies for components in the build
-    # (this happens before we even generate config...)
-    if(COMPONENTS)
-        # Make sure if an explicit list of COMPONENTS is given, it contains the "common" component requirements
-        # (otherwise, if COMPONENTS is empty then all components will be included in the build.)
-        set(COMPONENTS "${COMPONENTS} ${COMPONENT_REQUIRES_COMMON}")
-    endif()
+    # Sort the components list, as it may be found via filesystem
+    # traversal and therefore in a non-deterministic order
+    list(SORT COMPONENTS)
+
     execute_process(COMMAND "${CMAKE_COMMAND}"
         -D "COMPONENTS=${COMPONENTS}"
+        -D "COMPONENT_REQUIRES_COMMON=${COMPONENT_REQUIRES_COMMON}"
         -D "DEPENDENCIES_FILE=${CMAKE_BINARY_DIR}/component_depends.cmake"
         -D "COMPONENT_DIRS=${COMPONENT_DIRS}"
         -D "BOOTLOADER_BUILD=${BOOTLOADER_BUILD}"

tools/cmake/scripts/expand_requirements.cmake

@@ -4,6 +4,8 @@
 # Parameters:
 # - COMPONENTS = Space-separated list of initial components to include in the build.
 #   Can be empty, in which case all components are in the build.
+# - COMPONENT_REQUIRES_COMMON = Components to always include in the build, and treated as dependencies
+#   of all other components.
 # - DEPENDENCIES_FILE = Path of generated cmake file which will contain the expanded dependencies for these
 #   components.
 # - COMPONENT_DIRS = List of paths to search for all components.
@@ -13,6 +15,19 @@
 # components required for the build, and the get_component_requirements() function to return each component's
 # recursively expanded requirements.
 #
+# BUILD_COMPONENTS & BUILD_COMPONENT_PATHS will be ordered in a best-effort way so that dependencies are listed first.
+# (Note that IDF supports cyclic dependencies, and dependencies in a cycle have ordering guarantees.)
+#
+# Determinism:
+#
+# Given the the same list of names in COMPONENTS (regardless of order), and an identical value of
+# COMPONENT_REQUIRES_COMMON, and all the same COMPONENT_REQUIRES & COMPONENT_PRIV_REQUIRES values in
+# each component, then the output of BUILD_COMPONENTS should always be in the same
+# order.
+#
+# BUILD_COMPONENT_PATHS will be in the same component order as BUILD_COMPONENTS, even if the
+# actual component paths are different due to different paths.
+#
 # TODO: Error out if a component requirement is missing
 cmake_minimum_required(VERSION 3.5)
 include("${IDF_PATH}/tools/cmake/utilities.cmake")
@@ -26,6 +41,8 @@ if(NOT COMPONENT_DIRS)
 endif()
 spaces2list(COMPONENT_DIRS)
 
+spaces2list(COMPONENT_REQUIRES_COMMON)
+
 function(debug message)
     if(DEBUG)
         message(STATUS "${message}")
@@ -123,10 +140,11 @@ endfunction()
 # also invoking the components to call register_component() above,
 # which will add per-component global properties with dependencies, etc.
 function(expand_component_requirements component)
-    get_property(build_components GLOBAL PROPERTY BUILD_COMPONENTS)
-    if(${component} IN_LIST build_components)
-        return()  # already added this component
+    get_property(seen_components GLOBAL PROPERTY SEEN_COMPONENTS)
+    if(${component} IN_LIST seen_components)
+        return()  # already added, or in process of adding, this component
     endif()
+    set_property(GLOBAL APPEND PROPERTY SEEN_COMPONENTS ${component})
 
     find_component_path("${component}" "${ALL_COMPONENT_PATHS}" component_path)
     debug("Expanding dependencies of ${component} @ ${component_path}")
@@ -142,14 +160,17 @@ function(expand_component_requirements component)
     set(COMPONENT ${component})
     include(${component_path}/CMakeLists.txt)
 
-    set_property(GLOBAL APPEND PROPERTY BUILD_COMPONENT_PATHS ${component_path})
-    set_property(GLOBAL APPEND PROPERTY BUILD_COMPONENTS ${component})
-
     get_property(requires GLOBAL PROPERTY "${component}_REQUIRES")
     get_property(requires_priv GLOBAL PROPERTY "${component}_PRIV_REQUIRES")
-    foreach(req ${requires} ${requires_priv})
+
+    # Recurse dependencies first, so that they appear first in the list (when possible)
+    foreach(req ${COMPONENT_REQUIRES_COMMON} ${requires} ${requires_priv})
         expand_component_requirements(${req})
     endforeach()
+
+    # Now append this component to the full list (after its dependencies)
+    set_property(GLOBAL APPEND PROPERTY BUILD_COMPONENT_PATHS ${component_path})
+    set_property(GLOBAL APPEND PROPERTY BUILD_COMPONENTS ${component})
 endfunction()
 
 
@@ -166,6 +187,7 @@ spaces2list(COMPONENTS)
 debug("ALL_COMPONENT_PATHS ${ALL_COMPONENT_PATHS}")
 debug("ALL_COMPONENTS ${ALL_COMPONENTS}")
 
+set_property(GLOBAL PROPERTY SEEN_COMPONENTS "")  # anti-infinite-recursion
 set_property(GLOBAL PROPERTY BUILD_COMPONENTS "")
 set_property(GLOBAL PROPERTY BUILD_COMPONENT_PATHS "")
 set_property(GLOBAL PROPERTY COMPONENTS_NOT_FOUND "")