Tools: Allow custom Python installation path with IDF_PYTHON_ENV_PATH

IDF_PYTHON_ENV_PATH is the path where the Python environment is created
and used. By default it is inside IDF_TOOLS_PATH. IDF_PYTHON_ENV_PATH
was exported by idf_tools.py but was not imported back. This fixes the
issue and ESP-IDF will honor the value of IDF_PYTHON_ENV_PATH.

Closes https://github.com/espressif/esp-idf/issues/10489

docs/en/api-guides/tools/idf-tools.rst

@@ -40,7 +40,7 @@ Inside ``IDF_TOOLS_PATH``, the scripts performing tools installation create the
 - ``dist`` — where the archives of the tools are downloaded.
 - ``tools`` — where the tools are extracted. The tools are extracted into subdirectories: ``tools/TOOL_NAME/VERSION/``. This arrangement allows different versions of tools to be installed side by side.
 - ``idf-env.json`` — user install options (targets, features) are stored in this file. Targets are selected chip targets for which tools are installed and kept up-to-date. Features determine the Python package set which should be installed. These options will be discussed later.
-- ``python_env`` —  not tools related; virtual Python environments are installed in the sub-directories.
+- ``python_env`` —  not tools related; virtual Python environments are installed in the sub-directories. Note that the Python environment directory can be placed elsewhere by setting the ``IDF_PYTHON_ENV_PATH`` environment variable.
 - ``espidf.constraints.*.txt`` — one constraint file for each ESP-IDF release containing Python package version requirements.
 
 GitHub Assets Mirror
@@ -113,7 +113,7 @@ Any mirror server can be used provided the URL matches the ``github.com`` downlo
 
 * ``check``: For each tool, checks whether the tool is available in the system path and in ``IDF_TOOLS_PATH``.
 
-* ``install-python-env``: Create a Python virtual environment in the ``${IDF_TOOLS_PATH}/python_env`` directory and install there the required Python packages. An optional ``--features`` argument allows one to specify a comma-separated list of features to be added or removed. Feature that begins with ``-`` will be removed and features with ``+`` or without any sign will be added. Example syntax for removing feature ``XY`` is ``--features=-XY`` and for adding ``--features=+XY`` or ``--features=XY``. If both removing and adding options are provided with the same feature, no operation is performed. For each feature a requirements file must exist. For example, feature ``XY`` is a valid feature if ``${IDF_PATH}/tools/requirements/requirements.XY.txt`` is an existing file with a list of Python packages to be installed. There is one mandatory ``core`` feature ensuring core functionality of ESP-IDF (build, flash, monitor, debug in console). There can be an arbitrary number of optional features. The selected list of features is stored in ``idf-env.json``. The requirement files contain a list of the desired Python packages to be installed and ``espidf.constraints.*.txt`` downloaded from https://dl.espressif.com and stored in ``${IDF_TOOLS_PATH}`` the package version requirements for a given ESP-IDF version. Althought it is not recommended, the download and use of constraint files can be disabled with the ``--no-constraints`` argument or setting the ``IDF_PYTHON_CHECK_CONSTRAINTS`` environment variable to ``no``.
+* ``install-python-env``: Create a Python virtual environment in the ``${IDF_TOOLS_PATH}/python_env`` directory (or directly in the directory set by the ``IDF_PYTHON_ENV_PATH`` environment variable) and install there the required Python packages. An optional ``--features`` argument allows one to specify a comma-separated list of features to be added or removed. Feature that begins with ``-`` will be removed and features with ``+`` or without any sign will be added. Example syntax for removing feature ``XY`` is ``--features=-XY`` and for adding ``--features=+XY`` or ``--features=XY``. If both removing and adding options are provided with the same feature, no operation is performed. For each feature a requirements file must exist. For example, feature ``XY`` is a valid feature if ``${IDF_PATH}/tools/requirements/requirements.XY.txt`` is an existing file with a list of Python packages to be installed. There is one mandatory ``core`` feature ensuring core functionality of ESP-IDF (build, flash, monitor, debug in console). There can be an arbitrary number of optional features. The selected list of features is stored in ``idf-env.json``. The requirement files contain a list of the desired Python packages to be installed and ``espidf.constraints.*.txt`` downloaded from https://dl.espressif.com and stored in ``${IDF_TOOLS_PATH}`` the package version requirements for a given ESP-IDF version. Althought it is not recommended, the download and use of constraint files can be disabled with the ``--no-constraints`` argument or setting the ``IDF_PYTHON_CHECK_CONSTRAINTS`` environment variable to ``no``.
 
 * ``check-python-dependencies``: Checks if all required Python packages are installed. Packages from ``${IDF_PATH}/tools/requirements/requirements.*.txt`` files selected by the feature list of ``idf-env.json`` are checked with the package versions specified in the ``espidf.constraints.*.txt`` file. The constraint file is downloaded with ``install-python-env`` command. The use of constraints files can be disabled similarly to the ``install-python-env`` command.
 

tools/idf_tools.py

@@ -1357,8 +1357,8 @@ def get_python_env_path() -> Tuple[str, str, str, str]:
     python_ver_major_minor = '{}.{}'.format(sys.version_info.major, sys.version_info.minor)
 
     idf_version = get_idf_version()
-    idf_python_env_path = os.path.join(global_idf_tools_path or '', 'python_env',
-                                       'idf{}_py{}_env'.format(idf_version, python_ver_major_minor))
+    idf_python_env_path = os.getenv('IDF_PYTHON_ENV_PATH') or os.path.join(global_idf_tools_path or '', 'python_env',
+                                                                           'idf{}_py{}_env'.format(idf_version, python_ver_major_minor))
 
     python_exe, subdir = get_python_exe_and_subdir()
     idf_python_export_path = os.path.join(idf_python_env_path, subdir)

tools/test_idf_tools/test_idf_tools_python_env.py

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
+# SPDX-FileCopyrightText: 2022-2023 Espressif Systems (Shanghai) CO LTD
 # SPDX-License-Identifier: Apache-2.0
 
 import inspect
@@ -6,6 +6,7 @@ import os
 import shutil
 import subprocess
 import sys
+import tempfile
 import unittest
 from typing import List
 
@@ -80,5 +81,25 @@ class TestPythonInstall(unittest.TestCase):
         self.assertIn(REQ_CORE, output)
 
 
+class TestCustomPythonPathInstall(TestPythonInstall):
+
+    def setUp(self):  # type: () -> None
+        self.CUSTOM_PYTHON_DIR = tempfile.mkdtemp()
+        self.addCleanup(shutil.rmtree, self.CUSTOM_PYTHON_DIR)
+        os.environ['IDF_PYTHON_ENV_PATH'] = self.CUSTOM_PYTHON_DIR
+
+    def test_default_arguments(self):  # type: () -> None
+        output = self.run_idf_tools(['check-python-dependencies'])
+        self.assertIn(f"{self.CUSTOM_PYTHON_DIR}/bin/python doesn't exist", output)
+        self.assertNotIn(PYTHON_DIR, output)
+
+        output = self.run_idf_tools(['install-python-env'])
+        self.assertIn(self.CUSTOM_PYTHON_DIR, output)
+        self.assertNotIn(PYTHON_DIR, output)
+
+        output = self.run_idf_tools(['check-python-dependencies'])
+        self.assertIn(self.CUSTOM_PYTHON_DIR, output)
+
+
 if __name__ == '__main__':
     unittest.main()