Merge branch 'feature/add-shell-autocompletion' into 'master'

idf.py: Enable autocompletion for command idf.py

Closes IDF-1540

See merge request espressif/esp-idf!10742

docs/en/api-guides/build-system.rst

@@ -89,6 +89,12 @@ Multiple ``idf.py`` commands can be combined into one. For example, ``idf.py -p
 
 For commands that are not known to ``idf.py`` an attempt to execute them as a build system target will be made.
 
+The command ``idf.py`` supports `shell autocompletion <https://click.palletsprojects.com/bashcomplete/>`_ for bash, zsh and fish shells.
+In order to make `shell autocompletion <https://click.palletsprojects.com/bashcomplete/>`_ supported, please make sure you have at least Python 3.5 and `click <https://click.palletsprojects.com/>`_ 7.1 or newer (:ref:`see also <get-started-get-prerequisites>`).
+To enable autocompletion for ``idf.py`` use the ``export`` command (:ref:`see this <get-started-export>`).
+Autocompletion is initiated by pressing the TAB key. 
+Type "idf.py -" and press the TAB key to autocomplete options. The autocomplete support for PowerShell is planned in the future.
+
 .. note:: The environment variables ``ESPPORT`` and ``ESPBAUD`` can be used to set default values for the ``-p`` and ``-b`` options, respectively. Providing these options on the command line overrides the default.
 
 .. _idf.py-size:
@@ -124,7 +130,6 @@ Start a new project
 
 Use the command ``idf.py create-project`` for starting a new project. Execute ``idf.py create-project --help`` for more information.
 
-
 Example:
 
 .. code-block:: bash
@@ -320,7 +325,7 @@ Renaming ``main`` component
 The build system provides special treatment to the ``main`` component. It is a component that gets automatically added to the build provided
 that it is in the expected location, PROJECT_DIR/main. All other components in the build are also added as its dependencies,
 saving the user from hunting down dependencies and providing a build that works right out of the box. Renaming the ``main`` component
-causes the loss of these behind-the-scences heavy lifting, requiring the user to specify the location of the newly renamed component
+causes the loss of these behind-the-scenes heavy lifting, requiring the user to specify the location of the newly renamed component
 and manually specifying its dependencies. Specifically, the steps to renaming ``main`` are as follows:
 
 1. Rename ``main`` directory.
@@ -1054,7 +1059,7 @@ To select the target before building the project, use ``idf.py set-target <targe
     2. removing the sdkconfig file (``mv sdkconfig sdkconfig.old``)
     3. configuring the project with the new target (``idf.py -DIDF_TARGET=esp32 reconfigure``)
 
-It is also possible to pass the desired ``IDF_TARGET`` as an environement variable (e.g. ``export IDF_TARGET=esp32s2``) or as a CMake variable (e.g. ``-DIDF_TARGET=esp32s2`` argument to CMake or idf.py). Setting the environment variable is a convenient method if you mostly work with one type of the chip.
+It is also possible to pass the desired ``IDF_TARGET`` as an environment variable (e.g. ``export IDF_TARGET=esp32s2``) or as a CMake variable (e.g. ``-DIDF_TARGET=esp32s2`` argument to CMake or idf.py). Setting the environment variable is a convenient method if you mostly work with one type of the chip.
 
 To specify the _default_ value of ``IDF_TARGET`` for a given project, add ``CONFIG_IDF_TARGET`` value to ``sdkconfig.defaults``. For example, ``CONFIG_IDF_TARGET="esp32s2"``. This value will be used if ``IDF_TARGET`` is not specified by other method: using an environment variable, CMake variable, or ``idf.py set-target`` command.
 
@@ -1173,7 +1178,7 @@ It is possible to do so by using the :ref:`build system APIs provided<cmake_buil
   include($ENV{IDF_PATH}/tools/cmake/idf.cmake)
 
   # Include ESP-IDF components in the build, may be thought as an equivalent of 
-  # add_subdirectory() but with some additional procesing and magic for ESP-IDF build
+  # add_subdirectory() but with some additional processing and magic for ESP-IDF build
   # specific build processes.
   idf_build_process(esp32)
   

docs/en/get-started/index.rst

@@ -259,6 +259,8 @@ or with Windows PowerShell
 
     .$HOME/esp/esp-idf/export.ps1
 
+.. _get-started-export:
+
 Linux and macOS
 ~~~~~~~~~~~~~~~
 
@@ -268,6 +270,12 @@ In the terminal where you are going to use ESP-IDF, run:
 
     . $HOME/esp/esp-idf/export.sh
 
+or for fish (supported only since fish version 3.0.0):
+
+.. code-block:: bash
+
+    . $HOME/esp/esp-idf/export.fish
+
 Note the space between the leading dot and the path!
 
 If you plan to use esp-idf frequently, you can create an alias for executing ``export.sh``:

docs/en/get-started/linux-setup-scratch.rst

@@ -19,11 +19,11 @@ To compile with ESP-IDF you need to get the following packages:
 
 - Ubuntu and Debian::
 
-    sudo apt-get install git wget libncurses-dev flex bison gperf python python-pip python-setuptools python-serial python-click python-cryptography python-future python-pyparsing python-pyelftools cmake ninja-build ccache libffi-dev libssl-dev dfu-util
+    sudo apt-get install git wget libncurses-dev flex bison gperf python python-pip python-setuptools python-serial python-cryptography python-future python-pyparsing python-pyelftools cmake ninja-build ccache libffi-dev libssl-dev dfu-util
 
 - Arch::
 
-    sudo pacman -S --needed gcc git make ncurses flex bison gperf python-pyserial python-click python-cryptography python-future python-pyparsing python-pyelftools cmake ninja ccache dfu-util
+    sudo pacman -S --needed gcc git make ncurses flex bison gperf python-pyserial python-cryptography python-future python-pyparsing python-pyelftools cmake ninja ccache dfu-util
 
 .. note::
     CMake version 3.5 or newer is required for use with ESP-IDF. Older Linux distributions may require updating, enabling of a "backports" repository, or installing of a "cmake3" package rather than "cmake".

docs/zh_CN/get-started/linux-setup-scratch.rst

@@ -17,11 +17,11 @@
 
 - Ubuntu 和 Debian::
 
-    sudo apt-get install git wget libncurses-dev flex bison gperf python python-pip python-setuptools python-serial python-click python-cryptography python-future python-pyparsing python-pyelftools cmake ninja-build ccache libffi-dev libssl-dev dfu-util
+    sudo apt-get install git wget libncurses-dev flex bison gperf python python-pip python-setuptools python-serial python-cryptography python-future python-pyparsing python-pyelftools cmake ninja-build ccache libffi-dev libssl-dev dfu-util
 
 - Arch::
 
-    sudo pacman -S --needed gcc git make ncurses flex bison gperf python-pyserial python-click python-cryptography python-future python-pyparsing python-pyelftools cmake ninja ccache dfu-util
+    sudo pacman -S --needed gcc git make ncurses flex bison gperf python-pyserial python-cryptography python-future python-pyparsing python-pyelftools cmake ninja ccache dfu-util
 
 .. note::
     使用 ESP-IDF 需要 CMake 3.5 或以上版本。较早版本的 Linux 可能需要升级才能向后移植仓库，或安装 "cmake3" 软件包，而不是安装 "cmake"。

export.fish

@@ -63,4 +63,6 @@ end
 
 idf_export_main
 
+eval (env _IDF.PY_COMPLETE=source_fish idf.py)
+
 set -e idf_export_main

export.sh

@@ -136,7 +136,19 @@ idf_export_main() {
     echo ""
 }
 
+enable_autocomplete() {
+    if [ -n "$ZSH_VERSION" ]
+    then
+        autoload -Uz compinit && compinit -u
+        eval "$(env _IDF.PY_COMPLETE=source_zsh idf.py)"
+    else
+        eval "$(env _IDF.PY_COMPLETE=source_bash idf.py)"
+    fi
+}
+
 idf_export_main
+enable_autocomplete
 
 unset realpath_int
 unset idf_export_main
+

tools/ci/config/host-test.yml

@@ -291,3 +291,17 @@ test_docs:
     - ${IDF_PATH}/tools/ci/multirun_with_pyenv.sh -p 3.6.10 ./test_docs.py
     - ${IDF_PATH}/tools/ci/multirun_with_pyenv.sh -p 3.6.10 ./test_sphinx_idf_extensions.py
 
+
+test_autocomlete:
+    stage: host_test
+    image: $CI_DOCKER_REGISTRY/linux-shells:1
+    tags:
+        - host_test
+    dependencies: []
+    artifacts:
+        when: on_failure
+        paths:
+            - ${IDF_PATH}/*.out
+        expire_in: 1 week
+    script:
+        - ${IDF_PATH}/tools/ci/multirun_with_pyenv.sh ${IDF_PATH}/tools/ci/test_autocomplete.py

tools/ci/executable-list.txt

@@ -58,6 +58,7 @@ tools/ci/mirror-submodule-update.sh
 tools/ci/multirun_with_pyenv.sh
 tools/ci/normalize_clangtidy_path.py
 tools/ci/push_to_github.sh
+tools/ci/test_autocomplete.py
 tools/ci/test_build_system.sh
 tools/ci/test_build_system_cmake.sh
 tools/ci/test_configure_ci_environment.sh

tools/ci/test_autocomplete.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+import os
+import pexpect
+import unittest
+
+
+class Test(unittest.TestCase):
+    def test_fish(self):
+        os.environ["TERM"] = "vt100"
+        child = pexpect.spawn("fish -i")
+        child.logfile = open(os.environ["IDF_PATH"] + "/fish.out", "wb")
+        child.sendline(
+            '. {$IDF_PATH}/export.fish >> {$IDF_PATH}/debug.out')
+        child.send("idf.py \t\t")
+        result = child.expect(["all.*app.*app-flash.*bootloader.*", pexpect.EOF, pexpect.TIMEOUT], timeout=5)
+        self.assertEqual(result, 0, "Autocompletion for idf.py failed in fish!")
+        result = child.expect(["bootloader-flash.*build-system-targets.*clean.*", pexpect.EOF, pexpect.TIMEOUT],
+                              timeout=5)
+        self.assertEqual(result, 0, "Autocompletion for idf.py failed in fish!")
+
+    def test_bash(self):
+        os.environ["TERM"] = "xterm-256color"
+        child = pexpect.spawn("bash -i")
+        child.logfile = open(os.environ["IDF_PATH"] + "/bash.out", "wb")
+        child.sendline(
+            '. ${IDF_PATH}/export.sh >> ${IDF_PATH}/debug.out')
+        child.send("idf.py \t\t")
+        result = child.expect(
+            ["all.*app.*app-flash.*bootloader.*bootloader-flash.*build-system-targets.*clean.*", pexpect.EOF,
+             pexpect.TIMEOUT], timeout=5)
+        self.assertEqual(result, 0, "Autocompletion for idf.py failed in bash!")
+
+    def test_zsh(self):
+        child = pexpect.spawn("zsh -i")
+        child.logfile = open(os.environ["IDF_PATH"] + "/zsh.out", "wb")
+        child.sendline(
+            '. ${IDF_PATH}/export.sh >> ${IDF_PATH}/debug.out ')
+        child.send("idf.py \t")
+        result = child.expect(
+            ["all.*app.*app-flash.*bootloader.*bootloader-flash.*build-system-targets.*clean.*", pexpect.EOF,
+             pexpect.TIMEOUT], timeout=5)
+        self.assertEqual(result, 0, "Autocompletion for idf.py failed in zsh!")
+
+
+if __name__ == '__main__':
+    unittest.main()

tools/idf.py

@@ -38,7 +38,7 @@ from importlib import import_module
 from pkgutil import iter_modules
 
 # pyc files remain in the filesystem when switching between branches which might raise errors for incompatible
-# idf.py extentions. Therefore, pyc file generation is turned off:
+# idf.py extensions. Therefore, pyc file generation is turned off:
 sys.dont_write_bytecode = True
 
 from idf_py_actions.errors import FatalError  # noqa: E402
@@ -53,7 +53,7 @@ os.environ["PYTHON"] = sys.executable
 
 # Name of the program, normally 'idf.py'.
 # Can be overridden from idf.bat using IDF_PY_PROGRAM_NAME
-PROG = os.getenv("IDF_PY_PROGRAM_NAME", sys.argv[0])
+PROG = os.getenv("IDF_PY_PROGRAM_NAME", "idf.py")
 
 
 def check_environment():
@@ -184,7 +184,7 @@ def init_cli(verbose_output=None):
             return ("Deprecated! " + text) if self.deprecated else text
 
     def check_deprecation(ctx):
-        """Prints deprectation warnings for arguments in given context"""
+        """Prints deprecation warnings for arguments in given context"""
         for option in ctx.command.params:
             default = () if option.multiple else option.default
             if isinstance(option, Option) and option.deprecated and ctx.params[option.name] != default:
@@ -533,7 +533,7 @@ def init_cli(verbose_output=None):
                 print(
                     "WARNING: Command%s found in the list of commands more than once. " %
                     ("s %s are" % dupes if len(dupplicated_tasks) > 1 else " %s is" % dupes) +
-                    "Only first occurence will be executed.")
+                    "Only first occurrence will be executed.")
 
             for task in tasks:
                 # Show help and exit if help is in the list of commands
@@ -640,11 +640,11 @@ def init_cli(verbose_output=None):
             "ignore_unknown_options": True
         },
     )
-    @click.option("-C", "--project-dir", default=os.getcwd())
+    @click.option("-C", "--project-dir", default=os.getcwd(), type=click.Path())
     def parse_project_dir(project_dir):
         return realpath(project_dir)
-
-    project_dir = parse_project_dir(standalone_mode=False)
+    # Set `complete_var` to not existing environment variable name to prevent early cmd completion
+    project_dir = parse_project_dir(standalone_mode=False, complete_var="_IDF.PY_COMPLETE_NOT_EXISTING")
 
     all_actions = {}
     # Load extensions from components dir
@@ -660,7 +660,7 @@ def init_cli(verbose_output=None):
     extensions = {}
     for directory in extension_dirs:
         if directory and not os.path.exists(directory):
-            print('WARNING: Directroy with idf.py extensions doesn\'t exist:\n    %s' % directory)
+            print('WARNING: Directory with idf.py extensions doesn\'t exist:\n    %s' % directory)
             continue
 
         sys.path.append(directory)
@@ -709,7 +709,8 @@ def init_cli(verbose_output=None):
 def main():
     checks_output = check_environment()
     cli = init_cli(verbose_output=checks_output)
-    cli(sys.argv[1:], prog_name=PROG)
+    # the argument `prog_name` must contain name of the file - not the absolute path to it!
+    cli(sys.argv[1:], prog_name=PROG, complete_var="_IDF.PY_COMPLETE")
 
 
 def _valid_unicode_config():

tools/idf_py_actions/serial_ext.py

@@ -172,13 +172,13 @@ def action_extensions(base_actions, project_path):
                     port, {
                         "names": ["--print-filter", "--print_filter"],
                         "help":
-                        ("Filter monitor output.\n"
+                        ("Filter monitor output. "
                          "Restrictions on what to print can be specified as a series of <tag>:<log_level> items "
                          "where <tag> is the tag string and <log_level> is a character from the set "
                          "{N, E, W, I, D, V, *} referring to a level. "
                          'For example, "tag1:W" matches and prints only the outputs written with '
                          'ESP_LOGW("tag1", ...) or at lower verbosity level, i.e. ESP_LOGE("tag1", ...). '
-                         'Not specifying a <log_level> or using "*" defaults to Verbose level.\n'
+                         'Not specifying a <log_level> or using "*" defaults to Verbose level. '
                          'Please see the IDF Monitor section of the ESP-IDF documentation '
                          'for a more detailed description and further examples.'),
                         "default":
@@ -187,7 +187,7 @@ def action_extensions(base_actions, project_path):
                         "names": ["--monitor-baud", "-B"],
                         "type":
                         click.INT,
-                        "help": ("Baud rate for monitor.\n"
+                        "help": ("Baud rate for monitor. "
                                  "If this option is not provided IDF_MONITOR_BAUD and MONITORBAUD "
                                  "environment variables and project_description.json in build directory "
                                  "(generated by CMake from project's sdkconfig) "
@@ -195,7 +195,7 @@ def action_extensions(base_actions, project_path):
                     }, {
                         "names": ["--encrypted", "-E"],
                         "is_flag": True,
-                        "help": ("Enable encrypted flash targets.\n"
+                        "help": ("Enable encrypted flash targets. "
                                  "IDF Monitor will invoke encrypted-flash and encrypted-app-flash targets "
                                  "if this option is set. This option is set by default if IDF Monitor was invoked "
                                  "together with encrypted-flash or encrypted-app-flash target."),