Merge branch 'feature/cmake_confserver' into 'feature/cmake'

cmake: Add JSON configuration server for external config tool integration

See merge request idf/esp-idf!2410

.gitlab-ci.yml

@@ -250,7 +250,7 @@ test_nvs_on_host:
   stage: test
   image: $CI_DOCKER_REGISTRY/esp32-ci-env$BOT_DOCKER_IMAGE_TAG
   tags:
-    - nvs_host_test
+    - host_test
   dependencies: []
   script:
     - cd components/nvs_flash/test_nvs_host
@@ -260,7 +260,7 @@ test_nvs_coverage:
   stage: test
   image: $CI_DOCKER_REGISTRY/esp32-ci-env$BOT_DOCKER_IMAGE_TAG
   tags:
-    - nvs_host_test
+    - host_test
   dependencies: []
   artifacts:
     paths:
@@ -288,7 +288,7 @@ test_wl_on_host:
   stage: test
   image: $CI_DOCKER_REGISTRY/esp32-ci-env$BOT_DOCKER_IMAGE_TAG
   tags:
-    - wl_host_test
+    - host_test
   artifacts:
     paths:
       - components/wear_levelling/test_wl_host/coverage_report.zip
@@ -301,16 +301,25 @@ test_multi_heap_on_host:
   stage: test
   image: $CI_DOCKER_REGISTRY/esp32-ci-env$BOT_DOCKER_IMAGE_TAG
   tags:
-    - wl_host_test
+    - host_test
   script:
     - cd components/heap/test_multi_heap_host
     - ./test_all_configs.sh
 
+test_confserver:
+  stage: test
+  image: $CI_DOCKER_REGISTRY/esp32-ci-env$BOT_DOCKER_IMAGE_TAG
+  tags:
+    - host_test
+  script:
+    - cd tools/kconfig_new/test
+    - ./test_confserver.py
+
 test_build_system:
   stage: test
   image: $CI_DOCKER_REGISTRY/esp32-ci-env$BOT_DOCKER_IMAGE_TAG
   tags:
-    - build_test
+    - host_test
   dependencies: []
   script:
     - ${IDF_PATH}/tools/ci/test_configure_ci_environment.sh
@@ -323,7 +332,7 @@ test_build_system_cmake:
   stage: test
   image: $CI_DOCKER_REGISTRY/esp32-ci-env$BOT_DOCKER_IMAGE_TAG
   tags:
-    - build_test
+    - host_test
   dependencies: []
   script:
     - ${IDF_PATH}/tools/ci/test_configure_ci_environment.sh

docs/en/api-guides/build-system.rst

@@ -130,6 +130,8 @@ You can also use an IDE with CMake integration. The IDE will want to know the pa
 
 When adding custom non-build steps like "flash" to the IDE, it is recommended to execute ``idf.py`` for these "special" commands.
 
+For more detailed information about integrating ESP-IDF with CMake into an IDE, see `Build System Metadata`_.
+
 .. setting-python-interpreter:
 
 Setting the Python Interpreter
@@ -403,8 +405,8 @@ When creating a project
 Requirements in the build system implementation
 -----------------------------------------------
 
-- Very early in the cmake configuration process, the script ``expand_requirements.cmake`` is run. This script does a partial evaluation of all component CMakeLists.txt files and builds a graph of component requirements (this graph may have cycles). The graph is used to generate a file ``component_depends.cmake`` in the build directory.
-- The main cmake process then includes this file and uses it to determine the list of components to include in the build (internal ``BUILD_COMPONENTS`` variable).
+- Very early in the CMake configuration process, the script ``expand_requirements.cmake`` is run. This script does a partial evaluation of all component CMakeLists.txt files and builds a graph of component requirements (this graph may have cycles). The graph is used to generate a file ``component_depends.cmake`` in the build directory.
+- The main CMake process then includes this file and uses it to determine the list of components to include in the build (internal ``BUILD_COMPONENTS`` variable).
 - Configuration is then evaluated for the components included in the build.
 - Each component is included in the build normally and the CMakeLists.txt file is evaluated again to add the component libraries to the build.
 
@@ -427,12 +429,12 @@ The custom ``project()`` function performs the following steps:
 
 - Evaluates component dependencies and builds the ``BUILD_COMPONENTS`` list of components to include in the build (see :ref:`above<component-requirements-implementation>`).
 - Finds all components in the project (searching ``COMPONENT_DIRS`` and filtering by ``COMPONENTS`` if this is set).
-- Loads the project configuration from the ``sdkconfig`` file and produces a ``cmake`` include file and a C header file, to set config macros. If the project configuration changes, cmake will automatically be re-run to reconfigure the project.
+- Loads the project configuration from the ``sdkconfig`` file and produces a ``sdkconfig.cmake`` file and ``sdkconfig.h`` header, to define config values in CMake and C/C++, respectively. If the project configuration changes, cmake will automatically be re-run to reconfigure the project.
 - Sets the `CMAKE_TOOLCHAIN_FILE`_ variable to the ESP-IDF toolchain file with the Xtensa ESP32 toolchain.
 - Declare the actual cmake-level project by calling the `CMake project function <cmake project_>`_.
-- Load the git version. This includes some magic which will automatically re-run cmake if a new revision is checked out in git. See `File Globbing & Incremental Builds`_.
+- Load the git version. This includes some magic which will automatically re-run CMake if a new revision is checked out in git. See `File Globbing & Incremental Builds`_.
 - Include ``project_include.cmake`` files from any components which have them.
-- Add each component to the build. Each component CMakeLists file calls ``register_component``, calls the cmake `add_library <cmake add_library_>`_ function to add a library and then adds source files, compile options, etc.
+- Add each component to the build. Each component CMakeLists file calls ``register_component``, calls the CMake `add_library <cmake add_library_>`_ function to add a library and then adds source files, compile options, etc.
 - Add the final app executable to the build.
 - Go back and add inter-component dependencies between components (ie adding the public header directories of each component to each other component).
 
@@ -788,6 +790,58 @@ For integration into IDEs and other build systems, when CMake runs the build pro
 - ``flasher_args.json`` contains esptool.py arguments to flash the project's binary files. There are also ``flash_*_args`` files which can be used directly with esptool.py. See `Flash arguments`_.
 - ``CMakeCache.txt`` is the CMake cache file which contains other information about the CMake process, toolchain, etc.
 - ``config/sdkconfig.json`` is a JSON-formatted version of the project configuration values.
+- ``config/kconfig_menus.json`` is a JSON-formatted version of the menus shown in menuconfig, for use in external IDE UIs.
+
+JSON Configuration Server
+-------------------------
+
+.. highlight :: json
+
+A tool called ``confserver.py`` is provided to allow IDEs to easily integrate with the configuration system logic. ``confserver.py`` is designed to run in the background and interact with a calling process by reading and writing JSON over process stdin & stdout.
+
+You can run ``confserver.py`` from a project via ``idf.py confserver`` or ``ninja confserver``, or a similar target triggered from a different build generator.
+
+The config server outputs human-readable errors and warnings on stderr and JSON on stdout. On startup, it will output the full values of each configuration item in the system as a JSON dictionary, and the available ranges for values which are range constrained. The same information is contained in ``sdkconfig.json``::
+
+  {"version": 1, "values": { "ITEM": "value", "ITEM_2": 1024, "ITEM_3": false }, "ranges" : { "ITEM_2" : [ 0, 32768 ] } }
+
+Only visible configuration items are sent. Invisible/disabled items can be parsed from the static ``kconfig_menus.json`` file which also contains the menu structure and other metadata (descriptions, types, ranges, etc.)
+
+The Configuration Server will then wait for input from the client. The client passes a request to change one or more values, as a JSON object followed by a newline::
+
+   {"version": "1", "set": {"SOME_NAME": false, "OTHER_NAME": true } }
+
+The Configuration Server will parse this request, update the project ``sdkconfig`` file, and return a full list of changes::
+
+   {"version": 1, "values": {"SOME_NAME": false, "OTHER_NAME": true , "DEPENDS_ON_SOME_NAME": null}}
+
+Items which are now invisible/disabled will return value ``null``. Any item which is newly visible will return its newly visible current value.
+
+If the range of a config item changes, due to conditional range depending on another value, then this is also sent::
+
+   {"version": 1, "values": {"OTHER_NAME": true }, "ranges" : { "HAS_RANGE" : [ 3, 4 ] } }
+
+If invalid data is passed, an "error" field is present on the object::
+
+    {"version": 1, "values": {}, "error": ["The following config symbol(s) were not visible so were not updated: NOT_VISIBLE_ITEM"]}
+
+By default, no config changes are written to the sdkconfig file. Changes are held in memory until a "save" command is sent::
+
+    {"version": 1, "save": null }
+
+To reload the config values from a saved file, discarding any changes in memory, a "load" command can be sent::
+
+    {"version": 1, "load": null }
+
+The value for both "load" and "save" can be a new pathname, or "null" to load/save the previous pathname.
+
+The response to a "load" command is always the full set of config values and ranges, the same as when the server is initially started.
+
+Any combination of "load", "set", and "save" can be sent in a single command and commands are executed in that order. Therefore it's possible to load config from a file, set some config item values and then save to a file in a single command.
+
+.. note:: The configuration server does not automatically load any changes which are applied externally to the ``sdkconfig`` file. Send a "load" command or restart the server if the file is externally edited.
+
+.. note:: The configuration server does not re-run CMake to regenerate other build files or metadata files after ``sdkconfig`` is updated. This will happen automatically the next time ``CMake`` or ``idf.py`` is run.
 
 .. _gnu-make-to-cmake:
 
@@ -828,7 +882,7 @@ No Longer Available in CMake
 Some features are significantly different or removed in the CMake-based system. The following variables no longer exist in the CMake-based build system:
 
 - ``COMPONENT_BUILD_DIR``: Use ``CMAKE_CURRENT_BINARY_DIR`` instead.
-- ``COMPONENT_LIBRARY``: Defaulted to ``$(COMPONENT_NAME).a``, but the library name could be overriden by the user. The name of the component library can no longer be overriden by the user.
+- ``COMPONENT_LIBRARY``: Defaulted to ``$(COMPONENT_NAME).a``, but the library name could be overriden by the component. The name of the component library can no longer be overriden by the component.
 - ``CC``, ``LD``, ``AR``, ``OBJCOPY``: Full paths to each tool from the gcc xtensa cross-toolchain. Use ``CMAKE_C_COMPILER``, ``CMAKE_C_LINK_EXECUTABLE``, ``CMAKE_OBJCOPY``, etc instead. `Full list here <cmake language variables_>`_.
 - ``HOSTCC``, ``HOSTLD``, ``HOSTAR``: Full names of each tool from the host native toolchain. These are no longer provided, external projects should detect any required host toolchain manually.
 - ``COMPONENT_ADD_LDFLAGS``: Used to override linker flags. Use the CMake `target_link_libraries`_ command instead.
@@ -856,6 +910,7 @@ No Longer Necessary
 
 It is no longer necessary to set ``COMPONENT_SRCDIRS`` if setting ``COMPONENT_SRCS`` (in fact, in the CMake-based system ``COMPONENT_SRCDIRS`` is ignored if ``COMPONENT_SRCS`` is set).
 
+
 .. _esp-idf-template: https://github.com/espressif/esp-idf-template
 .. _cmake: https://cmake.org
 .. _ninja: https://ninja-build.org

tools/ci/executable-list.txt

@@ -37,4 +37,6 @@ tools/cmake/convert_to_cmake.py
 tools/cmake/run_cmake_lint.sh
 tools/idf.py
 tools/kconfig_new/confgen.py
+tools/kconfig_new/confserver.py
+tools/kconfig_new/test/test_confserver.py
 tools/windows/tool_setup/build_installer.sh

tools/cmake/kconfig.cmake

@@ -6,6 +6,7 @@ macro(kconfig_set_variables)
     set(SDKCONFIG_HEADER ${CONFIG_DIR}/sdkconfig.h)
     set(SDKCONFIG_CMAKE ${CONFIG_DIR}/sdkconfig.cmake)
     set(SDKCONFIG_JSON ${CONFIG_DIR}/sdkconfig.json)
+    set(KCONFIG_JSON_MENUS ${CONFIG_DIR}/kconfig_menus.json)
 
     set(ROOT_KCONFIG ${IDF_PATH}/Kconfig)
 
@@ -123,6 +124,15 @@ function(kconfig_process_config)
         VERBATIM
         USES_TERMINAL)
 
+    # Custom target to run confserver.py from the build tool
+    add_custom_target(confserver
+        COMMAND ${CMAKE_COMMAND} -E env
+        "COMPONENT_KCONFIGS=${kconfigs}"
+        "COMPONENT_KCONFIGS_PROJBUILD=${kconfigs_projbuild}"
+        ${IDF_PATH}/tools/kconfig_new/confserver.py --kconfig ${IDF_PATH}/Kconfig --config ${SDKCONFIG}
+        VERBATIM
+        USES_TERMINAL)
+
     # Generate configuration output via confgen.py
     # makes sdkconfig.h and skdconfig.cmake
     #
@@ -131,6 +141,7 @@ function(kconfig_process_config)
         --output header ${SDKCONFIG_HEADER}
         --output cmake ${SDKCONFIG_CMAKE}
         --output json ${SDKCONFIG_JSON}
+        --output json_menus ${KCONFIG_JSON_MENUS}
         RESULT_VARIABLE config_result)
     if(config_result)
         message(FATAL_ERROR "Failed to run confgen.py (${confgen_basecommand}). Error ${config_result}")

tools/idf.py

@@ -365,6 +365,7 @@ ACTIONS = {
     "fullclean":             ( fullclean,    [], [] ),
     "reconfigure":           ( reconfigure,  [], [ "menuconfig" ] ),
     "menuconfig":            ( build_target, [], [] ),
+    "confserver":            ( build_target, [], [] ),
     "size":                  ( build_target, [ "app" ], [] ),
     "size-components":       ( build_target, [ "app" ], [] ),
     "size-files":            ( build_target, [ "app" ], [] ),

tools/kconfig_new/confgen.py

@@ -29,6 +29,7 @@ import json
 
 import gen_kconfig_doc
 import kconfiglib
+import pprint
 
 __version__ = "0.1"
 
@@ -65,7 +66,7 @@ def main():
 
     for fmt, filename in args.output:
         if not fmt in OUTPUT_FORMATS.keys():
-            print("Format '%s' not recognised. Known formats: %s" % (fmt, OUTPUT_FORMATS))
+            print("Format '%s' not recognised. Known formats: %s" % (fmt, OUTPUT_FORMATS.keys()))
             sys.exit(1)
 
     try:
@@ -150,9 +151,8 @@ def write_cmake(config, filename):
                     prefix, sym.name, val))
         config.walk_menu(write_node)
 
-def write_json(config, filename):
+def get_json_values(config):
     config_dict = {}
-
     def write_node(node):
         sym = node.item
         if not isinstance(sym, kconfiglib.Symbol):
@@ -168,9 +168,94 @@ def write_json(config, filename):
                 val = int(val)
             config_dict[sym.name] = val
     config.walk_menu(write_node)
+    return config_dict
+
+def write_json(config, filename):
+    config_dict = get_json_values(config)
     with open(filename, "w") as f:
         json.dump(config_dict, f, indent=4, sort_keys=True)
 
+def write_json_menus(config, filename):
+    result = []  # root level items
+    node_lookup = {}  # lookup from MenuNode to an item in result
+
+    def write_node(node):
+        try:
+            json_parent = node_lookup[node.parent]["children"]
+        except KeyError:
+            assert not node.parent in node_lookup  # if fails, we have a parent node with no "children" entity (ie a bug)
+            json_parent = result  # root level node
+
+        # node.kconfig.y means node has no dependency,
+        if node.dep is node.kconfig.y:
+            depends = None
+        else:
+            depends = kconfiglib.expr_str(node.dep)
+
+        try:
+            is_menuconfig = node.is_menuconfig
+        except AttributeError:
+            is_menuconfig = False
+
+        new_json = None
+        if node.item == kconfiglib.MENU or is_menuconfig:
+            new_json = { "type" : "menu",
+                         "title" : node.prompt[0],
+                         "depends_on": depends,
+                         "children": []
+            }
+            if is_menuconfig:
+                sym = node.item
+                new_json["name"] = sym.name
+                new_json["help"] = node.help
+                new_json["is_menuconfig"] = is_menuconfig
+                greatest_range = None
+                if len(sym.ranges) > 0:
+                    # Note: Evaluating the condition using kconfiglib's expr_value
+                    # should have one result different from value 0 ("n").
+                    for min_range, max_range, cond_expr in sym.ranges:
+                        if kconfiglib.expr_value(cond_expr) != "n":
+                            greatest_range = [min_range, max_range]
+                new_json["range"] = greatest_range
+
+        elif isinstance(node.item, kconfiglib.Symbol):
+            sym = node.item
+            greatest_range = None
+            if len(sym.ranges) > 0:
+                # Note: Evaluating the condition using kconfiglib's expr_value
+                # should have one result different from value 0 ("n").
+                for min_range, max_range, cond_expr in sym.ranges:
+                    if kconfiglib.expr_value(cond_expr) != "n":
+                        greatest_range = [int(min_range.str_value), int(max_range.str_value)]
+
+            new_json = {
+                "type" : kconfiglib.TYPE_TO_STR[sym.type],
+                "name" : sym.name,
+                "title": node.prompt[0] if node.prompt else None,
+                "depends_on" : depends,
+                "help": node.help,
+                "range" : greatest_range,
+                "children": [],
+            }
+        elif isinstance(node.item, kconfiglib.Choice):
+            choice = node.item
+            new_json = {
+                "type": "choice",
+                "title": node.prompt[0],
+                "name": choice.name,
+                "depends_on" : depends,
+                "help": node.help,
+                "children": []
+            }
+
+        if new_json:
+            json_parent.append(new_json)
+            node_lookup[node] = new_json
+
+    config.walk_menu(write_node)
+    with open(filename, "w") as f:
+        f.write(json.dumps(result, sort_keys=True, indent=4))
+
 def update_if_changed(source, destination):
     with open(source, "r") as f:
         source_contents = f.read()
@@ -190,6 +275,7 @@ OUTPUT_FORMATS = {
     "cmake" : write_cmake,
     "docs" : gen_kconfig_doc.write_docs,
     "json" : write_json,
+    "json_menus" : write_json_menus,
     }
 
 class FatalError(RuntimeError):

tools/kconfig_new/confserver.py

@@ -0,0 +1,185 @@
+#!/usr/bin/env python
+#
+# Long-running server process uses stdin & stdout to communicate JSON
+# with a caller
+#
+from __future__ import print_function
+import argparse
+import json
+import kconfiglib
+import os
+import sys
+import confgen
+from confgen import FatalError, __version__
+
+def main():
+    parser = argparse.ArgumentParser(description='confserver.py v%s - Config Generation Tool' % __version__, prog=os.path.basename(sys.argv[0]))
+
+    parser.add_argument('--config',
+                       help='Project configuration settings',
+                       required=True)
+
+    parser.add_argument('--kconfig',
+                        help='KConfig file with config item definitions',
+                        required=True)
+
+    parser.add_argument('--env', action='append', default=[],
+                        help='Environment to set when evaluating the config file', metavar='NAME=VAL')
+
+    args = parser.parse_args()
+
+    try:
+       args.env = [ (name,value) for (name,value) in ( e.split("=",1) for e in args.env) ]
+    except ValueError:
+       print("--env arguments must each contain =. To unset an environment variable, use 'ENV='")
+       sys.exit(1)
+
+    for name, value in args.env:
+        os.environ[name] = value
+
+    print("Server running, waiting for requests on stdin...", file=sys.stderr)
+    run_server(args.kconfig, args.config)
+
+
+def run_server(kconfig, sdkconfig):
+    config = kconfiglib.Kconfig(kconfig)
+    config.load_config(sdkconfig)
+
+    config_dict = confgen.get_json_values(config)
+    ranges_dict = get_ranges(config)
+    json.dump({"version": 1, "values" : config_dict, "ranges" : ranges_dict}, sys.stdout)
+    print("\n")
+
+    while True:
+        line = sys.stdin.readline()
+        if not line:
+            break
+        req = json.loads(line)
+        before = confgen.get_json_values(config)
+        before_ranges = get_ranges(config)
+
+        if "load" in req:  # if we're loading a different sdkconfig, response should have all items in it
+            before = {}
+            before_ranges = {}
+
+            # if no new filename is supplied, use existing sdkconfig path, otherwise update the path
+            if req["load"] is None:
+                req["load"] = sdkconfig
+            else:
+                sdkconfig = req["load"]
+
+        if "save" in req:
+            if req["save"] is None:
+                req["save"] = sdkconfig
+            else:
+                sdkconfig = req["save"]
+
+        error = handle_request(config, req)
+
+        after = confgen.get_json_values(config)
+        after_ranges = get_ranges(config)
+
+        values_diff = diff(before, after)
+        ranges_diff = diff(before_ranges, after_ranges)
+        response = {"version" : 1, "values" : values_diff, "ranges" : ranges_diff}
+        if error:
+            for e in error:
+                print("Error: %s" % e, file=sys.stderr)
+            response["error"] = error
+        json.dump(response, sys.stdout)
+        print("\n")
+
+
+def handle_request(config, req):
+    if not "version" in req:
+        return [ "All requests must have a 'version'" ]
+    if int(req["version"]) != 1:
+        return [ "Only version 1 requests supported" ]
+
+    error = []
+
+    if "load" in req:
+        print("Loading config from %s..." % req["load"], file=sys.stderr)
+        try:
+            config.load_config(req["load"])
+        except Exception as e:
+            error += [ "Failed to load from %s: %s" % (req["load"], e) ]
+
+    if "set" in req:
+        handle_set(config, error, req["set"])
+
+    if "save" in req:
+        try:
+            print("Saving config to %s..." % req["save"], file=sys.stderr)
+            confgen.write_config(config, req["save"])
+        except Exception as e:
+            error += [ "Failed to save to %s: %s" % (req["save"], e) ]
+
+    return error
+
+def handle_set(config, error, to_set):
+    missing = [ k for k in to_set if not k in config.syms ]
+    if missing:
+        error.append("The following config symbol(s) were not found: %s" % (", ".join(missing)))
+    # replace name keys with the full config symbol for each key:
+    to_set = dict((config.syms[k],v) for (k,v) in to_set.items() if not k in missing)
+
+    # Work through the list of values to set, noting that
+    # some may not be immediately applicable (maybe they depend
+    # on another value which is being set). Therefore, defer
+    # knowing if any value is unsettable until then end
+
+    while len(to_set):
+        set_pass = [ (k,v) for (k,v) in to_set.items() if k.visibility ]
+        if not set_pass:
+            break  # no visible keys left
+        for (sym,val) in set_pass:
+            if sym.type in (kconfiglib.BOOL, kconfiglib.TRISTATE):
+                if val == True:
+                    sym.set_value(2)
+                elif val == False:
+                    sym.set_value(0)
+                else:
+                    error.append("Boolean symbol %s only accepts true/false values" % sym.name)
+            else:
+                sym.set_value(str(val))
+            print("Set %s" % sym.name)
+            del to_set[sym]
+
+    if len(to_set):
+        error.append("The following config symbol(s) were not visible so were not updated: %s" % (", ".join(s.name for s in to_set)))
+
+
+
+def diff(before, after):
+    """
+    Return a dictionary with the difference between 'before' and 'after' (either with the new value if changed,
+    or None as the value if a key in 'before' is missing in 'after'
+    """
+    diff = dict((k,v) for (k,v) in after.items() if before.get(k, None) != v)
+    hidden = dict((k,None) for k in before if k not in after)
+    diff.update(hidden)
+    return diff
+
+
+def get_ranges(config):
+    ranges_dict = {}
+    def handle_node(node):
+        sym = node.item
+        if not isinstance(sym, kconfiglib.Symbol):
+            return
+        active_range = sym.active_range
+        if active_range[0] is not None:
+            ranges_dict[sym.name] = active_range
+
+    config.walk_menu(handle_node)
+    return ranges_dict
+
+
+if __name__ == '__main__':
+    try:
+        main()
+    except FatalError as e:
+        print("A fatal error occurred: %s" % e, file=sys.stderr)
+        sys.exit(2)
+

tools/kconfig_new/kconfiglib.py

@@ -2419,24 +2419,11 @@ class Symbol(object):
             base = _TYPE_TO_BASE[self.orig_type]
 
             # Check if a range is in effect
-            for low_expr, high_expr, cond in self.ranges:
-                if expr_value(cond):
-                    has_active_range = True
-
-                    # The zeros are from the C implementation running strtoll()
-                    # on empty strings
-                    low = int(low_expr.str_value, base) if \
-                      _is_base_n(low_expr.str_value, base) else 0
-                    high = int(high_expr.str_value, base) if \
-                      _is_base_n(high_expr.str_value, base) else 0
-
-                    break
-            else:
-                has_active_range = False
+            low, high = self.active_range
 
             if vis and self.user_value is not None and \
                _is_base_n(self.user_value, base) and \
-               (not has_active_range or
+               (low is None or
                 low <= int(self.user_value, base) <= high):
 
                 # If the user value is well-formed and satisfies range
@@ -2463,7 +2450,7 @@ class Symbol(object):
                     val_num = 0  # strtoll() on empty string
 
                 # This clamping procedure runs even if there's no default
-                if has_active_range:
+                if low is not None:
                     clamp = None
                     if val_num < low:
                         clamp = low
@@ -2714,6 +2701,28 @@ class Symbol(object):
             if self._is_user_assignable():
                 self._rec_invalidate()
 
+    @property
+    def active_range(self):
+        """
+        Returns a tuple of (low, high) integer values if a range
+        limit is active for this symbol, or (None, None) if no range
+        limit exists.
+        """
+        base = _TYPE_TO_BASE[self.orig_type]
+
+        for low_expr, high_expr, cond in self.ranges:
+            if expr_value(cond):
+                # The zeros are from the C implementation running strtoll()
+                # on empty strings
+                low = int(low_expr.str_value, base) if \
+                    _is_base_n(low_expr.str_value, base) else 0
+                high = int(high_expr.str_value, base) if \
+                    _is_base_n(high_expr.str_value, base) else 0
+
+                return (low, high)
+        return (None, None)
+
+
     def __repr__(self):
         """
         Returns a string with information about the symbol (including its name,

tools/kconfig_new/test/Kconfig

@@ -0,0 +1,44 @@
+menu "Test config"
+
+config TEST_BOOL
+  bool "Test boolean"
+  default n
+
+config TEST_CHILD_BOOL
+  bool "Test boolean"
+  depends on TEST_BOOL
+  default y
+
+config TEST_CHILD_STR
+  string "Test str"
+  depends on TEST_BOOL
+  default "OHAI!"
+
+choice TEST_CHOICE
+  prompt "Some choice"
+  default CHOICE_A
+
+config CHOICE_A
+  bool "A"
+
+config CHOICE_B
+  bool "B"
+
+endchoice
+
+config DEPENDS_ON_CHOICE
+   string "Depends on choice"
+   default "Depends on A" if CHOICE_A
+   default "Depends on B" if CHOICE_B
+   default "WAT"
+
+config SOME_UNRELATED_THING
+   bool "Some unrelated thing"
+
+config TEST_CONDITIONAL_RANGES
+   int "Something with a range"
+   range 0 100 if TEST_BOOL
+   range 0 10
+   default 1
+
+endmenu

tools/kconfig_new/test/sdkconfig

@@ -0,0 +1 @@
+CONFIG_SOME_UNRELATED_THING=y

tools/kconfig_new/test/test_confserver.py

@@ -0,0 +1,114 @@
+#!/usr/bin/env python
+import os
+import sys
+import threading
+import time
+import json
+import argparse
+import shutil
+
+import pexpect
+
+sys.path.append("..")
+import confserver
+
+def create_server_thread(*args):
+    t = threading.Thread()
+
+def parse_testcases():
+    with open("testcases.txt", "r") as f:
+        cases = [ l for l in f.readlines() if len(l.strip()) > 0 ]
+    # Each 3 lines in the file should be formatted as:
+    # * Description of the test change
+    # * JSON "changes" to send to the server
+    # * Result JSON to expect back from the server
+    if len(cases) % 3 != 0:
+        print("Warning: testcases.txt has wrong number of non-empty lines (%d). Should be 3 lines per test case, always." % len(cases))
+
+    for i in range(0, len(cases), 3):
+        desc = cases[i]
+        send = cases[i+1]
+        expect = cases[i+2]
+        if not desc.startswith("* "):
+            raise RuntimeError("Unexpected description at line %d: '%s'" % (i+1, desc))
+        if not send.startswith("> "):
+            raise RuntimeError("Unexpected send at line %d: '%s'" % (i+2, send))
+        if not expect.startswith("< "):
+            raise RuntimeError("Unexpected expect at line %d: '%s'" % (i+3, expect))
+        desc = desc[2:]
+        send = json.loads(send[2:])
+        expect = json.loads(expect[2:])
+        yield (desc, send, expect)
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--logfile', type=argparse.FileType('w'), help='Optional session log of the interactions with confserver.py')
+    args = parser.parse_args()
+
+    # set up temporary file to use as sdkconfig copy
+    temp_sdkconfig_path = os.tmpnam()
+    try:
+        with open(temp_sdkconfig_path, "w") as temp_sdkconfig:
+            with open("sdkconfig") as orig:
+                temp_sdkconfig.write(orig.read())
+
+        cmdline = "../confserver.py --kconfig Kconfig --config %s" % temp_sdkconfig_path
+        print("Running: %s" % cmdline)
+        p = pexpect.spawn(cmdline, timeout=0.5)
+        p.logfile = args.logfile
+        p.setecho(False)
+
+        def expect_json():
+            # run p.expect() to expect a json object back, and return it as parsed JSON
+            p.expect("{.+}\r\n")
+            return json.loads(p.match.group(0).strip())
+
+        p.expect("Server running.+\r\n")
+        initial = expect_json()
+        print("Initial: %s" % initial)
+        cases = parse_testcases()
+
+        for (desc, send, expected) in cases:
+            print(desc)
+            req = { "version" : "1", "set" : send }
+            req = json.dumps(req)
+            print("Sending: %s" % (req))
+            p.send("%s\n" % req)
+            readback = expect_json()
+            print("Read back: %s" % (json.dumps(readback)))
+            if readback.get("version", None) != 1:
+                raise RuntimeError('Expected {"version" : 1} in response')
+            for expect_key in expected.keys():
+                read_vals = readback[expect_key]
+                exp_vals = expected[expect_key]
+                if read_vals != exp_vals:
+                    expect_diff = dict((k,v) for (k,v) in exp_vals.items() if not k in read_vals or v != read_vals[k])
+                    raise RuntimeError("Test failed! Was expecting %s: %s" % (expect_key, json.dumps(expect_diff)))
+            print("OK")
+
+        print("Testing load/save...")
+        before = os.stat(temp_sdkconfig_path).st_mtime
+        p.send("%s\n" % json.dumps({ "version" : "1", "save" : temp_sdkconfig_path }))
+        save_result = expect_json()
+        print("Save result: %s" % (json.dumps(save_result)))
+        assert len(save_result["values"]) == 0
+        assert len(save_result["ranges"]) == 0
+        after = os.stat(temp_sdkconfig_path).st_mtime
+        assert after > before
+
+        p.send("%s\n" % json.dumps({ "version" : "1", "load" : temp_sdkconfig_path }))
+        load_result = expect_json()
+        print("Load result: %s" % (json.dumps(load_result)))
+        assert len(load_result["values"]) > 0  # loading same file should return all config items
+        assert len(load_result["ranges"]) > 0
+        print("Done. All passed.")
+
+    finally:
+        try:
+            os.remove(temp_sdkconfig_path)
+        except OSError:
+            pass
+
+if __name__ == "__main__":
+    main()
+

tools/kconfig_new/test/testcases.txt

@@ -0,0 +1,31 @@
+* Set TEST_BOOL, showing child items
+> { "TEST_BOOL" : true }
+< { "values" : { "TEST_BOOL" : true, "TEST_CHILD_STR" : "OHAI!", "TEST_CHILD_BOOL" : true }, "ranges": {"TEST_CONDITIONAL_RANGES": [0, 100]} }
+
+* Set TEST_CHILD_STR
+> { "TEST_CHILD_STR" : "Other value" }
+< { "values" : { "TEST_CHILD_STR" : "Other value" } }
+
+* Clear TEST_BOOL, hiding child items
+> { "TEST_BOOL" : false }
+< { "values" : { "TEST_BOOL" : false, "TEST_CHILD_STR" : null, "TEST_CHILD_BOOL" : null },  "ranges": {"TEST_CONDITIONAL_RANGES": [0, 10]} }
+
+* Set TEST_CHILD_BOOL, invalid as parent is disabled
+> { "TEST_CHILD_BOOL" : false }
+< { "values" : { } }
+
+* Set TEST_BOOL & TEST_CHILD_STR together
+> { "TEST_BOOL" : true, "TEST_CHILD_STR" : "New value" }
+< { "values" : { "TEST_BOOL" : true, "TEST_CHILD_STR" : "New value", "TEST_CHILD_BOOL" : true } }
+
+* Set choice
+> { "CHOICE_B" : true }
+< { "values" : { "CHOICE_B" : true, "CHOICE_A" : false, "DEPENDS_ON_CHOICE" : "Depends on B" } }
+
+* Set string which depends on choice B
+> { "DEPENDS_ON_CHOICE" : "oh, really?" }
+< { "values" : { "DEPENDS_ON_CHOICE" : "oh, really?" } }
+
+* Try setting boolean values to invalid types
+> { "CHOICE_A" : 11, "TEST_BOOL" : "false" }
+< { "values" : { } }