nvs_flash: Modify the default NVS initialization API with internal nvs encryption

    handling (only when nvs encryption is enabled)
    * NVS Encryption will now be turned on by default with flash encryption
    * Updated the flash encryption example to shocase NVS encryption
    along with information on how to configure and use NVS encryption
    * Updated respective test case
    * Added two partition tables for NVS encryption
     i) Table 1- Single factory app, no OTA, encrypted NVS
     ii) Table 2- Factory app, Two OTA, encrypted NVS

components/nvs_flash/Kconfig

@@ -2,7 +2,7 @@ menu "NVS"
 
     config NVS_ENCRYPTION
         bool "Enable NVS encryption"
-        default n
+        default y
         depends on SECURE_FLASH_ENC_ENABLED
         help
             This option enables encryption for NVS. When enabled, AES-XTS is used to encrypt

components/nvs_flash/README.rst

@@ -256,14 +256,18 @@ Each node in the hash list contains a 24-bit hash and 8-bit item index. Hash is
 NVS Encryption
 --------------
 
-Data stored in NVS partitions can be encrypted using AES-XTS in the manner similar to the one mentioned in disk encryption standard IEEE P1619. For the purpose of encryption, each entry is treated as one `sector` and relative address of the entry (w.r.t. partition-start) is fed to the encryption algorithm as `sector-number`. The keys required for NVS encryption are stored in yet another partition, which is protected using :doc:`Flash Encryption <../../security/flash-encryption>`. Therefore, enabling :doc:`Flash Encryption <../../security/flash-encryption>` is a prerequisite for NVS encryption.
+Data stored in NVS partitions can be encrypted using AES-XTS in the manner similar to the one mentioned in disk encryption standard IEEE P1619. For the purpose of encryption, each entry is treated as one `sector` and relative address of the entry (w.r.t. partition-start) is fed to the encryption algorithm as `sector-number`. The NVS Encryption can be enabled by enabling :ref:`CONFIG_NVS_ENCRYPTION`. The keys required for NVS encryption are stored in yet another partition, which is protected using :doc:`Flash Encryption <../../security/flash-encryption>`. Therefore, enabling :doc:`Flash Encryption <../../security/flash-encryption>` is a prerequisite for NVS encryption.
+
+The NVS Encryption is enabled by default when :doc:`Flash Encryption <../../security/flash-encryption>` is enabled. This is done because WiFi driver stores credentials (like SSID and passphrase) in the default NVS partition. It is important to encrypt them as default choice if platform level encryption is already enabled.
+
+For using NVS encryption, the partition table must contain the :ref:`nvs_key_partition`. Two partition tables containing the :ref:`nvs_key_partition` are provided for NVS encryption under the partition table option (menuconfig->Partition Table). They can be selected with the project configuration menu (``idf.py menuconfig``). Please refer to the example :example:`security/flash_encryption` for how to configure and use NVS encryption feature.
 
 .. _nvs_key_partition:
 
 NVS key partition
 ^^^^^^^^^^^^^^^^^
 
-An application requiring NVS encryption support needs to be compiled with a key-partition of the type `data` and subtype `key`. This partition should be marked as `encrypted`. Refer to :doc:`Partition Tables <../../api-guides/partition-tables>` for more details. The size of the partition should be 4096 bytes (minimum partition size). The structure of this partition is depicted below.
+    An application requiring NVS encryption support needs to be compiled with a key-partition of the type `data` and subtype `key`. This partition should be marked as `encrypted`. Refer to :doc:`Partition Tables <../../api-guides/partition-tables>` for more details. Two additional partition tables which contain the :ref:`nvs_key_partition` are provided under the partition table option (menuconfig->Partition Table). They can be directly used for :ref:`nvs_encryption`. The structure of these partitions is depicted below.
 
 ::
 
@@ -275,16 +279,45 @@ An application requiring NVS encryption support needs to be compiled with a key-
     |                  CRC32(4)                   |
     +---------------------------------------------+
 
-This partition can be generated using `nvs partition generator` utility and flashed onto the device. Since the partition is marked `encrypted` and :doc:`Flash Encryption <../../security/flash-encryption>` is enabled, bootloader will encrypt this partition using flash encryption key on the first boot. Alternatively, the keys can be generated after startup using the ``nvs_flash_generate_keys`` API function provided by ``nvs_flash.h``, which will then write those keys onto the key-partition in encrypted form.
+The XTS encryption keys in the :ref:`nvs_key_partition` can be generated with one of the following two ways.
+
+1. Generate the keys on the ESP chip:
+
+    When NVS encryption is enabled the :cpp:func:`nvs_flash_init` API function can be used to initialize the encrypted default NVS partition. The API function internally generates the XTS encryption keys on the ESP chip. The API function finds the first :ref:`nvs_key_partition`.
+    Then the API function automatically generates and stores the nvs keys in that partition by making use of the :cpp:func:`nvs_flash_generate_keys` API function provided by ``nvs_flash.h``. New keys are generated and stored only when the respective key partiton is empty. The same key partition can then be used to read the security configurations for initializing a custom encrypted NVS partition with help of :cpp:func:`nvs_flash_secure_init_partition`.
+
+    The API functions :cpp:func:`nvs_flash_secure_init` and :cpp:func:`nvs_flash_secure_init_partition` do not generate the keys internally. When these API functions are used for initializing encrypted NVS partitions, the keys can be generated after startup using the :cpp:func:`nvs_flash_generate_keys` API function provided by ``nvs_flash.h``. The API function will then write those keys onto the key-partition in encrypted form.
+
+2. Use pre-generated key partition:
+
+    This option will be required by the user when keys in the :ref:`nvs_key_partition` are not generated by the application. The :ref:`nvs_key_partition` containing the XTS encryption keys can be generated with the help of :doc:`NVS Partition Generator Utility</api-reference/storage/nvs_partition_gen>`. Then the user can store the pre generated key partition on the flash with help of the following two commands:
+
+    i) Build and flash the partition table
+    ::
+
+        idf.py partition_table partition_table-flash
+
+    ii) Store the keys in the :ref:`nvs_key_partition` (on the flash) with the help of :component_file:`parttool.py<partition_table/parttool.py>` (see Partition Tool section in :doc:`partition-tables </api-guides/partition-tables>` for more details)
+    ::
+
+        parttool.py --port /dev/ttyUSB0 --partition-table-offset "nvs_key partition offset" write_partition --partition-name="name of nvs_key partition" --input "nvs_key partition"
+
+Since the key partition is marked as `encrypted` and :doc:`Flash Encryption <../../security/flash-encryption>` is enabled, the bootloader will encrypt this partition using flash encryption key on the first boot.
 
 It is possible for an application to use different keys for different NVS partitions and thereby have multiple key-partitions. However, it is a responsibility of the application to provide correct key-partition/keys for the purpose of encryption/decryption.
 
 Encrypted Read/Write
 ^^^^^^^^^^^^^^^^^^^^
 
-The same NVS API functions ``nvs_get_*`` or ``nvs_set_*`` can be used for reading of, and writing to an encrypted nvs partition as well. However, the API functions for initialising NVS partitions are different: ``nvs_flash_secure_init`` and ``nvs_flash_secure_init_partition`` instead of ``nvs_flash_init`` and ``nvs_flash_init_partition`` respectively. The ``nvs_sec_cfg_t`` structure required for these API functions can be populated using ``nvs_flash_generate_keys`` or ``nvs_flash_read_security_cfg``.
+The same NVS API functions ``nvs_get_*`` or ``nvs_set_*`` can be used for reading of, and writing to an encrypted nvs partition as well.
+
+**Encrypt the default NVS partition:**
+To enable encryption for the default NVS partition no additional steps are necessary. When :ref:`CONFIG_NVS_ENCRYPTION` is enabled, the :cpp:func:`nvs_flash_init` API function internally performs some additional steps using the first :ref:`nvs_key_partition` found to enable encryption for the default NVS partition (refer to the API documentation for more details). Alternatively, :cpp:func:`nvs_flash_secure_init` API function can also be used to enable encryption for the default NVS partition.
+
+**Encrypt a custom NVS partition:**
+To enable encryption for a custom NVS partition, :cpp:func:`nvs_flash_secure_init_partition` API function is used instead of :cpp:func:`nvs_flash_init_partition`.
 
-Applications are expected to follow the steps below in order to perform NVS read/write operations with encryption enabled.
+When :cpp:func:`nvs_flash_secure_init` and :cpp:func:`nvs_flash_secure_init_partition` API functions are used, the applications are expected to follow the steps below in order to perform NVS read/write operations with encryption enabled.
 
     1. Find key partition and NVS data partition using ``esp_partition_find*`` API functions.
     2. Populate the ``nvs_sec_cfg_t`` struct using the ``nvs_flash_read_security_cfg`` or ``nvs_flash_generate_keys`` API functions.

components/nvs_flash/include/nvs_flash.h

@@ -38,6 +38,19 @@ typedef struct {
  * This API initialises the default NVS partition. The default NVS partition
  * is the one that is labeled "nvs" in the partition table.
  *
+ * When "NVS_ENCRYPTION" is enabled in the menuconfig, this API enables
+ * the NVS encryption for the default NVS partition as follows
+ *      1. Read security configurations from the first NVS key
+ *         partition listed in the partition table. (NVS key partition is
+ *         any "data" type partition which has the subtype value set to "nvs_keys")
+ *      2. If the NVS key partiton obtained in the previous step is empty,
+ *         generate and store new keys in that NVS key partiton.
+ *      3. Internally call "nvs_flash_secure_init()" with
+ *         the security configurations obtained/generated in the previous steps.
+ *
+ * Post initialization NVS read/write APIs
+ * remain the same irrespective of NVS encryption.
+ *
  * @return
  *      - ESP_OK if storage was successfully initialized.
  *      - ESP_ERR_NVS_NO_FREE_PAGES if the NVS storage contains no empty pages
@@ -45,6 +58,9 @@ typedef struct {
  *      - ESP_ERR_NOT_FOUND if no partition with label "nvs" is found in the partition table
  *      - ESP_ERR_NO_MEM in case memory could not be allocated for the internal structures
  *      - one of the error codes from the underlying flash storage driver
+ *      - error codes from nvs_flash_read_security_cfg API (when "NVS_ENCRYPTION" is enabled).
+ *      - error codes from nvs_flash_generate_keys API (when "NVS_ENCRYPTION" is enabled).
+ *      - error codes from nvs_flash_secure_init_partition API (when "NVS_ENCRYPTION" is enabled) .
  */
 esp_err_t nvs_flash_init(void);
 

components/nvs_flash/src/nvs_api.cpp

@@ -21,6 +21,7 @@
 #include "sdkconfig.h"
 #include <functional>
 #include "nvs_handle_simple.hpp"
+#include "esp_err.h"
 
 #ifdef LINUX_TARGET
 #include "crc.h"
@@ -136,7 +137,39 @@ extern "C" esp_err_t nvs_flash_init_partition(const char *part_name)
 
 extern "C" esp_err_t nvs_flash_init(void)
 {
+#ifdef CONFIG_NVS_ENCRYPTION
+    esp_err_t ret = ESP_FAIL;
+    const esp_partition_t *key_part = esp_partition_find_first(
+                                          ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_NVS_KEYS, NULL);
+    if (key_part == NULL) {
+        ESP_LOGE(TAG, "CONFIG_NVS_ENCRYPTION is enabled, but no partition with subtype nvs_keys found in the partition table.");
+        return ret;
+    }
+
+    nvs_sec_cfg_t cfg = {};
+    ret = nvs_flash_read_security_cfg(key_part, &cfg);
+    if (ret == ESP_ERR_NVS_KEYS_NOT_INITIALIZED) {
+        ESP_LOGI(TAG, "NVS key partition empty, generating keys");
+        ret = nvs_flash_generate_keys(key_part, &cfg);
+        if (ret != ESP_OK) {
+            ESP_LOGE(TAG, "Failed to generate keys: [0x%02X] (%s)", ret, esp_err_to_name(ret));
+            return ret;
+        }
+    } else if (ret != ESP_OK) {
+        ESP_LOGE(TAG, "Failed to read NVS security cfg: [0x%02X] (%s)", ret, esp_err_to_name(ret));
+        return ret;
+    }
+
+    ret = nvs_flash_secure_init_partition(NVS_DEFAULT_PART_NAME, &cfg);
+    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
+        ESP_LOGE(TAG, "Failed to initialize NVS partition: [0x%02X] (%s)", ret, esp_err_to_name(ret));
+        return ret;
+    }
+    ESP_LOGI(TAG, "NVS partition \"%s\" is encrypted.", NVS_DEFAULT_PART_NAME);
+    return ret;
+#else // CONFIG_NVS_ENCRYPTION
     return nvs_flash_init_partition(NVS_DEFAULT_PART_NAME);
+#endif
 }
 
 #ifdef CONFIG_NVS_ENCRYPTION

components/partition_table/Kconfig.projbuild

@@ -18,6 +18,12 @@ menu "Partition Table"
             bool "Factory app, two OTA definitions"
         config PARTITION_TABLE_CUSTOM
             bool "Custom partition table CSV"
+        config PARTITION_TABLE_SINGLE_APP_ENCRYPTED_NVS
+            bool "Single factory app, no OTA, encrypted NVS"
+            depends on !ESP32_COREDUMP_ENABLE_TO_FLASH && NVS_ENCRYPTION
+        config PARTITION_TABLE_TWO_OTA_ENCRYPTED_NVS
+            bool "Factory app, two OTA definitions, encrypted NVS"
+            depends on !ESP_COREDUMP_ENABLE_TO_FLASH && NVS_ENCRYPTION
     endchoice
 
     config PARTITION_TABLE_CUSTOM_FILENAME
@@ -31,8 +37,10 @@ menu "Partition Table"
         string
         default "partitions_singleapp.csv" if PARTITION_TABLE_SINGLE_APP && !ESP_COREDUMP_ENABLE_TO_FLASH
         default "partitions_singleapp_coredump.csv" if PARTITION_TABLE_SINGLE_APP && ESP_COREDUMP_ENABLE_TO_FLASH
+        default "partitions_singleapp_encr_nvs.csv" if PARTITION_TABLE_SINGLE_APP_ENCRYPTED_NVS
         default "partitions_two_ota.csv" if PARTITION_TABLE_TWO_OTA && !ESP_COREDUMP_ENABLE_TO_FLASH
         default "partitions_two_ota_coredump.csv" if PARTITION_TABLE_TWO_OTA && ESP_COREDUMP_ENABLE_TO_FLASH
+        default "partitions_two_ota_encr_nvs.csv" if PARTITION_TABLE_TWO_OTA_ENCRYPTED_NVS
         default PARTITION_TABLE_CUSTOM_FILENAME if PARTITION_TABLE_CUSTOM
 
     config PARTITION_TABLE_OFFSET

components/partition_table/partitions_singleapp_encr_nvs.csv

@@ -0,0 +1,6 @@
+# Name,   Type, SubType, Offset,  Size, Flags
+# Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap
+nvs,      data, nvs,     ,        0x6000,
+phy_init, data, phy,     ,        0x1000,
+factory,  app,  factory, ,        1M,
+nvs_key,  data, nvs_keys,,        0x1000, encrypted

components/partition_table/partitions_two_ota_encr_nvs.csv

@@ -0,0 +1,9 @@
+# Name,   Type, SubType, Offset,   Size, Flags
+# Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap
+nvs,      data, nvs,     ,        0x4000,
+otadata,  data, ota,     ,        0x2000,
+phy_init, data, phy,     ,        0x1000,
+factory,  app,  factory, ,        1M,
+ota_0,    app,  ota_0,   ,        1M,
+ota_1,    app,  ota_1,   ,        1M,
+nvs_key,  data, nvs_keys,,        0x1000, encrypted

examples/security/flash_encryption/README.md

@@ -16,11 +16,39 @@ The example also demonstrates writing and reading encrypted partitions in flash.
 ```
 idf.py menuconfig
 ```
-
+#### Configuration for flash encryption
 * Enable the flash encryption mode (Development or Release) under Security Features. Default usage mode is Development (recommended during test and development phase).
 
 Note: After enabling flash encryption, the bootloader size increases, which means that the offset of the partition table must be changed to 0x9000 from 0x8000 to prevent the bootloader from overlapping with the partition table. In this example, the default offset of the partition table is 0x9000.
 
+For better security, the NVS encryption is enabled by default when the flash encryption is enabled. If you choose to disable the NVS encryption, you can skip the NVS configuration step given below.
+
+#### Configuration for NVS encryption
+For using NVS encryption, the partition table must contain the [NVS key partition](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_flash.html#nvs-key-partition). Two partition tables containing the NVS keys partition are provided for NVS encryption under the partition table option . They can be selected with the project configuration menu (`menuconfig -> Partition Table`). This particular example uses a custom partition table as it requires a `storage` partition along with the `nvs_keys` partition.
+
+The configuration for NVS encryption involves generating the XTS encryption keys in the [NVS key partition](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_flash.html#nvs-key-partition) partition. It can be done with one of the following method.
+
+1. Generate the XTS encryption keys on the ESP chip:
+
+    When NVS encryption is enabled the `nvs_flash_init` API function can internally generate the XTS encryption keys on the ESP chip. The API function finds the first [NVS key partition](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_flash.html#nvs-key-partition) i.e. a partition of type  `data` and subtype `nvs_keys`.
+    Then the API function automatically generates and stores the
+    nvs keys in that partition. New keys are generated and stored only when the respective key partiton is empty. (Consult the [`nvs_flash_init`](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_flash.html#_CPPv414nvs_flash_initv) API documentation in the ESP-IDF programming guide for more details)
+
+2. Use pre-generated XTS encryption keys:
+    This method will be required by the user when the `XTS encryption keys` in [NVS key partition](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_flash.html#nvs-key-partition) are not generated by the application.
+    The pre generated `Sample XTS encryption keys` can be stored on the flash with help of the following two commands
+
+    i) Build and flash the partition table:
+    ```
+    idf.py partition_table partition_table-flash
+    ```
+    ii) Store the `sample_encryption_keys.bin` in the `nvs_key`partition (on the flash) with the help of [parttool.py](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/partition-tables.html#partition-tool-parttool-py):
+    ```
+    parttool.py --port /dev/ttyUSB0 --partition-table-offset 0x9000 write_partition --partition-name="nvs_key" --input sample_encryption_keys.bin
+    ```
+    The sample [NVS key partition](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_flash.html#nvs-key-partition) partition used in this example is generated with the help of [NVS Partition Generator Utility](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_partition_gen.html#nvs-partition-generator-utility)
+    .
+
 ### Build and Flash
 
 When building the project and flashing it to the board FOR THE FIRST TIME after enabling flash encryption feature in menuconfig, run following command to program ESP32 and monitor the output:
@@ -104,6 +132,11 @@ I (491) example: 0x3ffb4b30   35 9b f2 07 b4 6d 40 89  28 b4 1e 22 98 7b 4a 36
 I (491) example: 0x3ffb4b40   ba 89 81 67 77 a3 60 5e  0a e7 51 01 b3 58 c2 f6  |...gw.`^..Q..X..|
 ```
 
+If the NVS encryption is enabled, then the output will show the status of the encrypted partition as follows
+
+```
+I (667) example_nvs: NVS partition "nvs" is encrypted.
+```
 ## Troubleshooting
 
 It is also possible to use esptool.py utility to read the eFuse values and check if flash encryption is enabled or not
@@ -113,4 +146,3 @@ python $IDF_PATH/components/esptool_py/esptool/espefuse.py --port PORT summary
 ```
 
 If FLASH_CRYPT_CNT eFuse value is non-zero flash encryption is enabled
-

examples/security/flash_encryption/example_test.py

@@ -52,7 +52,9 @@ def test_examples_security_flash_encryption(env, extra_data):
         'with esp_partition_read',
         plain_hex_str,
         'with spi_flash_read',
-        expected_str
+        expected_str,
+        # The status of NVS encryption for the "nvs" partition
+        'NVS partition "nvs" is encrypted.'
     ]
     for line in lines:
         dut.expect(line, timeout=2)

examples/security/flash_encryption/main/flash_encrypt_main.c

@@ -16,6 +16,7 @@
 #include "esp_partition.h"
 #include "esp_flash_encrypt.h"
 #include "esp_efuse_table.h"
+#include "nvs_flash.h"
 
 static void example_print_chip_info(void);
 static void example_print_flash_encryption_status(void);
@@ -41,6 +42,13 @@ void app_main(void)
     example_print_chip_info();
     example_print_flash_encryption_status();
     example_read_write_flash();
+    /* Initialize the default NVS partition */
+    esp_err_t ret = nvs_flash_init();
+    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
+        ESP_ERROR_CHECK(nvs_flash_erase());
+        ret = nvs_flash_init();
+    }
+    ESP_ERROR_CHECK(ret);
 }
 
 

examples/security/flash_encryption/partitions_example.csv

@@ -3,3 +3,5 @@ nvs,        data, nvs,      ,  0x6000,
 # Extra partition to demonstrate reading/writing of encrypted flash
 storage,    data, 0xff,     ,  0x1000, encrypted
 factory,    app,  factory,  , 1M,
+# nvs_key partition contains the key that encrypts the NVS partition named nvs. The nvs_key partition needs to be encrypted.
+nvs_key,    data, nvs_keys, , 0x1000, encrypted,

examples/system/ota/simple_ota_example/example_test.py

@@ -174,6 +174,9 @@ def test_examples_protocol_simple_ota_example_with_flash_encryption(env, extra_d
     binary_file = os.path.join(dut1.app.binary_path, "simple_ota.bin")
     bin_size = os.path.getsize(binary_file)
     ttfw_idf.log_performance("simple_ota_bin_size", "{}KB".format(bin_size // 1024))
+    # erase flash on the device
+    print("Erasing the flash in order to have an empty NVS key partiton")
+    dut1.erase_flash()
     # start test
     host_ip = get_my_ip()
     thread1 = Thread(target=start_https_server, args=(dut1.app.binary_path, host_ip, 8000))

examples/system/ota/simple_ota_example/sdkconfig.ci.flash_enc

@@ -18,3 +18,5 @@ CONFIG_EXAMPLE_ETH_MDIO_GPIO=18
 CONFIG_EXAMPLE_ETH_PHY_RST_GPIO=5
 CONFIG_EXAMPLE_ETH_PHY_ADDR=1
 CONFIG_EXAMPLE_CONNECT_IPV6=y
+# This is required for nvs encryption (which is enabled by default with flash encryption)
+CONFIG_PARTITION_TABLE_TWO_OTA_ENCRYPTED_NVS=y