docs: Added documentation for FreeRTOS SMP changes

Added documentation about the ESP-IDF changes to FreeRTOS.
The documentation covers changes to the following FreeRTOS aspects.

    - Task Creation
    - Affects on scheduling (Task skipping, scheduler suspension, tick synchronicity)
    - Critical sections and disabling interrupts
    - Thread Local Storage Pointers and deletion callbacks
    - Configuring ESP-IDF FreeRTOS

components/freertos/readme_smp.txt

@@ -1,27 +0,0 @@
-This version of FreeRTOS has been modified by Espressif to be SMP-aware. The 
-API is similar to the original FreeRTOS API, with the following changes:
-
-- The xTaskCreate() function now creates tasks that will run on the first 
-core only, for backwards compatibility. To schedule tasks on another core,
-use xTaskCreatePinnedToCore(), which will accept a core ID as the last
-argument. If this is the constant tskNO_AFFINITY, the task will be dynamically
-scheduled on whichever core has time.
-
-- vTaskSuspendAll/vTaskResumeAll in non-SMP FreeRTOS will suspend the scheduler
-so no other tasks than the current one will run. In this SMP version, it will
-only suspend the scheduler ON THE CURRENT CORE. That is, tasks scheduled to
-run on the other core(s) or without a specific CPU affinity, will still be
-able to run.
-
-- Enabling and disabling interrupts will only affect the current core. 
-Disabling the interrupts will not disallow other tasks to run as
-it would on a single-core system: the other core still will keep on 
-executing all it's own. Use a mux, queue or semaphore to protect your
-structures instead.
-
-- This FreeRTOS version has the task local storage backported from the 8.2.x
-versions. It, however, has an addition: you can also set a callback when you 
-set the pointer. This callback will be called by the idle task, with the 
-pointer as an argument, when the thread is destroyed. This depends on the idle
-task getting CPU time; when a thread is hogging the CPU without yielding,
-the idle thread won't be called and the delete callback won't be called either.

docs/api-guides/freertos-smp.rst

@@ -0,0 +1,369 @@
+ESP-IDF FreeRTOS SMP Changes
+============================
+
+Overview
+--------
+
+The vanilla FreeRTOS is designed to run on a single core. However the ESP32 is 
+dual core containing a Protocol CPU (known as **CPU 0** or **PRO_CPU**) and an 
+Application CPU (known as **CPU 1** or **APP_CPU**). The two cores are 
+identical in practice and share the same memory. This allows the two cores to 
+run tasks interchangeably between them.
+
+The ESP-IDF FreeRTOS is a modified version of vanilla FreeRTOS which supports 
+symmetric multiprocessing (SMP). ESP-IDF FreeRTOS is based on the Xtensa port 
+of FreeRTOS v8.2.0, however features such as static task creation and Thread 
+Local Storage Pointers have been backported from later versions of FreeRTOS. 
+This guide outlines the major differences between vanilla FreeRTOS and 
+ESP-IDF FreeRTOS. The API reference for vanilla FreeRTOS can be found 
+via http://www.freertos.org/a00106.html
+
+:ref:`tasks-and-task-creation`: Use ``xTaskCreatePinnedToCore()`` or 
+``xTaskCreateStaticPinnedToCore()`` to create tasks in ESP-IDF FreeRTOS. The 
+last parameter of the two functions is ``xCoreID``. This parameter specifies 
+which core the task is pinned to. Acceptable values are ``0`` for **PRO_CPU**, 
+``1`` for **APP_CPU**, or ``tskNO_AFFINITY`` which allows the task to run on
+both.
+        
+:ref:`round-robin-scheduling`: The ESP-IDF FreeRTOS scheduler will skip tasks when 
+implementing Round-Robin scheduling between multiple tasks in the Ready state 
+that are of the same priority. To avoid this behavior, ensure that those tasks either 
+enter a blocked state, or are distributed across a wider range of priorities.
+        
+:ref:`scheduler-suspension`: Suspending the scheduler in ESP-IDF FreeRTOS will only 
+affect the scheduler on the the calling core. In other words, calling 
+``vTaskSuspendAll()`` on **PRO_CPU** will not prevent **APP_CPU** from scheduling, and
+vice versa. Use critical sections or semaphores instead for simultaneous
+access protection.
+    
+:ref:`tick-interrupt-synchronicity`: Tick interrupts of **PRO_CPU** and **APP_CPU** 
+are not synchronized. Do not expect to use ``vTaskDelay`` or 
+``vTaskDelayUntil`` as an accurate method of synchronizing task execution 
+between the two cores. Use a counting semaphore instead as their context 
+switches are not tied to tick interrupts due to preemption.
+        
+:ref:`critical-sections`: In ESP-IDF FreeRTOS, critical sections are implemented using
+mutexes. Entering critical sections involve taking a mutex, then disabling the 
+scheduler and interrupts of the calling core. However the other core is left 
+unaffected. If the other core attemps to take same mutex, it will spin until
+the calling core has released the mutex by exiting the critical section.
+        
+:ref:`deletion-callbacks`: ESP-IDF FreeRTOS has 
+backported the Thread Local Storage Pointers feature. However they have the 
+extra feature of deletion callbacks. Deletion callbacks are used to
+automatically free memory used by Thread Local Storage Pointers during the task
+deletion. Call ``vTaskSetThreadLocalStoragePointerAndDelCallback()``
+to set Thread Local Storage Pointers and deletion callbacks.
+
+:ref:`esp-idf-freertos-configuration`: Several aspects of ESP-IDF FreeRTOS can be 
+configured using ``make meunconfig`` such as running ESP-IDF in Unicore Mode,
+or configuring the number of Thread Local Storage Pointers each task will have.
+
+
+.. _tasks-and-task-creation:
+
+Tasks and Task Creation
+-----------------------
+
+Tasks in ESP-IDF FreeRTOS are designed to run on a particular core, therefore 
+two new task creation functions have been added to ESP-IDF FreeRTOS by 
+appending ``PinnedToCore`` to the names of the task creation functions in 
+vanilla FreeRTOS. The vanilla FreeRTOS functions of ``xTaskCreate()``
+and ``xTaskCreateStatic()`` have led to the addition of 
+``xTaskCreatePinnedToCore()`` and ``xTaskCreateStaticPinnedToCore()`` in 
+ESP-IDF FreeRTOS.
+
+For more details see :component_file:`freertos/task.c`
+
+The ESP-IDF FreeRTOS task creation functions are nearly identical to their 
+vanilla counterparts with the exception of the extra parameter known as 
+``xCoreID``. This parameter specifies the core on which the task should run on 
+and can be one of the following values.
+
+    -	``0`` pins the task to **PRO_CPU**
+    -	``1`` pins the task to **APP_CPU**
+    -	``tskNO_AFFINITY`` allows the task to be run on both CPUs
+
+For example ``xTaskCreatePinnedToCore(tsk_callback, “APP_CPU Task”, 1000, NULL, 10, NULL, 1)`` 
+creates a task of priority 10 that is pinned to **APP_CPU** with a stack size 
+of 1000 bytes. It should be noted that the ``uxStackDepth`` parameter in 
+vanilla FreeRTOS specifies a task’s stack depth in terms of the number of 
+words, whereas ESP-IDF FreeRTOS specifies the stack depth in terms of bytes.
+
+Note that the vanilla FreeRTOS functions ``xTaskCreate`` and 
+``xTaskCreateStatic`` have been macro defined in ESP-IDF FreeRTOS to call 
+``xTaskCreatePinnedToCore()`` and ``xTaskCreateStaticPinnedToCore()``
+respectively with ``tskNO_AFFINITY`` as the ``xCoreID`` value. 
+
+Each Task Control Block (TCB) in ESP-IDF stores the ``xCoreID`` as a member. 
+Hence when each core calls the scheduler to select a task to run, the 
+``xCoreID`` member will allow the scheduler to determine if a given task is  
+permitted to run on the core that called it.
+
+Scheduling
+----------
+
+The vanilla FreeRTOS implements scheduling in the ``vTaskSwitchContext()`` 
+function. This function is responsible for selecting the highest priority task
+to run from a list of tasks in the Ready state known as the Ready Tasks List 
+(described in the next section). In ESP-IDF FreeRTOS, each core will call 
+``vTaskSwitchContext()`` independently to select a task to run from the 
+Ready Tasks List which is shared between both cores. There are several 
+differences in scheduling behavior between vanilla and ESP-IDF FreeRTOS such as 
+differences in Round Robin scheduling, scheduler suspension, and tick interrupt 
+synchronicity. 
+
+.. _round-robin-scheduling:
+
+Round Robin Scheduling
+^^^^^^^^^^^^^^^^^^^^^^
+
+Given multiple tasks in the Ready state and of the same priority, vanilla 
+FreeRTOS implements Round Robin scheduling between each task. This will result
+in running those tasks in turn each time the scheduler is called 
+(e.g. every tick interrupt). On the other hand, the ESP-IDF FreeRTOS scheduler 
+may skip tasks when Round Robin scheduling multiple Ready state tasks of the 
+same priority.
+
+The issue of skipping tasks during Round Robin scheduling arises from the way 
+the Ready Tasks List is implemented in FreeRTOS. In vanilla FreeRTOS, 
+``pxReadyTasksList`` is used to store a list of tasks that are in the Ready 
+state. The list is implemented as an array of length ``configMAX_PRIORITIES`` 
+where each element of the array is a linked list. Each linked list is of type 
+``List_t`` and contains TCBs of tasks of the same priority that are in the 
+Ready state. The following diagram illustrates the ``pxReadyTasksList`` 
+structure.
+
+.. figure:: ../_static/freertos-ready-task-list.png
+    :align: center
+    :alt: Vanilla FreeRTOS Ready Task List Structure
+    
+    Illustration of FreeRTOS Ready Task List Data Structure 
+
+
+Each linked list also contains a ``pxIndex`` which points to the last TCB 
+returned when the list was queried. This index allows the ``vTaskSwitchContext()`` 
+to start traversing the list at the TCB immediately after ``pxIndex`` hence 
+implementing Round Robin Scheduling between tasks of the same priority.
+
+In ESP-IDF FreeRTOS, the Ready Tasks List is shared between cores hence 
+``pxReadyTasksList`` will contain tasks pinned to different cores. When a core 
+calls the scheduler, it is able to look at the ``xCoreID`` member of each TCB 
+in the list to determine if a task is allowed to run on calling the core. The 
+ESP-IDF FreeRTOS ``pxReadyTasksList`` is illustrated below.
+
+.. figure:: ../_static/freertos-ready-task-list-smp.png
+    :align: center
+    :alt: ESP-IDF FreeRTOS Ready Task List Structure
+    
+    Illustration of FreeRTOS Ready Task List Data Structure in ESP-IDF
+    
+Therefore when **PRO_CPU** calls the scheduler, it will only consider the tasks 
+in blue or purple. Whereas when **APP_CPU** calls the scheduler, it will only 
+consider the tasks in orange or purple.
+
+Although each TCB has an ``xCoreID`` in ESP-IDF FreeRTOS, the linked list of 
+each priority only has a single ``pxIndex``. Therefore when the scheduler is 
+called from a particular core and traverses the linked list, it will skip all 
+TCBs pinned to the other core and point the pxIndex at the selected task. If 
+the other core then calls the scheduler, it will traverse the linked list 
+starting at the TCB immediately after ``pxIndex``. Therefore, TCBs skipped on
+the previous scheduler call from the other core would not be considered on the 
+current scheduler call. This issue is demonstrated in the following 
+illustration.
+
+.. figure:: ../_static/freertos-ready-task-list-smp-pxIndex.png
+    :align: center
+    :alt: ESP-IDF pxIndex Behavior
+    
+    Illustration of pxIndex behavior in ESP-IDF FreeRTOS
+
+Referring to the illustration above, assume that priority 9 is the highest 
+priority, and none of the tasks in priority 9 will block hence will always be 
+either in the running or Ready state.
+
+1)	**PRO_CPU** calls the scheduler and selects Task A to run, hence moves 
+``pxIndex`` to point to Task A
+
+2)	**APP_CPU** calls the scheduler and starts traversing from the task after 
+``pxIndex`` which is Task B. However Task B is not selected to run as it is not 
+pinned to **APP_CPU** hence it is skipped and Task C is selected instead. 
+``pxIndex`` now points to Task C
+
+3)	**PRO_CPU** calls the scheduler and starts traversing from Task D. It skips 
+Task D and selects Task E to run and points ``pxIndex`` to Task E. Notice that 
+Task B isn’t traversed because it was skipped the last time **APP_CPU** called 
+the scheduler to traverse the list.
+
+4)	The same situation with Task D will occur if **APP_CPU** calls the 
+scheduler again as ``pxIndex`` now points to Task E
+
+One solution to the issue of task skipping is to ensure that every task will
+enter a blocked state so that they are removed from the Ready Task List.
+Another solution is to distribute tasks across multiple priorities such that 
+a given priority will not be assigned multiple tasks that are pinned to 
+different cores.
+
+.. _scheduler-suspension:
+
+Scheduler Suspension
+^^^^^^^^^^^^^^^^^^^^
+
+In vanilla FreeRTOS, suspending the scheduler via ``vTaskSuspendAll()`` will 
+prevent calls of ``vTaskSwitchContext()`` from context switching until the 
+scheduler has been resumed with ``vTaskResumeAll()``. However servicing ISRs 
+are still permitted. Therefore any changes in task states as a result from the
+current running task or ISRSs will not be executed until the scheduler is 
+resumed. Scheduler suspension in vanilla FreeRTOS is a common protection method 
+against simultaneous access of data shared between tasks, whilst still allowing 
+ISRs to be serviced.
+
+In ESP-IDF FreeRTOS, ``vTaskSuspendAll()`` will only prevent calls of 
+``vTaskSwitchContext()`` from switching contexts on the core that called for the
+suspension. Hence if **PRO_CPU** calls ``vTaskSuspendAll()``, **APP_CPU** will 
+still be able to switch contexts. If data is shared between tasks that are 
+pinned to different cores, scheduler suspension is **NOT** a valid method of 
+protection against simultaneous access. Consider using critical sections 
+(disables interrupts) or semaphores (does not disable interrupts) instead when 
+protecting shared resources in ESP-IDF FreeRTOS.
+
+.. _tick-interrupt-synchronicity:
+
+Tick Interrupt Synchronicity 
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In ESP-IDF FreeRTOS, tasks on different cores that unblock on the same tick 
+count might not run at exactly the same time due to the scheduler calls from 
+each core being independent, and the tick interrupts to each core being 
+unsynchronized.
+
+In vanilla FreeRTOS the tick interrupt triggers a call to 
+``xTaskIncrementTick()`` which is responsible for incrementing the tick 
+counter, checking if tasks which have called ``vTaskDelay()`` have fulfilled 
+their delay period, and moving those tasks from the Delayed Task List to the 
+Ready Task List. The tick interrupt will then call the scheduler if a context 
+switch is necessary.
+
+In ESP-IDF FreeRTOS, delayed tasks are unblocked with reference to the tick 
+interrupt on PRO_CPU as PRO_CPU is responsible for incrementing the shared tick 
+count. However tick interrupts to each core might not be synchronized (same 
+frequency but out of phase) hence when PRO_CPU receives a tick interrupt, 
+APP_CPU might not have received it yet. Therefore if multiple tasks of the same 
+priority are unblocked on the same tick count, the task pinned to PRO_CPU will 
+run immediately whereas the task pinned to APP_CPU must wait until APP_CPU 
+receives its out of sync tick interrupt. Upon receiving the tick interrupt, 
+APP_CPU will then call for a context switch and finally switches contexts to
+the newly unblocked task.
+
+Therefore, task delays should **NOT** be used as a method of synchronization 
+between tasks in ESP-IDF FreeRTOS. Instead, consider using a counting semaphore 
+to unblock multiple tasks at the same time.
+
+.. _critical-sections:
+
+Critical Sections & Disabling Interrupts
+----------------------------------------
+
+Vanilla FreeRTOS implements critical sections in ``vTaskEnterCritical`` which 
+disables the scheduler and calls ``portDISABLE_INTERRUPTS``. This prevents 
+context switches and servicing of ISRs during a critical section. Therefore, 
+critical sections are used as a valid protection method against simultaneous 
+access in vanilla FreeRTOS.
+
+On the other hand, the ESP32 has no hardware method for cores to disable each 
+other’s interrupts. Calling ``portDISABLE_INTERRUPTS()`` will have no effect on 
+the interrupts of the other core. Therefore, disabling interrupts is **NOT** 
+a valid protection method against simultaneous access to shared data as it 
+leaves the other core free to access the data even if the current core has 
+disabled its own interrupts. 
+
+For this reason, ESP-IDF FreeRTOS implements critical sections using mutexes, 
+and calls to enter or exit a critical must provide a mutex that is associated 
+with a shared resource requiring access protection. When entering a critical 
+section in ESP-IDF FreeRTOS, the calling core will disable its scheduler and 
+interrupts similar to the vanilla FreeRTOS implementation. However, the calling 
+core will also take the mutex whilst the other core is left unaffected during 
+the critical section. If the other core attempts to take the same mutex, it 
+will spin until the mutex is released. Therefore, the ESP-IDF FreeRTOS 
+implementation of critical sections allows a core to have protected access to a
+shared resource without disabling the other core. The other core will only be 
+affected if it tries to concurrently access the same resource.
+
+The ESP-IDF FreeRTOS critical section functions have been modified as follows…
+
+ - ``taskENTER_CRITICAL(mux)``, ``taskENTER_CRITICAL_ISR(mux)``, 
+   ``portENTER_CRITICAL(mux)``, ``portENTER_CRITICAL_ISR(mux)`` are all macro 
+   defined to call ``vTaskEnterCritical()`` 
+
+ - ``taskEXIT_CRITICAL(mux)``, ``taskEXIT_CRITICAL_ISR(mux)``, 
+   ``portEXIT_CRITICAL(mux)``, ``portEXIT_CRITICAL_ISR(mux)`` are all macro 
+   defined to call ``vTaskExitCritical()``
+
+For more details see :component_file:`freertos/include/freertos/portmacro.h` 
+and :component_file:`freertos/task.c`
+
+It should be noted that when modifying vanilla FreeRTOS code to be ESP-IDF 
+FreeRTOS compatible, it is trivial to modify the type of critical section 
+called as they are all defined to call the same function. As long as the same 
+mutex is provided upon entering and exiting, the type of call should not 
+matter.
+
+.. _deletion-callbacks:
+
+Thread Local Storage Pointers & Deletion Callbacks
+--------------------------------------------------
+
+Thread Local Storage Pointers are pointers stored directly in the TCB which 
+allows each task to have a pointer to a data structure containing that is 
+specific to that task. However vanilla FreeRTOS provides no functionality to 
+free the memory pointed to by the Thread Local Storage Pointers. Therefore if 
+the memory pointed to by the Thread Local Storage Pointers is not explicitly 
+freed by the user before a task is deleted, memory leak will occur.
+
+ESP-IDF FreeRTOS provides the added feature of deletion callbacks. These 
+deletion callbacks are used to automatically free the memory pointed to by the 
+Thread Local Storage Pointers when a task is deleted. Each Thread Local Storage 
+Pointer can have its own call back, and these call backs are called when the 
+Idle tasks cleans up a deleted tasks.
+
+Vanilla FreeRTOS sets a Thread Local Storage Pointers using 
+``vTaskSetThreadLocalStoragePointer()`` whereas ESP-IDF FreeRTOS sets a Thread 
+Local Storage Pointers and Deletion Callbacks using 
+``vTaskSetThreadLocalStoragePointerAndDelCallback()`` which accepts a pointer 
+to the deletion call back as an extra parameter of type 
+```TlsDeleteCallbackFunction_t``. Calling the vanilla FreeRTOS API 
+``vTaskSetThreadLocalStoragePointer()`` is still valid however it is internally
+defined to call ``vTaskSetThreadLocalStoragePointerAndDelCallback()`` with a
+``NULL`` pointer as the deletion call back. This results in the selected Thread 
+Local Storage Pointer to have no deletion call back.
+
+For more details see :component_file:`freertos/include/freertos/task.h`
+
+.. _esp-idf-freertos-configuration:
+
+Configuring ESP-IDF FreeRTOS
+----------------------------
+
+The ESP-IDF FreeRTOS can be configured using ``make menuconfig`` under 
+``Component_Config/FreeRTOS``. The following section highlights some of the
+ESP-IDF FreeRTOS configuration options. For a full list of ESP-IDF
+FreeRTOS configurations, see :doc:`FreeRTOS <../api-reference/kconfig>`
+
+:ref:`CONFIG_FREERTOS_UNICORE` will run ESP-IDF FreeRTOS only
+on **PRO_CPU**. Note that this is **not equivalent to running vanilla 
+FreeRTOS**. Behaviors of multiple components in ESP-IDF will be modified such 
+as :component_file:`esp32/cpu_start.c`. For more details regarding the 
+effects of running ESP-IDF FreeRTOS on a single core, search for 
+occurences of ``CONFIG_FREERTOS_UNICORE`` in the ESP-IDF components.
+    
+:ref:`CONFIG_FREERTOS_THREAD_LOCAL_STORAGE_POINTERS` will define the 
+number of Thread Local Storage Pointers each task will have in ESP-IDF 
+FreeRTOS.
+
+:ref:`CONFIG_SUPPORT_STATIC_ALLOCATION` will enable the backported
+functionality of ``xTaskCreateStaticPinnedToCore()`` in ESP-IDF FreeRTOS
+    
+:ref:`CONFIG_FREERTOS_ASSERT_ON_UNTESTED_FUNCTION` will trigger a halt in
+particular functions in ESP-IDF FreeRTOS which have not been fully tested
+in an SMP context.
+    

docs/api-guides/index.rst

@@ -9,6 +9,7 @@ API Guides
    Deep Sleep Wake Stubs <deep-sleep-stub>
    ESP32 Core Dump <core_dump>
    Flash Encryption <../security/flash-encryption>
+   FreeRTOS SMP Changes <freertos-smp>
    High Level Interrupts <hlinterrupts>
    JTAG Debugging <jtag-debugging/index>
    Partition Tables <partition-tables>