Skip to main content

Video Decoding

This section shows how to configure the TouchGFX Generator to generate a TouchGFX HAL that has video decoding capabilities.

Before reading this scenario, please read the documentation on using MJPEG Video in TouchGFX and the documentation on the Video widget itself.

The following scenario details how to enable TouchGFX HAL to support video decoding through either software (LibJPEG) or hardware (JPEG).

Be sure to read all sections in this article before configuring the MCU for your specific project.

Generally, the LibJPEG and JPEG configurations can be found in STM32CubeMX in the Middleware and Multimedia catagories:

JPEG and LibJPEG setting in STM32CubeMX

Tip
  • Several of the TouchGFX Board Setups (TBSs) in TouchGFX Designer include support for MJPEG video decoding.
  • Use the STM32CubeMX configurations found in these example projects as inspiration to enable MJPEG video decoding on your specific MCU.
  • An example of software decoding can be found in the TBS for STM32H735G-DK, while hardware decoding is used in the TBS for STM32U5G9J-DK2.

RTOS Support​

The Generator User Guide mentions that Single- and Double buffer video decoding strategies require a CMSIS compliant RTOS, such as FreeRTOS. TouchGFX Generator generates an entry point function videoTaskFunc(), that must be associated with a Video decoding task. STM32CubeMX can generate this configuration by defining the task and entry point function in the Tasks and queues tab of the FreeRTOS Middleware configuration.

The video task stack size (defined in words for CMSIS V2) and RTOS heap size are two important factors.

For software decoding, the stack size must be carefully set because LibJPEG uses dynamic memory allocation. This means that the FreeRTOS heap should be large enough to accommodate your general application + 0xA000 (~41kB). For hardware decoding, the stack size can be substantially lower because there is no software stack that dynamically allocates memory.

FreeRTOS Task Configuration

FreeRTOS Heap Size Configuration

Based on the above configuration, STM32CubeMX generates the following code for the video task:

main.c
/* Definitions for VideoTask */
osThreadId_t VideoTaskHandle;
const osThreadAttr_t VideoTask_attributes = {
  .name = "VideoTask",
  .stack_size = 1000 * 4,
  .priority = (osPriority_t) osPriorityLow,
};
...
void main()
{
    ...
    /* creation of VideoTask */
    VideoTaskHandle = osThreadNew(videoTaskFunc, NULL, &VideoTask_attributes);
    ...
}
FreeRTOSConfig.h
#define configTOTAL_HEAP_SIZE                    ((size_t)75000)

Software Decoding​

The software decoding solution requires that the LibJPEG middleware is enabled from STM32CubeMX, as specified in Generator User Guide. Software decoding setup is identical for all LibJPEG capable MCUs (e.g. STM32F4, STM32F7, STM32H7).

The TouchGFX Software Decoder expects the data decoded by LibJPEG to have BGR pixel ordering. If this setting is left as RGB, R and B color components will be swapped in your application.

Furthermore, the size of each pixel should be 3 bytes if using a 16-bit or 24-bit video RGB buffer or 4 if using a 32-bit video RGB buffer.

LibJPEG Configuration for a 16-bit or 24-bit video RGB buffer

Caution
  • The RGB_ORDERING setting must be configured as BGR, and the pixel size must be set according to which video RGB buffer format is used.
  • If your application uses Direct to Framebuffer strategy in a ARGB8888 (32-bit) application the pixel size must be set to 4.

Video Data​

TouchGFX links the video data to the ExtFlashSection, which needs to be defined in the linker script. If ExtFlashSection is placed in non-memory-mapped flash, the data must be read from the flash memory using a data reader and handed to the decoder.

For projects with video data in non-memory-mapped memory, please refer to the FileReader section later in this article in this article.

Hardware Decoding​

The Generator User Guide mentions that the JPEG IP must be enabled in STM32CubeMX to enable hardware decoding.

The RGB_FORMAT setting found in the JPEG configurations must respect the pixel format of the TouchGFX Framebuffer, JPEG_RGB565 in the example below.

JPEG IP Parameter Settings

When using hardware decoding, a DMA is used to transfer data to (memory-to-peripheral) and from (peripheral-to-memory) the JPEG IP. Which DMA to use and how to configure it differs between MCU families.

Tip
Here are some examples of TouchGFX Board Setups available in TouchGFX Designer that demonstrate how to set up JPEG decoding with different types of DMAs:
  • STM32U5G9J-DK2: JPEG + GPDMA
  • STM32H750B-DK: JPEG + MDMA
  • STM32H7S78-DK: JPEG + HPDMA

STM32H750 example​

Here is an example of how to set up the DMA on STM32H750 for JPEG decoding. For the STM32H750, the JPEG peripheral must be configured to use the MDMA peripheral instead of the regular DMA peripherals. Add an MDMA configuration for both the input and output FIFO threshold signals, as shown in the image below. The configuration is identical for both the input and output buffers.

JPEG DMA settings for STM32H750

FileReader interface​

When storing MJPEG videos on non-memory-mapped memories developers can specify an implementation of touchgfx::VideoDataReader that the TouchGFX Video Controllers can use to hand data for decoding to the configured decoder (Software/Hardware). The below is a simple example of such an interface that copies video data from one buffer to another.

VideoView.cpp
class MyReader : public touchgfx::VideoDataReader
{
public:
    MyReader() : position(0) { }
    virtual uint32_t getDataLength() { return video_len; }
    virtual void seek(uint32_t pos) { position = pos; }
    virtual bool readData(void* dst, uint32_t bytes)
    {
        memcpy(dst, &video_data[position], bytes);
        position += bytes;
        return true;
    }
    private:
        uint32_t position;
} myReader;

Instead of pointing a Video widget to the start of the video in mapped memory, developers can configure the widget to use the data reader instead:

VideoView.cpp
video.setVideoData(myReader);

Migrating TBS to support video decoding​

If you want to migrate a project created from a TouchGFX Board Setups (TBS) - with a MCU that supports video decoding - from a version of TouchGFX Designer before video decoding was supported and you want to be able to use 'Run Target' in TouchGFX Designer with that project, some manual changes to the GCC Makefile are required.

A run-down of the required changes and why they are needed will be listed in the following sections. There are some general changes that must always be applied and some LibJPEG (software decoding) and JPEG (hardware decoding) specific changes depending on what the developers application uses. The changes are an extension to the already existing GCC Makefile from the older TBS.

Other than updating the Makefile you will also have to setup video decoding in STM32CubeMX as described in the above scenarios.

General changes​

Define the LIBJPEG path in your project:

# LibJPEG path
libjpeg_path := $(cubemx_middlewares_path)/Third_Party/LibJPEG

The video assets input path must then be defined:

asset_texts_input  := TouchGFX/assets/texts

asset_videos_input := TouchGFX/assets/videos

The video assets output path must also be defined bellow the already existing assets output paths:

asset_images_output := $(asset_root_path)/images
asset_fonts_output := $(asset_root_path)/fonts
asset_texts_output := $(asset_root_path)/texts
asset_videos_output := $(asset_root_path)/videos

Add the video output assets to the components list:

all_components := $(components) \
    $(asset_fonts_output) \
    $(asset_images_output) \
    $(asset_texts_output)
    $(asset_texts_output) \
    $(asset_videos_output)

Video object files must be defined. The video object files are separated from the already existing object:

c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)

video_object_files := $(call find, $(asset_videos_output),*.o)

A path to the video converter script must be defined:

textconvert_script_path := $(touchgfx_path)/framework/tools/textconvert
textconvert_executable := $(call find, $(textconvert_script_path), *.rb)
videoconvert_script_path := $(touchgfx_path)/framework/tools/videoconvert

An optional echo can be added to see all video objects files. The video object files must be added to the linking stage. This line has the $(video_object_files) added together with the other object files:

$(binary_output_path)/$(target_executable): $(object_files) $(object_asm_files)
    @echo Video Objects: $(video_object_files)
    @echo Linking $(@)
    @mkdir -p $(@D)
    @mkdir -p $(object_output_path)
    @$(file >$(build_root_path)/objects.tmp) $(foreach F,$(object_files) $(video_object_files),$(file >>$(build_root_path)/objects.tmp,$F))

Add the video rule to the existing assets and .PHONY:

_assets_: BitmapDatabase TextKeysAndLanguages Videos

.PHONY: BitmapDatabase TextKeysAndLanguages Videos

The video rule is added with the video conversion added:

Videos:
    @ruby $(videoconvert_script_path)/videoconvert.rb $(asset_videos_input) $(asset_videos_output)

Lastly update the clean rule to also remove video related output.

_clean_:
    @echo Cleaning: $(board_name)
    @rm -rf $(build_root_path)
    # Do not remove gui_generated
    @rm -rf $(asset_images_output)
    @rm -rf $(asset_fonts_output)
    @rm -rf $(asset_texts_output)
    @rm -rf $(asset_videos_output)
    # Create directory to avoid error if it does not exist
    @mkdir -p $(asset_root_path)
    # Remove assets folder if it is empty (i.e. no gui_generated folder)
    @rmdir --ignore-fail-on-non-empty $(asset_root_path)
    # Clean bootloader project
    @$(MAKE) -r -f ExtMem_Boot/gcc/Makefile -s $(MFLAGS) clean

Software changes​

Add all the LIBJPEG paths to the include paths:

include_paths := $(library_includes) \
    $(foreach comp, $(all_components), $(comp)/include) \
    $(foreach comp, $(cubemx_components), $(comp)/Inc) \
    $(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
    $(framework_includes) \
    $(cubemx_middlewares_path) \
    $(touchgfx_middlewares_path) \
    $(touchgfx_generator_components) \
    LIBJPEG/Target \
    $(libjpeg_path)/include \
    LIBJPEG/App

LIBJPEG source paths must be defined:

c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)

libjpeg_source_path = Middlewares/Third_Party/LibJPEG/source

Next all the LIBJPEG source files must be added to board_c_files:

board_c_files := \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
    $(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
    $(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
    $(libjpeg_source_path)/jaricom.c \
    $(libjpeg_source_path)/jcomapi.c \
    $(libjpeg_source_path)/jdapimin.c \
    $(libjpeg_source_path)/jdapistd.c \
    $(libjpeg_source_path)/jdarith.c \
    $(libjpeg_source_path)/jdatasrc.c \
    $(libjpeg_source_path)/jdcoefct.c \
    $(libjpeg_source_path)/jdcolor.c \
    $(libjpeg_source_path)/jddctmgr.c \
    $(libjpeg_source_path)/jdhuff.c \
    $(libjpeg_source_path)/jdinput.c \
    $(libjpeg_source_path)/jdmainct.c \
    $(libjpeg_source_path)/jdmarker.c \
    $(libjpeg_source_path)/jdmaster.c \
    $(libjpeg_source_path)/jdmerge.c \
    $(libjpeg_source_path)/jdpostct.c \
    $(libjpeg_source_path)/jdsample.c \
    $(libjpeg_source_path)/jdtrans.c \
    $(libjpeg_source_path)/jerror.c \
    $(libjpeg_source_path)/jidctflt.c \
    $(libjpeg_source_path)/jidctfst.c \
    $(libjpeg_source_path)/jidctint.c \
    $(libjpeg_source_path)/jmemmgr.c \
    $(libjpeg_source_path)/jmemnobs.c \
    $(libjpeg_source_path)/jquant1.c \
    $(libjpeg_source_path)/jquant2.c \
    $(libjpeg_source_path)/jutils.c \
    LIBJPEG/App/libjpeg.c

The LIBJPEG source files are then added to the existing object files the same way it is done with the rest of the middleware source files:

# Start converting paths
object_files := $(object_files:$(touchgfx_path)/%.cpp=$(object_output_path)/touchgfx/%.o)
object_files := $(object_files:%.cpp=$(object_output_path)/%.o)
object_files := $(object_files:$(touchgfx_middlewares_path)/%.c=$(object_output_path)/$(touchgfx_middlewares_path)/%.o)
object_files := $(object_files:$(cubemx_middlewares_path)/%.c=$(object_output_path)/$(cubemx_middlewares_path)/%.o)
object_files := $(object_files:$(libjpeg_source_path)/%.c=$(object_output_path)/$(libjpeg_source_path)/%.o)
object_files := $(object_files:$(Drivers_path)/%.c=$(object_output_path)/Drivers/%.o)
object_files := $(object_files:%.c=$(object_output_path)/%.o)

Hardware changes​

Add all the JPEG paths to the include paths:

include_paths := $(library_includes) \
    $(foreach comp, $(all_components), $(comp)/include) \
    $(foreach comp, $(cubemx_components), $(comp)/Inc) \
    $(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
    $(framework_includes) \
    $(cubemx_middlewares_path) \
    $(touchgfx_middlewares_path) \
    $(touchgfx_generator_components) \
    Utilities/JPEG

Next all the JPEG source files must be added to board_c_files:

board_c_files := \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
    $(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
    $(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
    Utilities/JPEG/jpeg_utils.c

Placing Video Buffer in External Memory​

When developers create new projects with STM32CubeMX, the linker scripts associated with the generated projects do not contain default regions used by TouchGFX. Because of this, 'ExtFlashSection' where the video is stored is placed in internal flash by default. It will often be required to place 'ExtFlashSection' in external flash memory to avoid running out of internal flash memory.

Tip
The following TBSs are examples of preconfigured TBSs that are ready to decode videos from external memory:
  1. STM32F746-DISCO
  2. STM32H735G-DK
  3. STM32H750-DK
  4. STM32U5G9J-DK2

If the video strategy selected is not direct to framebuffer, an RGB buffer dedicated to JPEG decoding is generated by TouchGFX Generator when video is enabled. The definition is instrumented with a location pragma that tells the linker which section the buffer should be placed in. If the linker cannot find this memory region in the linker script the buffer will be placed in internal memory.

LOCATION_PRAGMA_NOLOAD("Video_RGB_Buffer")
uint32_t videoRGBBuffer[57600] LOCATION_ATTRIBUTE_NOLOAD("Video_RGB_Buffer");

The following compiler-specific subsections describe an example of the modifications that developers can make to achieve placing the buffer in SDRAM on STM32F746G-DISCO. Video_RGB_Buffer represents the buffer used for video decoding. The linker script examples reserve some space in SDRAM (starting at 0xC0000000) for TouchGFX framebuffer(s).

  1. EWARM
  2. STM32CubeIDE
  3. MDK-ARM
Further reading
Please read the section in the Generator User Guide on configuring a project for Video Decoding.

The following examples all reserves some space at the start of SDRAM on an STM32F746G-DISCO board (0xC0000000->0xC00FF000), allowing the application to reference framebuffers by address (e.g. 0xC0000000) without the risk of the linker overwriting framebuffer data. Each example will allow the linker to place Video_RGB_Buffer into the defined SDRAM region.

Tip
If you're allocating the Touchgfx framebuffers rather than using directly addressing then your linker script should also place TouchGFX_Framebuffer into SDRAM.

EWARM (IAR)​

stm32f746xx_flash.icf
define symbol __ICFEDIT_region_SDRAM_start__   = 0xC00FF000;
define symbol __ICFEDIT_region_SDRAM_end__     = 0xC0700FFF;

define region SDRAM_region    = mem:[from __ICFEDIT_region_SDRAM_start__ to __ICFEDIT_region_SDRAM_end__];

place in SDRAM_region    { first section Video_RGB_Buffer };

After linking, EWARM\STM32F746G_DISCO\List\STM32F746G_DISCO.map contains the following placement information for Video_RGB_Buffer:

STM32F746G_DISCO.map
Video_RGB_Buffer        zero     0xc00f'f000   0x3'8400  TouchGFXGeneratedHAL.o [2]
                               - 0xc013'7400   0x3'8400

STM32CubeIDE​

STM32F746NGHX_FLASH.ld
MEMORY
{
  ...
  SDRAM      (xrw)    : ORIGIN = 0xC00FF000,   LENGTH = 8M
}

BufferSection (NOLOAD) :
{
  *(Video_RGB_Buffer Video_RGB_Buffer.*)
  *(.gnu.linkonce.r.*)
  . = ALIGN(0x4);
} >SDRAM

After compilation, STM32CubeIDE\Debug\STM32F746G_DISCO.map contains the following placement information for Video_RGB_Buffer:

STM32F746G_DISCO.map
BufferSection   0x00000000c00ff000    0x1c200
 *(Video_RGB_Buffer Video_RGB_Buffer.*)
 Video_RGB_Buffer
                0x00000000c00ff000    0x1c200 Application/User/TouchGFX/target/generated/TouchGFXGeneratedHAL.o
                0x00000000c00ff000                videoRGBBuffer

MDK-ARM (Keil)​

STM32F746G_DISCO.sct
LR_IROM1 0x08000000 0x00200000  {    ; load region size_region
  ER_IROM1 0x08000000 0x00200000  {  ; load address = execution address
    *.o (RESET, +First)
    *(InRoot$$Sections)
    .ANY (+RO)
  }
  RW_IRAM1 0x20000000 0x00050000  {  ; RW data
    .ANY (+RW +ZI)
  }
  RW_SDRAM 0xC00FF000 UNINIT 0xC0700FFF  {
    *.o (Video_RGB_Buffer)
  }
}

Make sure to include the UNINIT attribute in the section that holds the Video Buffer. This ensures leaving the memory region as uninitialized data. After linking, MDK-ARM\STM32F746G_DISCO\STM32F746G_DISCO.map contains the following placement information for Video_RGB_Buffer:

STM32F746G_DISCO.map
Video_RGB_Buffer                         0xc00ff000   Section    115200  touchgfxgeneratedhal.o(Video_RGB_Buffer)