Video Decoding

This section shows how to configure the TouchGFX Generator to generate a TouchGFX HAL that has video decoding capabilities.

Before reading this scenario, please read the documentation on using MJPEG Video in TouchGFX and the documentation on the Video widget itself.

The following scenario details how to enable TouchGFX HAL to support video decoding through either software (LibJPEG) or hardware (JPEG). It covers both STM32F7 and STM32H7 because the STM32CubeMX configuration to support Hardware (JPEG) decoding is slightly different from the STM32F7.

Be sure to read all sections in this article before configuring the MCU for your specific project.

Generally, the LibJPEG and JPEG configurations can be found in STM32CubeMX in the Middleware and Multimedia catagories:

JPEG and LibJPEG setting in STM32CubeMX

Tip
The latest version of the STM32H750-DK, STM32F769-DISCO and STM32F746G-DISCO TouchGFX Board Setups (TBS) support both Software and JPEG hardware decoding through their STM32CubeMX configurations.
Use the STM32CubeMX configurations found in these example projects as inspiration to get your specific MCU to support MJPEG Video Decoding.

RTOS Support

The Generator User Guide mentions that Single- and Double buffer decoding strategies require a CMSIS compliant RTOS, such as FreeRTOS. TouchGFX Generator generates an entry point function videoTaskFunc(), that must be associated with a Video decoding task. STM32CubeMX can generate this configuration by defining the task and entry point function in the Tasks and queues tab of the FreeRTOS Middleware configuration.

The video task stack size (defined in words for CMSIS V2) and RTOS heap size are two important factors.

For software decoding the stack size must be carefully set because LibJPEG uses dynamic memory allocation. For hardware decoding the stack size can be substantially lower because there is no software stack which dynamically allocates memory.

The FreeRTOS Heap should be large enough for your general application + 0xA000.

FreeRTOS Task Configuration

FreeRTOS Heap Size Configuration

Based on the above configuration, STM32CubeMX generates the following code for the video task:

main.c

/* Definitions for VideoTask */
osThreadId_t VideoTaskHandle;
const osThreadAttr_t VideoTask_attributes = {
  .name = "VideoTask",
  .stack_size = 1000 * 4,
  .priority = (osPriority_t) osPriorityLow,
};
...
void main()
{
    ...
    /* creation of VideoTask */
    VideoTaskHandle = osThreadNew(videoTaskFunc, NULL, &VideoTask_attributes);
    ...
}

FreeRTOSConfig.h

#define configTOTAL_HEAP_SIZE                    ((size_t)75000)

Software Decoding

The software decoding solution requires that the LibJPEG middleware is enabled from STM32CubeMX, as specified in Generator User Guide. Software decoding setup is identical for all LibJPEG capable MCUs (e.g. STM32F4, STM32F7, STM32H7).

The TouchGFX Software Decoder expects the data decoded by LibJPEG to have BGR pixel ordering. If this setting is left as RGB, R and B color components will be swapped in your application.

Furthermore, the size of each pixel should be 3 bytes if using a 16-bit or 24-bit video RGB buffer or 4 if using a 32-bit video RGB buffer.

LibJPEG Configuration for a 16-bit or 24-bit video RGB buffer

Caution
The RGB_ORDERING setting must be configured as BGR, and the pixel size must be set according to which video RGB buffer format is used.
If your application uses Direct to Framebuffer strategy in a ARGB8888 (32-bit) application the pixel size must be set to 4

Video Data

Caution
The current decoding solution using STM32CubeMX, TouchGFX Generator and the Video widget from TouchGFX Designer will currently only work for ST boards because the video data is linked into the application at a specific address in memory mapped flash that is already known.

The below snippet from an EWARM project shows the extra options that are created by TouchGFX Designer and how it places the data in ExtFlashSection which is defined inside the linker script for all TouchGFX Board Packages. This will not work for non-memory-mapped flash or projects that don't define this section.

ewarm_project.ewp

        <option>
          <name>IlinkExtraOptions</name>
          <state>--image_input $PROJ_DIR$\..\TouchGFX\generated\videos\bin\washerdryer.bin,video_washerdryer_bin_start,ExtFlashSection,4</state>
          <state>--keep video_washerdryer_bin_start</state>
        </option>

For projects with video data in non-memory mapped memory, please read the section FileReader in this article.

Hardware Decoding

The Generator User Guide mentions that the JPEG IP must be enabled in STM32CubeMX to enable Hardware decoding.

STM32F769-DISCO

The JPEG configuration for JPEG capable STM32F7 series, e.g. STM32F769, differs slightly from the STM32H7 line. The RGB_FORMAT setting found in the JPEG configurations must respect the pixel format of the TouchGFX Framebuffer, JPEG_RGB565 in the example below.

JPEG Parameter Settings

Use DMA to transfer data to (memory-to-peripheral) and from (peripheral-to-memory) the JPEG peripheral through the DMA Settings tab found in JPEG configurations. Adding a DMA request for IN and one for OUT will set up the direction parameters automatically.

JPEG DMA Settings

This concludes the TouchGFX HAL configuration for Hardware Decoding on the JPEG capable STM32F7s (e.g. STM32F769). After Generating code from STM32CubeMX the application will be able to decode videos using the JPEG peripheral when using the Video widget from the designer.

Caution
The RGB_FORMAT of the JPEG peripheral configuration must respect the format of the TouchGFX Framebuffer

STM32H750-DK

The only thing that seperates Hardware decoding (JPEG capable) on e.g. an STM32H750 from an STM32F769 is the way DMA transfers are configured in STM32CubeMX. Not only is the UI different, but the DMA concepts are as well.

For the STM32H750, the JPEG peripheral can only be configured to use the MDMA peripheral rather than the regular DMA peripherals. Add an MDMA configuration for both input- and output FIFO Threshold signals as shown in the images below.

Note
MDMA is a much higher performance DMA engine compared to DMA1/DMA2

JPEG DMA Settings

`FileReader` interface

When storing MJPEG videos on non-memory-mapped memories developers can specify an implementation of touchgfx::VideoDataReader that the TouchGFX Video Controllers can use to hand data for decoding to the configured decoder (Software/Hardware). The below is a simple example of such an interface that copies video data from one buffer to another.

VideoView.cpp

class MyReader : public touchgfx::VideoDataReader
{
public:
    MyReader() : position(0) { }
    virtual uint32_t getDataLength() { return video_len; }
    virtual void seek(uint32_t pos) { position = pos; }
    virtual bool readData(void* dst, uint32_t bytes)
    {
        memcpy(dst, &video_data[position], bytes);
        position += bytes;
        return true;
    }
    private:
        uint32_t position;
} myReader;

Instead of pointing a Video widget to the start of the video in mapped memory, developers can configure the widget to use the data reader instead:

VideoView.cpp

video.setVideoData(myReader);

Migrating TBS to support video decoding

If you want to migrate a project created from a TouchGFX Board Setups (TBS) - with a MCU that supports video decoding - from a version of TouchGFX Designer before video decoding was supported and you want to be able to use 'Run Target' in TouchGFX Designer with that project, some manual changes to the GCC Makefile are required.

A run-down of the required changes and why they are needed will be listed in the following sections. There are some general changes that must always be applied and some LibJPEG (software decoding) and JPEG (hardware decoding) specific changes depending on what the developers application uses. The changes are an extension to the already existing GCC Makefile from the older TBS.

Other than updating the Makefile you will also have to setup video decoding in STM32CubeMX as described in the above scenarios.

General changes

Define the LIBJPEG path in your project:

# LibJPEG path
libjpeg_path := $(cubemx_middlewares_path)/Third_Party/LibJPEG

The video assets input path must then be defined:

asset_texts_input  := TouchGFX/assets/texts

asset_videos_input := TouchGFX/assets/videos

The video assets output path must also be defined bellow the already existing assets output paths:

asset_images_output := $(asset_root_path)/images
asset_fonts_output := $(asset_root_path)/fonts
asset_texts_output := $(asset_root_path)/texts
asset_videos_output := $(asset_root_path)/videos

Add the video output assets to the components list:

all_components := $(components) \
    $(asset_fonts_output) \
    $(asset_images_output) \
    $(asset_texts_output)
    $(asset_texts_output) \
    $(asset_videos_output)

Video object files must be defined. The video object files are separated from the already existing object:

c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)

video_object_files := $(call find, $(asset_videos_output),*.o)

A path to the video converter script must be defined:

textconvert_script_path := $(touchgfx_path)/framework/tools/textconvert
textconvert_executable := $(call find, $(textconvert_script_path), *.rb)
videoconvert_script_path := $(touchgfx_path)/framework/tools/videoconvert

An optional echo can be added to see all video objects files. The video object files must be added to the linking stage. This line has the $(video_object_files) added together with the other object files:

$(binary_output_path)/$(target_executable): $(object_files) $(object_asm_files)
    @echo Video Objects: $(video_object_files)
    @echo Linking $(@)
    @mkdir -p $(@D)
    @mkdir -p $(object_output_path)
    @$(file >$(build_root_path)/objects.tmp) $(foreach F,$(object_files) $(video_object_files),$(file >>$(build_root_path)/objects.tmp,$F))

Add the video rule to the existing assets and .PHONY:

_assets_: BitmapDatabase TextKeysAndLanguages Videos

.PHONY: BitmapDatabase TextKeysAndLanguages Videos

The video rule is added with the video conversion added:

Videos:
    @ruby $(videoconvert_script_path)/videoconvert.rb $(asset_videos_input) $(asset_videos_output)

Lastly update the clean rule to also remove video related output.

_clean_:
    @echo Cleaning: $(board_name)
    @rm -rf $(build_root_path)
    # Do not remove gui_generated
    @rm -rf $(asset_images_output)
    @rm -rf $(asset_fonts_output)
    @rm -rf $(asset_texts_output)
    @rm -rf $(asset_videos_output)
    # Create directory to avoid error if it does not exist
    @mkdir -p $(asset_root_path)
    # Remove assets folder if it is empty (i.e. no gui_generated folder)
    @rmdir --ignore-fail-on-non-empty $(asset_root_path)
    # Clean bootloader project
    @$(MAKE) -r -f ExtMem_Boot/gcc/Makefile -s $(MFLAGS) clean

Software changes

Add all the LIBJPEG paths to the include paths:

include_paths := $(library_includes) \
    $(foreach comp, $(all_components), $(comp)/include) \
    $(foreach comp, $(cubemx_components), $(comp)/Inc) \
    $(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
    $(framework_includes) \
    $(cubemx_middlewares_path) \
    $(touchgfx_middlewares_path) \
    $(touchgfx_generator_components) \
    LIBJPEG/Target \
    $(libjpeg_path)/include \
    LIBJPEG/App

LIBJPEG source paths must be defined:

c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)

libjpeg_source_path = Middlewares/Third_Party/LibJPEG/source

Next all the LIBJPEG source files must be added to board_c_files:

board_c_files := \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
    $(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
    $(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
    $(libjpeg_source_path)/jaricom.c \
    $(libjpeg_source_path)/jcomapi.c \
    $(libjpeg_source_path)/jdapimin.c \
    $(libjpeg_source_path)/jdapistd.c \
    $(libjpeg_source_path)/jdarith.c \
    $(libjpeg_source_path)/jdatasrc.c \
    $(libjpeg_source_path)/jdcoefct.c \
    $(libjpeg_source_path)/jdcolor.c \
    $(libjpeg_source_path)/jddctmgr.c \
    $(libjpeg_source_path)/jdhuff.c \
    $(libjpeg_source_path)/jdinput.c \
    $(libjpeg_source_path)/jdmainct.c \
    $(libjpeg_source_path)/jdmarker.c \
    $(libjpeg_source_path)/jdmaster.c \
    $(libjpeg_source_path)/jdmerge.c \
    $(libjpeg_source_path)/jdpostct.c \
    $(libjpeg_source_path)/jdsample.c \
    $(libjpeg_source_path)/jdtrans.c \
    $(libjpeg_source_path)/jerror.c \
    $(libjpeg_source_path)/jidctflt.c \
    $(libjpeg_source_path)/jidctfst.c \
    $(libjpeg_source_path)/jidctint.c \
    $(libjpeg_source_path)/jmemmgr.c \
    $(libjpeg_source_path)/jmemnobs.c \
    $(libjpeg_source_path)/jquant1.c \
    $(libjpeg_source_path)/jquant2.c \
    $(libjpeg_source_path)/jutils.c \
    LIBJPEG/App/libjpeg.c

The LIBJPEG source files are then added to the existing object files the same way it is done with the rest of the middleware source files:

# Start converting paths
object_files := $(object_files:$(touchgfx_path)/%.cpp=$(object_output_path)/touchgfx/%.o)
object_files := $(object_files:%.cpp=$(object_output_path)/%.o)
object_files := $(object_files:$(touchgfx_middlewares_path)/%.c=$(object_output_path)/$(touchgfx_middlewares_path)/%.o)
object_files := $(object_files:$(cubemx_middlewares_path)/%.c=$(object_output_path)/$(cubemx_middlewares_path)/%.o)
object_files := $(object_files:$(libjpeg_source_path)/%.c=$(object_output_path)/$(libjpeg_source_path)/%.o)
object_files := $(object_files:$(Drivers_path)/%.c=$(object_output_path)/Drivers/%.o)
object_files := $(object_files:%.c=$(object_output_path)/%.o)

Hardware changes

Add all the JPEG paths to the include paths:

include_paths := $(library_includes) \
    $(foreach comp, $(all_components), $(comp)/include) \
    $(foreach comp, $(cubemx_components), $(comp)/Inc) \
    $(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
    $(framework_includes) \
    $(cubemx_middlewares_path) \
    $(touchgfx_middlewares_path) \
    $(touchgfx_generator_components) \
    Utilities/JPEG

Next all the JPEG source files must be added to board_c_files:

board_c_files := \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
    $(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
    $(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
    Utilities/JPEG/jpeg_utils.c

Placing Video Buffer in External Memory

When developers create new projects with STM32CubeMX, the linker scripts associated with the generated projects do not contain any of the possible regions used by TouchGFX. Because of this, the specific buffer used to decode MJPEG videos is placed in internal flash by the linker until developers modify their linker scripts to place the buffer elsewhere. Without such modifications developers will experience large internal memory usage and perhaps inability to fit a large video buffer, like for full screen video decoding.

Tip
The following TBSs are preconfigured and ready to decode videos from external memory:STM32F746-DISCO
STM32F769-DISCO
STM32H750-DK

The definition for the RGB buffer dedicated to JPEG decoding is generated by TouchGFX Generator when video decoding is enabled. The definition is instrumented with a location pragma that tells the linker which section the buffer should be placed in. If the linker cannot find this memory region in the linker script the buffer will be placed in internal memory.

LOCATION_PRAGMA_NOLOAD("Video_RGB_Buffer")
uint32_t videoRGBBuffer[57600] LOCATION_ATTRIBUTE_NOLOAD("Video_RGB_Buffer");

The following compiler-specific subsections describe the modifications that developers can make to achieve placing the buffer in SDRAM. Video_RGB_Buffer represents the buffer used for video decoding. The linker script examples reserve some space in SDRAM (starting at 0xC0000000) for TouchGFX framebuffer(s).

EWARM
STM32CubeIDE
MDK-ARM

EWARM (IAR)

stm32f746xx_flash.icf

define symbol __ICFEDIT_region_SDRAM_start__   = 0xC00FF000;
define symbol __ICFEDIT_region_SDRAM_end__     = 0xC0700FFF;

define region SDRAM_region    = mem:[from __ICFEDIT_region_SDRAM_start__ to __ICFEDIT_region_SDRAM_end__];

place in SDRAM_region    { first section Video_RGB_Buffer };

After linking, EWARM\STM32F746G_DISCO\List\STM32F746G_DISCO.map contains the following placement information for Video_RGB_Buffer:

STM32F746G_DISCO.map

Video_RGB_Buffer        zero     0xc00f'f000   0x3'8400  TouchGFXGeneratedHAL.o [2]
                               - 0xc013'7400   0x3'8400

STM32CubeIDE

STM32F746NGHX_FLASH.ld

MEMORY
{
  ...
  SDRAM      (xrw)    : ORIGIN = 0xC00FF000,   LENGTH = 8M
}

BufferSection (NOLOAD) :
{
  *(Video_RGB_Buffer Video_RGB_Buffer.*)
  *(.gnu.linkonce.r.*)
  . = ALIGN(0x4);
} >SDRAM

After compilation, STM32CubeIDE\Debug\STM32F746G_DISCO.map contains the following placement information for Video_RGB_Buffer:

STM32F746G_DISCO.map

BufferSection   0x00000000c00ff000    0x1c200
 *(Video_RGB_Buffer Video_RGB_Buffer.*)
 Video_RGB_Buffer
                0x00000000c00ff000    0x1c200 Application/User/TouchGFX/target/generated/TouchGFXGeneratedHAL.o
                0x00000000c00ff000                videoRGBBuffer

MDK-ARM (Keil)

STM32F746G_DISCO.sct

LR_IROM1 0x08000000 0x00200000  {    ; load region size_region
  ER_IROM1 0x08000000 0x00200000  {  ; load address = execution address
    *.o (RESET, +First)
    *(InRoot$$Sections)
    .ANY (+RO)
  }
  RW_IRAM1 0x20000000 0x00050000  {  ; RW data
    .ANY (+RW +ZI)
  }
  RW_SDRAM 0xC00FF000 UNINIT 0xC0700FFF  {
    *.o (Video_RGB_Buffer)
  }
}

Make sure to include the UNINIT attribute in the section that holds the Video Buffer. This ensures leaving the memory region as uninitialized data. After linking, MDK-ARM\STM32F746G_DISCO\STM32F746G_DISCO.map contains the following placement information for Video_RGB_Buffer:

STM32F746G_DISCO.map

Video_RGB_Buffer                         0xc00ff000   Section    115200  touchgfxgeneratedhal.o(Video_RGB_Buffer)

Tip

RTOS Support​

main.c

FreeRTOSConfig.h

Software Decoding​

Caution

Video Data​

Caution

ewarm_project.ewp

Hardware Decoding​

STM32F769-DISCO​

Caution

STM32H750-DK​

Note

FileReader interface​

VideoView.cpp

VideoView.cpp

Migrating TBS to support video decoding​

General changes​

Software changes​

Hardware changes​

Placing Video Buffer in External Memory​

Tip

Further reading

Tip

EWARM (IAR)​

stm32f746xx_flash.icf

STM32F746G_DISCO.map

STM32CubeIDE​

STM32F746NGHX_FLASH.ld

STM32F746G_DISCO.map

MDK-ARM (Keil)​

STM32F746G_DISCO.sct

STM32F746G_DISCO.map

RTOS Support

Software Decoding

Video Data

Hardware Decoding

STM32F769-DISCO

STM32H750-DK

`FileReader` interface

Migrating TBS to support video decoding

General changes

Software changes

Hardware changes

Placing Video Buffer in External Memory

EWARM (IAR)

STM32CubeIDE

MDK-ARM (Keil)