Skip to main content

Video Decoding

This section shows how to configure the TouchGFX Generator to generate a TouchGFX HAL that has video decoding capabilities.

The following scenario details how to enable TouchGFX HAL to support video decoding through either software (LibJPEG) or hardware (JPEG). It covers both STM32F7 and STM32H7 because the STM32CubeMX configuration to support Hardware (JPEG) decoding is slightly different from the STM32F7.

Generally, the LibJPEG and JPEG configurations can be found in STM32CubeMX in the Middleware and Multimedia catagories:

Single framebuffer, by address

Before reading this scenario, please read the documentation on using MJPEG Video in TouchGFX and the documentation on the Video widget itself.

Be sure to read all sections in this article before proceeding.

Tip
The latest version of the STM32H750-DK, STM32F769-DISCO and STM32F746G-DISCO TouchGFX Board Setups (TBS) support both Software and JPEG hardware decoding through their STM32CubeMX configurations.

Software Decoding​

The software decoding solution requires that the LibJPEG middleware is enabled from STM32CubeMX, as specified in TouchGFX Generator User Guide. Software decoding setup is identical for all LibJPEG capable MCUs (e.g. STM32F4, STM32F7, STM32H7).

The TouchGFX Software Decoder expects the data decoded by LibJPEG to have BGR pixel ordering. If this setting is left as RGB, R and B color components will be swapped in your application. Furthermore, the size of each pixel should be 3 bytes (4 bytes means the video is encoded in an XRGB format).

Caution
The RGB_ORDERING setting must be configured as BGR, and the pixel size must be 3 (24-bit).

LibJPEG Configuration

Once LibJPEG is enabled from STM32CubeMX, Software Decoding can now be enabled from TouchGFX Generator, as described in the TouchGFX Generator User Guide.

TouchGFX Generator: Video Decoding

RTOS Support​

The TouchGFX Generator User Guide mentions that Single- and Double buffer decoding strategies require a CMSIS compliant RTOS, such as FreeRTOS. TouchGFX Generator generates an entry point function videoTaskFunc that must be associated with a Video decoding task. STM32CubeMX can generate this configuration by defining the task and entry point function in the Tasks and queues tab of the FreeRTOS Middleware configuration.

The video task stack size (defined in words for CMSIS V2) and RTOS heap size are two important factors, the latter because LibJPEG, for software decoding, uses dynamic memory allocation. The FreeRTOS Heap should be large enough for your general application + 0xA000.

FreeRTOS Task Configuration

FreeRTOS Heap Size Configuration

Based on the above configuration, STM32CubeMX generates the following code:

main.c
/* Definitions for VideoTask */
osThreadId_t VideoTaskHandle;
const osThreadAttr_t VideoTask_attributes = {
  .name = "VideoTask",
  .stack_size = 1000 * 4,
  .priority = (osPriority_t) osPriorityLow,
};

void main()
{
    ...
    /* creation of VideoTask */
    VideoTaskHandle = osThreadNew(videoTaskFunc, NULL, &VideoTask_attributes);
    ...
}
FreeRTOSConfig.h
#define configTOTAL_HEAP_SIZE                    ((size_t)75000)

This concludes the TouchGFX HAL configuration for Software Decoding. After Generating code from STM32CubeMX the application will be able to decode videos using LibJPEG when using the Video widget from the designer.

Video Data​

Caution
The current decoding solution using STM32CubeMX, TouchGFX Generator and the Video widget from TouchGFX Designer will currently only work for ST boards because the video data is linked into the application at a specific address in memory mapped flash that is already known.

The below snippet from an EWARM project shows the extra options that are created by TouchGFX Designer and how it places the data in ExtFlashSection which is defined inside the linker script for all TouchGFX Board Packages. This will not work for non-memory-mapped flash or projects that don't define this section.

ewarm_project.ewp
        <option>
          <name>IlinkExtraOptions</name>
          <state>--image_input $PROJ_DIR$\..\TouchGFX\generated\videos\bin\washerdryer.bin,video_washerdryer_bin_start,ExtFlashSection,4</state>
          <state>--keep video_washerdryer_bin_start</state>
        </option>

For projects with video data in non-memory mapped memory, please read the section FileReader in this article.

Hardware Decoding​

The TouchGFX Generator User Guide mentions that the JPEG IP must be enabled in STM32CubeMX to enable Hardware decoding. Software- and Hardware decoding share several traits and the differences will be outlined in this section.

RTOS Support​

FreeRTOS stack size can be substantially lower than for Software decoding because there is no software stack which dynamically allocates memory. Keep the size close to what it would be to accommodate the other tasks in the project.

STM32F769-DISCO​

The JPEG configuration for JPEG capable STM32F7 series, e.g. STM32F769, differs slightly from the STM32H7 line. The RGB_FORMAT must respect the format of the TouchGFX Framebuffer, JPEG_RGB565 in the example below.

JPEG Parameter Settings

Use DMA to transfer data to (memory-to-peripheral) and from (peripheral-to-memory) the JPEG peripheral through the DMA Settings. Adding a DMA request for IN and one for OUT will set up the direction parameters automatically.

JPEG DMA Settings

JPEG DMA Settings

This concludes the TouchGFX HAL configuration for Hardware Decoding on the JPEG capable STM32F7s (e.g. STM32F769). After Generating code from STM32CubeMX the application will be able to decode videos using the JPEG peripheral when using the Video widget from the designer.

Caution
The RGB_FORMAT of the JPEG peripheral configuration must respect the format of the TouchGFX Framebuffer

STM32H750-DK​

The only thing that seperates Hardware decoding (JPEG capable) on e.g. an STM32H750 from an STM32F769 is the way DMA transfers are configured in STM32CubeMX. Not only is the UI different, but the DMA concepts are as well.

For the STM32H750, the JPEG peripheral can only be configured to use the MDMA peripheral rather than the regular DMA peripherals. Add an MDMA configuration for both input- and output FIFO Threshold signals as shown in the images below.

Note
MDMA is a much higher performance DMA engine compared to DMA1/DMA2

JPEG DMA Settings

JPEG DMA Settings

Caution
For H7, *User Code* must be added to correctly configure the DMA requests for data in/out.

When generating code for e.g. an STM32H750 through STM32CubeMX, unfortunately, the code to configure the MDMA (defined above) to the MDMA handlers is missing and developers must add the highlighted user code, manually.

Core\Src\stm32h7xx_hal_msp.c
void HAL_JPEG_MspInit(JPEG_HandleTypeDef* hjpeg)
{
  if(hjpeg->Instance==JPEG)
  {
  /* USER CODE BEGIN JPEG_MspInit 0 */
     hmdma_jpeg_infifo_th.Init.Request = MDMA_REQUEST_JPEG_INFIFO_TH;
     hmdma_jpeg_outfifo_th.Init.Request = MDMA_REQUEST_JPEG_OUTFIFO_TH;
  /* USER CODE END JPEG_MspInit 0 */
   ...

FileReader interface​

When storing MJPEG videos on non-memory-mapped memories developers can specify an implementation of touchgfx::VideoDataReader that the TouchGFX Video Controllers can use to hand data for decoding to the configured decoder (Software/Hardware). The below is a simple example of such an interface that copies video data from one buffer to another.

VideoView.cpp
class MyReader : public touchgfx::VideoDataReader
{
public:
    MyReader() : position(0) { }
    virtual uint32_t getDataLength() { return video_len; }
    virtual void seek(uint32_t pos) { position = pos; }
    virtual bool readData(void* dst, uint32_t bytes)
    {
        memcpy(dst, &video_data[position], bytes);
        position += bytes;
        return true;
    }
    private:
        uint32_t position;
} myReader;

Instead of pointing a Video widget to the start of the video in mapped memory, developers can configure the widget to use the data reader instead:

VideoView.cpp
video.setVideoData(myReader);

Migrating TBS to support video decoding​

If you want to migrate a project created from a TBS of one of the boards supporting video decoding before it were available and you want to be able to use 'Run Target' in TouchGFX Desginer with that project, some manual changes to the GCC Makefile are required. A run-down of the required changes and why they are needed will be listed in the following sections. The changes are an extension to the already existing GCC Makefile from the older TBS.

Other than updating the Makefile you will also have to setup video decoding in STM32CubeMX as described in the above scenarios.

Generel changes​

Define the LIBJPEG path in your project:

# LibJPEG path
libjpeg_path := $(cubemx_middlewares_path)/Third_Party/LibJPEG

The video assets input path must then be defined:

asset_texts_input  := TouchGFX/assets/texts
 
asset_videos_input := TouchGFX/assets/videos

The video assets output path must also be defined bellow the other assets output paths:

asset_images_output := $(asset_root_path)/images
asset_fonts_output := $(asset_root_path)/fonts
asset_texts_output := $(asset_root_path)/texts
asset_videos_output := $(asset_root_path)/videos

Add the video output assets to the components list:

all_components := $(components) \
    $(asset_fonts_output) \
    $(asset_images_output) \
    $(asset_texts_output)
    $(asset_texts_output) \
    $(asset_videos_output)

Video object files must be defined. The video object files are seperated from the already existing object:

c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)

video_object_files := $(call find, $(asset_videos_output),*.o)

A path to the video converter script must be defined:

textconvert_script_path := $(touchgfx_path)/framework/tools/textconvert
textconvert_executable := $(call find, $(textconvert_script_path), *.rb)
videoconvert_script_path := $(touchgfx_path)/framework/tools/videoconvert

An optional Echo can be added to see all video objects files. The video object files must be added to the linking stage. This line has the $(video_object_files) added together with the other object files:

$(binary_output_path)/$(target_executable): $(object_files) $(object_asm_files)
    @echo Video Objects: $(video_object_files)
    @echo Linking $(@)
    @mkdir -p $(@D)
    @mkdir -p $(object_output_path)
    @$(file >$(build_root_path)/objects.tmp) $(foreach F,$(object_files) $(video_object_files),$(file >>$(build_root_path)/objects.tmp,$F))

Add the video rule to the existing assets and .PHONY:

_assets_: BitmapDatabase TextKeysAndLanguages Videos

.PHONY: BitmapDatabase TextKeysAndLanguages Videos

The video rule is added with the video conversion added:

Videos:
    @ruby $(videoconvert_script_path)/videoconvert.rb $(asset_videos_input) $(asset_videos_output)

Lastly update the clean rule to also remove video related output. 

_clean_:
    @echo Cleaning: $(board_name)
    @rm -rf $(build_root_path)
    # Do not remove gui_generated
    @rm -rf $(asset_images_output)
    @rm -rf $(asset_fonts_output)
    @rm -rf $(asset_texts_output)
    @rm -rf $(asset_videos_output)  
    # Create directory to avoid error if it does not exist
    @mkdir -p $(asset_root_path)
    # Remove assets folder if it is empty (i.e. no gui_generated folder)
    @rmdir --ignore-fail-on-non-empty $(asset_root_path)
    # Clean bootloader project
    @$(MAKE) -r -f ExtMem_Boot/gcc/Makefile -s $(MFLAGS) clean

Software changes​

Add all the LIBJPEG paths to the include paths:

include_paths := $(library_includes) \
    $(foreach comp, $(all_components), $(comp)/include) \
    $(foreach comp, $(cubemx_components), $(comp)/Inc) \
    $(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
    $(framework_includes) \
    $(cubemx_middlewares_path) \
    $(touchgfx_middlewares_path) \
    $(touchgfx_generator_components) \
    LIBJPEG/Target \
    $(libjpeg_path)/include \
    LIBJPEG/App

LIBJPEG source paths must be defined:

c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)

libjpeg_source_path = Middlewares/Third_Party/LibJPEG/source

Next all the LIBJPEG source files must be added to board_c_files:

board_c_files := \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
    $(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
    $(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
    $(libjpeg_source_path)/jaricom.c \
    $(libjpeg_source_path)/jcomapi.c \
    $(libjpeg_source_path)/jdapimin.c \
    $(libjpeg_source_path)/jdapistd.c \
    $(libjpeg_source_path)/jdarith.c \
    $(libjpeg_source_path)/jdatasrc.c \
    $(libjpeg_source_path)/jdcoefct.c \
    $(libjpeg_source_path)/jdcolor.c \
    $(libjpeg_source_path)/jddctmgr.c \
    $(libjpeg_source_path)/jdhuff.c \
    $(libjpeg_source_path)/jdinput.c \
    $(libjpeg_source_path)/jdmainct.c \
    $(libjpeg_source_path)/jdmarker.c \
    $(libjpeg_source_path)/jdmaster.c \
    $(libjpeg_source_path)/jdmerge.c \
    $(libjpeg_source_path)/jdpostct.c \
    $(libjpeg_source_path)/jdsample.c \
    $(libjpeg_source_path)/jdtrans.c \
    $(libjpeg_source_path)/jerror.c \
    $(libjpeg_source_path)/jidctflt.c \
    $(libjpeg_source_path)/jidctfst.c \
    $(libjpeg_source_path)/jidctint.c \
    $(libjpeg_source_path)/jmemmgr.c \
    $(libjpeg_source_path)/jmemnobs.c \
    $(libjpeg_source_path)/jquant1.c \
    $(libjpeg_source_path)/jquant2.c \
    $(libjpeg_source_path)/jutils.c \
    LIBJPEG/App/libjpeg.c

The LIBJPEG source files are then added to the existing object files the same way it is done with the rest of the middleware source files:

# Start converting paths
object_files := $(object_files:$(touchgfx_path)/%.cpp=$(object_output_path)/touchgfx/%.o)
object_files := $(object_files:%.cpp=$(object_output_path)/%.o)
object_files := $(object_files:$(touchgfx_middlewares_path)/%.c=$(object_output_path)/$(touchgfx_middlewares_path)/%.o)
object_files := $(object_files:$(cubemx_middlewares_path)/%.c=$(object_output_path)/$(cubemx_middlewares_path)/%.o)
object_files := $(object_files:$(libjpeg_source_path)/%.c=$(object_output_path)/$(libjpeg_source_path)/%.o)
object_files := $(object_files:$(Drivers_path)/%.c=$(object_output_path)/Drivers/%.o)
object_files := $(object_files:%.c=$(object_output_path)/%.o)

Hardware changes​

Add all the JPEG paths to the include paths:

include_paths := $(library_includes) \
    $(foreach comp, $(all_components), $(comp)/include) \
    $(foreach comp, $(cubemx_components), $(comp)/Inc) \
    $(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
    $(framework_includes) \
    $(cubemx_middlewares_path) \
    $(touchgfx_middlewares_path) \
    $(touchgfx_generator_components) \
    Utilities/JPEG

Next all the JPEG source files must be added to board_c_files:

board_c_files := \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
    $(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
    $(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
    Utilities/JPEG/jpeg_utils.c

Placing Video Buffer in External Memory​

When developers create new projects with CubeMX, the linker scripts associated with the generated projects do not contain any of the possible regions used by TouchGFX. Because of this, the specific buffer used to decode MJPEG videos is placed in internal flash by the linker until developers modify their linker scripts to place the buffer elsewhere. Without such modifications developers will experience large internal memory usage and perhaps inability to fit a large video buffer, like for full screen video decoding.

Tip
The following TBSs are preconfigured and ready to decode videos from external memory:
  1. STM32F746-DISCO
  2. STM32F769-DISCO
  3. STM32H750-DK

The definition for the RGB buffer dedicated to JPEG decoding is generated by TouchGFX Generator when video decoding is enabled. The definition is instrumented with a location pragma that tells the linker which section the buffer should be placed in. If the linker cannot find this memory region in the linker script the buffer will be placed in internal memory. 

LOCATION_PRAGMA("Video_RGB_Buffer")
uint32_t videoRGBBuffer[57600] LOCATION_ATTRIBUTE("Video_RGB_Buffer");

The following compiler-specific subsections describe the modifications that developers can make to achieve placing the buffer in SDRAM. Video_RGB_Buffer represents the buffer used for video decoding. The linker script examples reserve some space in SDRAM (starting at 0xC0000000) for TouchGFX framebuffer(s).

  1. EWARM
  2. STM32CubeIDE
  3. MDK-ARM
Further reading
Please read the section in the TouchGFX Generator User Guide on configuring a project for Video Decoding.

The following examples all reserves some space at the start of SDRAM on an STM32F746G-DISCO board (0xC0000000->0xC00FF000), allowing the application to reference framebuffers by address (e.g. 0xC0000000) without the risk of the linker overwriting framebuffer data. Each example will allow the linker to place Video_RGB_Buffer into the defined SDRAM region.

Tip
If you're allocating the Touchgfx framebuffers rather than using directly addressing then your linker script should also place TouchGFX_Framebuffer into SDRAM.

EWARM (IAR)​

stm32f746xx_flash.icf
define symbol __ICFEDIT_region_SDRAM_start__   = 0xC00FF000;  
define symbol __ICFEDIT_region_SDRAM_end__     = 0xC0700FFF;

define region SDRAM_region    = mem:[from __ICFEDIT_region_SDRAM_start__ to __ICFEDIT_region_SDRAM_end__];

place in SDRAM_region    { first section Video_RGB_Buffer };

After linking, EWARM\STM32F746G_DISCO\List\STM32F746G_DISCO.map contains the following placement information for Video_RGB_Buffer:

STM32F746G_DISCO.map
Video_RGB_Buffer        zero     0xc00f'f000   0x3'8400  TouchGFXGeneratedHAL.o [2]
                               - 0xc013'7400   0x3'8400

STM32CubeIDE​

STM32F746NGHX_FLASH.ld
MEMORY
{
  ...  
  SDRAM      (xrw)    : ORIGIN = 0xC00FF000,   LENGTH = 8M
}

BufferSection :
{
  *(Video_RGB_Buffer Video_RGB_Buffer.*)
  *(.gnu.linkonce.r.*)
  . = ALIGN(0x4);
} >SDRAM

After compilation, STM32CubeIDE\Debug\STM32F746G_DISCO.map contains the following placement information for Video_RGB_Buffer:

STM32F746G_DISCO.map
BufferSection   0x00000000c00ff000    0x1c200
 *(Video_RGB_Buffer Video_RGB_Buffer.*)
 Video_RGB_Buffer
                0x00000000c00ff000    0x1c200 Application/User/TouchGFX/target/generated/TouchGFXGeneratedHAL.o
                0x00000000c00ff000                videoRGBBuffer

MDK-ARM (Keil)​

STM32F746G_DISCO.sct
LR_SDRAM 0xC00FF000 0xC0700FFF  {    ; load region size_region
  RW_SDRAM 0xC00FF000 0xC0700FFF {
   *.o (Video_RGB_Buffer)
  }
}

After linking, MDK-ARM\STM32F746G_DISCO\STM32F746G_DISCO.map contains the following placement information for Video_RGB_Buffer:

STM32F746G_DISCO.map
Video_RGB_Buffer                         0xc00ff000   Section    115200  touchgfxgeneratedhal.o(Video_RGB_Buffer)