Video Decoding
This section shows how to configure the TouchGFX Generator to generate a TouchGFX HAL that has video decoding capabilities.
The following scenario details how to enable TouchGFX HAL to support video decoding through either software (LibJPEG) or hardware (JPEG). It covers both STM32F7 and STM32H7 because the STM32CubeMX configuration to support Hardware (JPEG) decoding is slightly different from the STM32F7.
Generally, the LibJPEG and JPEG configurations can be found in STM32CubeMX in the Middleware and Multimedia catagories:
Before reading this scenario, please read the documentation on using MJPEG Video in TouchGFX and the documentation on the Video widget itself.
Be sure to read all sections in this article before proceeding.
Tip
Software Decoding
The software decoding solution requires that the LibJPEG middleware is enabled from STM32CubeMX, as specified in TouchGFX Generator User Guide. Software decoding setup is identical for all LibJPEG capable MCUs (e.g. STM32F4, STM32F7, STM32H7).
The TouchGFX Software Decoder expects the data decoded by LibJPEG to have BGR pixel ordering. If this setting is left as RGB, R and B color components will be swapped in your application. Furthermore, the size of each pixel should be 3 bytes (4 bytes means the video is encoded in an XRGB format).
Caution
Once LibJPEG is enabled from STM32CubeMX, Software Decoding can now be enabled from TouchGFX Generator, as described in the TouchGFX Generator User Guide.
RTOS Support
The TouchGFX Generator User Guide mentions that Single- and Double buffer decoding strategies require a CMSIS compliant RTOS, such as FreeRTOS. TouchGFX Generator generates an entry point function videoTaskFunc
that must be associated with a Video decoding task. STM32CubeMX can generate this configuration by defining the task and entry point function in the Tasks and queues tab of the FreeRTOS Middleware configuration.
The video task stack size (defined in words for CMSIS V2) and RTOS heap size are two important factors, the latter because LibJPEG, for software decoding, uses dynamic memory allocation. The FreeRTOS Heap should be large enough for your general application + 0xA000
.
Based on the above configuration, STM32CubeMX generates the following code:
main.c
/* Definitions for VideoTask */
osThreadId_t VideoTaskHandle;
const osThreadAttr_t VideoTask_attributes = {
.name = "VideoTask",
.stack_size = 1000 * 4,
.priority = (osPriority_t) osPriorityLow,
};
void main()
{
...
/* creation of VideoTask */
VideoTaskHandle = osThreadNew(videoTaskFunc, NULL, &VideoTask_attributes);
...
}
FreeRTOSConfig.h
#define configTOTAL_HEAP_SIZE ((size_t)75000)
This concludes the TouchGFX HAL configuration for Software Decoding. After Generating code from STM32CubeMX the application will be able to decode videos using LibJPEG when using the Video widget from the designer.
Video Data
Caution
The below snippet from an EWARM project shows the extra options that are created by TouchGFX Designer and how it places the data in ExtFlashSection
which is defined inside the linker script for all TouchGFX Board Packages. This will not work for non-memory-mapped flash or projects that don't define this section.
ewarm_project.ewp
<option>
<name>IlinkExtraOptions</name>
<state>--image_input $PROJ_DIR$\..\TouchGFX\generated\videos\bin\washerdryer.bin,video_washerdryer_bin_start,ExtFlashSection,4</state>
<state>--keep video_washerdryer_bin_start</state>
</option>
For projects with video data in non-memory mapped memory, please read the section FileReader
in this article.
Hardware Decoding
The TouchGFX Generator User Guide mentions that the JPEG IP must be enabled in STM32CubeMX to enable Hardware decoding. Software- and Hardware decoding share several traits and the differences will be outlined in this section.
RTOS Support
FreeRTOS stack size can be substantially lower than for Software decoding because there is no software stack which dynamically allocates memory. Keep the size close to what it would be to accommodate the other tasks in the project.
STM32F769-DISCO
The JPEG configuration for JPEG capable STM32F7 series, e.g. STM32F769, differs slightly from the STM32H7 line. The RGB_FORMAT
must respect the format of the TouchGFX Framebuffer, JPEG_RGB565
in the example below.
Use DMA to transfer data to (memory-to-peripheral) and from (peripheral-to-memory) the JPEG peripheral through the DMA Settings. Adding a DMA request for IN
and one for OUT
will set up the direction parameters automatically.
This concludes the TouchGFX HAL configuration for Hardware Decoding on the JPEG capable STM32F7s (e.g. STM32F769). After Generating code from STM32CubeMX the application will be able to decode videos using the JPEG peripheral when using the Video widget from the designer.
Caution
STM32H750-DK
The only thing that seperates Hardware decoding (JPEG capable) on e.g. an STM32H750 from an STM32F769 is the way DMA transfers are configured in STM32CubeMX. Not only is the UI different, but the DMA concepts are as well.
For the STM32H750, the JPEG peripheral can only be configured to use the MDMA peripheral rather than the regular DMA peripherals. Add an MDMA configuration for both input- and output FIFO Threshold signals as shown in the images below.
Note
Caution
When generating code for e.g. an STM32H750 through STM32CubeMX, unfortunately, the code to configure the MDMA (defined above) to the MDMA handlers is missing and developers must add the highlighted user code, manually.
Core\Src\stm32h7xx_hal_msp.c
void HAL_JPEG_MspInit(JPEG_HandleTypeDef* hjpeg)
{
if(hjpeg->Instance==JPEG)
{
/* USER CODE BEGIN JPEG_MspInit 0 */
hmdma_jpeg_infifo_th.Init.Request = MDMA_REQUEST_JPEG_INFIFO_TH;
hmdma_jpeg_outfifo_th.Init.Request = MDMA_REQUEST_JPEG_OUTFIFO_TH;
/* USER CODE END JPEG_MspInit 0 */
...
FileReader
interface
When storing MJPEG videos on non-memory-mapped memories developers can specify an implementation of touchgfx::VideoDataReader
that the TouchGFX Video Controllers can use to hand data for decoding to the configured decoder (Software/Hardware). The below is a simple example of such an interface that copies video data from one buffer to another.
VideoView.cpp
class MyReader : public touchgfx::VideoDataReader
{
public:
MyReader() : position(0) { }
virtual uint32_t getDataLength() { return video_len; }
virtual void seek(uint32_t pos) { position = pos; }
virtual bool readData(void* dst, uint32_t bytes)
{
memcpy(dst, &video_data[position], bytes);
position += bytes;
return true;
}
private:
uint32_t position;
} myReader;
Instead of pointing a Video
widget to the start of the video in mapped memory, developers can configure the widget to use the data reader instead:
VideoView.cpp
video.setVideoData(myReader);
Migrating TBS to support video decoding
If you want to migrate a project created from a TBS of one of the boards supporting video decoding before it were available and you want to be able to use 'Run Target' in TouchGFX Desginer with that project, some manual changes to the GCC Makefile are required. A run-down of the required changes and why they are needed will be listed in the following sections. The changes are an extension to the already existing GCC Makefile from the older TBS.
Other than updating the Makefile you will also have to setup video decoding in STM32CubeMX as described in the above scenarios.
Generel changes
Define the LIBJPEG path in your project:
# LibJPEG path
libjpeg_path := $(cubemx_middlewares_path)/Third_Party/LibJPEG
The video assets input path must then be defined:
asset_texts_input := TouchGFX/assets/texts
asset_videos_input := TouchGFX/assets/videos
The video assets output path must also be defined bellow the other assets output paths:
asset_images_output := $(asset_root_path)/images
asset_fonts_output := $(asset_root_path)/fonts
asset_texts_output := $(asset_root_path)/texts
asset_videos_output := $(asset_root_path)/videos
Add the video output assets to the components list:
all_components := $(components) \
$(asset_fonts_output) \
$(asset_images_output) \
$(asset_texts_output)
$(asset_texts_output) \
$(asset_videos_output)
Video object files must be defined. The video object files are seperated from the already existing object:
c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)
video_object_files := $(call find, $(asset_videos_output),*.o)
A path to the video converter script must be defined:
textconvert_script_path := $(touchgfx_path)/framework/tools/textconvert
textconvert_executable := $(call find, $(textconvert_script_path), *.rb)
videoconvert_script_path := $(touchgfx_path)/framework/tools/videoconvert
An optional Echo can be added to see all video objects files. The video object files must be added to the linking stage. This line has the $(video_object_files) added together with the other object files:
$(binary_output_path)/$(target_executable): $(object_files) $(object_asm_files)
@echo Video Objects: $(video_object_files)
@echo Linking $(@)
@mkdir -p $(@D)
@mkdir -p $(object_output_path)
@$(file >$(build_root_path)/objects.tmp) $(foreach F,$(object_files) $(video_object_files),$(file >>$(build_root_path)/objects.tmp,$F))
Add the video rule to the existing assets and .PHONY:
_assets_: BitmapDatabase TextKeysAndLanguages Videos
.PHONY: BitmapDatabase TextKeysAndLanguages Videos
The video rule is added with the video conversion added:
Videos:
@ruby $(videoconvert_script_path)/videoconvert.rb $(asset_videos_input) $(asset_videos_output)
Lastly update the clean rule to also remove video related output.
_clean_:
@echo Cleaning: $(board_name)
@rm -rf $(build_root_path)
# Do not remove gui_generated
@rm -rf $(asset_images_output)
@rm -rf $(asset_fonts_output)
@rm -rf $(asset_texts_output)
@rm -rf $(asset_videos_output)
# Create directory to avoid error if it does not exist
@mkdir -p $(asset_root_path)
# Remove assets folder if it is empty (i.e. no gui_generated folder)
@rmdir --ignore-fail-on-non-empty $(asset_root_path)
# Clean bootloader project
@$(MAKE) -r -f ExtMem_Boot/gcc/Makefile -s $(MFLAGS) clean
Software changes
Add all the LIBJPEG paths to the include paths:
include_paths := $(library_includes) \
$(foreach comp, $(all_components), $(comp)/include) \
$(foreach comp, $(cubemx_components), $(comp)/Inc) \
$(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
$(framework_includes) \
$(cubemx_middlewares_path) \
$(touchgfx_middlewares_path) \
$(touchgfx_generator_components) \
LIBJPEG/Target \
$(libjpeg_path)/include \
LIBJPEG/App
LIBJPEG source paths must be defined:
c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)
libjpeg_source_path = Middlewares/Third_Party/LibJPEG/source
Next all the LIBJPEG source files must be added to board_c_files:
board_c_files := \
$(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
$(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
$(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
$(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
$(Drivers_path)/BSP/Components/ft5336/ft5336.c \
$(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
$(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
$(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
$(libjpeg_source_path)/jaricom.c \
$(libjpeg_source_path)/jcomapi.c \
$(libjpeg_source_path)/jdapimin.c \
$(libjpeg_source_path)/jdapistd.c \
$(libjpeg_source_path)/jdarith.c \
$(libjpeg_source_path)/jdatasrc.c \
$(libjpeg_source_path)/jdcoefct.c \
$(libjpeg_source_path)/jdcolor.c \
$(libjpeg_source_path)/jddctmgr.c \
$(libjpeg_source_path)/jdhuff.c \
$(libjpeg_source_path)/jdinput.c \
$(libjpeg_source_path)/jdmainct.c \
$(libjpeg_source_path)/jdmarker.c \
$(libjpeg_source_path)/jdmaster.c \
$(libjpeg_source_path)/jdmerge.c \
$(libjpeg_source_path)/jdpostct.c \
$(libjpeg_source_path)/jdsample.c \
$(libjpeg_source_path)/jdtrans.c \
$(libjpeg_source_path)/jerror.c \
$(libjpeg_source_path)/jidctflt.c \
$(libjpeg_source_path)/jidctfst.c \
$(libjpeg_source_path)/jidctint.c \
$(libjpeg_source_path)/jmemmgr.c \
$(libjpeg_source_path)/jmemnobs.c \
$(libjpeg_source_path)/jquant1.c \
$(libjpeg_source_path)/jquant2.c \
$(libjpeg_source_path)/jutils.c \
LIBJPEG/App/libjpeg.c
The LIBJPEG source files are then added to the existing object files the same way it is done with the rest of the middleware source files:
# Start converting paths
object_files := $(object_files:$(touchgfx_path)/%.cpp=$(object_output_path)/touchgfx/%.o)
object_files := $(object_files:%.cpp=$(object_output_path)/%.o)
object_files := $(object_files:$(touchgfx_middlewares_path)/%.c=$(object_output_path)/$(touchgfx_middlewares_path)/%.o)
object_files := $(object_files:$(cubemx_middlewares_path)/%.c=$(object_output_path)/$(cubemx_middlewares_path)/%.o)
object_files := $(object_files:$(libjpeg_source_path)/%.c=$(object_output_path)/$(libjpeg_source_path)/%.o)
object_files := $(object_files:$(Drivers_path)/%.c=$(object_output_path)/Drivers/%.o)
object_files := $(object_files:%.c=$(object_output_path)/%.o)
Hardware changes
Add all the JPEG paths to the include paths:
include_paths := $(library_includes) \
$(foreach comp, $(all_components), $(comp)/include) \
$(foreach comp, $(cubemx_components), $(comp)/Inc) \
$(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
$(framework_includes) \
$(cubemx_middlewares_path) \
$(touchgfx_middlewares_path) \
$(touchgfx_generator_components) \
Utilities/JPEG
Next all the JPEG source files must be added to board_c_files:
board_c_files := \
$(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
$(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
$(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
$(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
$(Drivers_path)/BSP/Components/ft5336/ft5336.c \
$(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
$(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
$(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
Utilities/JPEG/jpeg_utils.c
Placing Video Buffer in External Memory
When developers create new projects with CubeMX, the linker scripts associated with the generated projects do not contain any of the possible regions used by TouchGFX. Because of this, the specific buffer used to decode MJPEG videos is placed in internal flash by the linker until developers modify their linker scripts to place the buffer elsewhere. Without such modifications developers will experience large internal memory usage and perhaps inability to fit a large video buffer, like for full screen video decoding.
Tip
- STM32F746-DISCO
- STM32F769-DISCO
- STM32H750-DK
The definition for the RGB buffer dedicated to JPEG decoding is generated by TouchGFX Generator when video decoding is enabled. The definition is instrumented with a location pragma that tells the linker which section the buffer should be placed in. If the linker cannot find this memory region in the linker script the buffer will be placed in internal memory.
LOCATION_PRAGMA_NOLOAD("Video_RGB_Buffer")
uint32_t videoRGBBuffer[57600] LOCATION_ATTRIBUTE_NOLOAD("Video_RGB_Buffer");
The following compiler-specific subsections describe the modifications that developers can make to achieve placing the buffer in SDRAM. Video_RGB_Buffer
represents the buffer used for video decoding. The linker script examples reserve some space in SDRAM (starting at 0xC0000000
) for TouchGFX framebuffer(s).
- EWARM
- STM32CubeIDE
- MDK-ARM
Further reading
The following examples all reserves some space at the start of SDRAM on an STM32F746G-DISCO board (0xC0000000
->0xC00FF000
), allowing the application to reference framebuffers by address (e.g. 0xC0000000
) without the risk of the linker overwriting framebuffer data. Each example will allow the linker to place Video_RGB_Buffer
into the defined SDRAM region.
Tip
EWARM (IAR)
stm32f746xx_flash.icf
define symbol __ICFEDIT_region_SDRAM_start__ = 0xC00FF000;
define symbol __ICFEDIT_region_SDRAM_end__ = 0xC0700FFF;
define region SDRAM_region = mem:[from __ICFEDIT_region_SDRAM_start__ to __ICFEDIT_region_SDRAM_end__];
place in SDRAM_region { first section Video_RGB_Buffer };
After linking, EWARM\STM32F746G_DISCO\List\STM32F746G_DISCO.map
contains the following placement information for Video_RGB_Buffer
:
STM32F746G_DISCO.map
Video_RGB_Buffer zero 0xc00f'f000 0x3'8400 TouchGFXGeneratedHAL.o [2]
- 0xc013'7400 0x3'8400
STM32CubeIDE
STM32F746NGHX_FLASH.ld
MEMORY
{
...
SDRAM (xrw) : ORIGIN = 0xC00FF000, LENGTH = 8M
}
BufferSection (NOLOAD) :
{
*(Video_RGB_Buffer Video_RGB_Buffer.*)
*(.gnu.linkonce.r.*)
. = ALIGN(0x4);
} >SDRAM
After compilation, STM32CubeIDE\Debug\STM32F746G_DISCO.map
contains the following placement information for Video_RGB_Buffer
:
STM32F746G_DISCO.map
BufferSection 0x00000000c00ff000 0x1c200
*(Video_RGB_Buffer Video_RGB_Buffer.*)
Video_RGB_Buffer
0x00000000c00ff000 0x1c200 Application/User/TouchGFX/target/generated/TouchGFXGeneratedHAL.o
0x00000000c00ff000 videoRGBBuffer
MDK-ARM (Keil)
STM32F746G_DISCO.sct
LR_IROM1 0x08000000 0x00200000 { ; load region size_region
ER_IROM1 0x08000000 0x00200000 { ; load address = execution address
*.o (RESET, +First)
*(InRoot$$Sections)
.ANY (+RO)
}
RW_IRAM1 0x20000000 0x00050000 { ; RW data
.ANY (+RW +ZI)
}
RW_SDRAM 0xC00FF000 UNINIT 0xC0700FFF {
*.o (Video_RGB_Buffer)
}
}
Make sure to include the UNINIT
attribute in the section that holds the Video Buffer. This ensures leaving the memory region as uninitialized data. After linking, MDK-ARM\STM32F746G_DISCO\STM32F746G_DISCO.map
contains the following placement information for Video_RGB_Buffer
:
STM32F746G_DISCO.map
Video_RGB_Buffer 0xc00ff000 Section 115200 touchgfxgeneratedhal.o(Video_RGB_Buffer)