跳转到主要内容

视频解码

本节展示如何配置TouchGFX Generator，以生成具有视频解码功能的TouchGFX HAL。

在了解该场景之前，请阅读关于在TouchGFX中使用MJPEG视频的文档和关于视频控件本身的文档。

以下场景详细说明了如何赋能TouchGFX HAL，以通过软件（LibJPEG）或硬件（JPEG）支持视频解码。

在为您的特定项目配置MCU之前，请务必阅读本文中的所有章节。

一般情况下，LibJPEG和JPEG配置可以在STM32CubeMX的中间件和多媒体类别中找到：

STM32CubeMX中的JPEG和LibJPEG设置

Tip
  • Several of the TouchGFX Board Setups (TBSs) in TouchGFX Designer include support for MJPEG video decoding.
  • Use the STM32CubeMX configurations found in these example projects as inspiration to enable MJPEG video decoding on your specific MCU.
  • An example of software decoding can be found in the TBS for STM32H735G-DK, while hardware decoding is used in the TBS for STM32U5G9J-DK2.

支持RTOS​

The Generator User Guide mentions that Single- and Double buffer video decoding strategies require a CMSIS compliant RTOS, such as FreeRTOS. TouchGFX Generator 生成一个必须与视频解码任务关联的入口函数videoTaskFunc()。 STM32CubeMX可以生成此配置，方法是在FreeRTOS中间件配置的任务和队列选项卡中定义任务和入口函数。

视频任务堆栈大小（以CMSIS V2的文字定义）和RTOS堆大小是两个重要因素。

For software decoding, the stack size must be carefully set because LibJPEG uses dynamic memory allocation. This means that the FreeRTOS heap should be large enough to accommodate your general application + 0xA000 (~41kB). For hardware decoding, the stack size can be substantially lower because there is no software stack that dynamically allocates memory.

FreeRTOS任务配置

FreeRTOS堆大小配置

基于上述配置，STM32CubeMX为视频任务生成以下代码：

main.c
/* Definitions for VideoTask */
osThreadId_t VideoTaskHandle;
const osThreadAttr_t VideoTask_attributes = {
  .name = "VideoTask",
  .stack_size = 1000 * 4,
  .priority = (osPriority_t) osPriorityLow,
};
...
void main()
{
    ...
    /* creation of VideoTask */
    VideoTaskHandle = osThreadNew(videoTaskFunc, NULL, &VideoTask_attributes);
    ...

}
FreeRTOSConfig.h
#define configTOTAL_HEAP_SIZE                    ((size_t)75000)

软件解码​

软件解码解决方案要求按照Generator用户指南中指定的，从STM32CubeMX启用LibJPEG中间件。 软件解码设置对于所有具备LibJPEG功能的MCU（如STM32F4、STM32F7、STM32H7）都是相同的。

TouchGFX软件解码器期望LibJPEG解码的数据按照BGR像素排序。 如果该设置保留为RGB，R和B颜色分量将在应用中互换。

此外，如果使用16位或24位视频RGB缓冲区，则每个像素的大小应为3字节，如果使用32位视频RGB缓冲区，则为4字节。

16位或24位视频RGB缓冲区的LibJPEG Configuration（配置）

Caution
  • The RGB_ORDERING setting must be configured as BGR, and the pixel size must be set according to which video RGB buffer format is used.
  • If your application uses Direct to Framebuffer strategy in a ARGB8888 (32-bit) application the pixel size must be set to 4.

视频数据​

TouchGFX links the video data to the ExtFlashSection, which needs to be defined in the linker script. If ExtFlashSection is placed in non-memory-mapped flash, the data must be read from the flash memory using a data reader and handed to the decoder.

For projects with video data in non-memory-mapped memory, please refer to the FileReader section later in this article in this article.

硬件解码​

The Generator User Guide mentions that the JPEG IP must be enabled in STM32CubeMX to enable hardware decoding.

JPEG配置中的RGB_FORMAT设置必须符合TouchGFX Framebuffer（帧缓存）的像素格式，即以下示例中的JPEG_RGB565。

JPEG IP Parameter Settings

When using hardware decoding, a DMA is used to transfer data to (memory-to-peripheral) and from (peripheral-to-memory) the JPEG IP. Which DMA to use and how to configure it differs between MCU families.

Tip
Here are some examples of TouchGFX Board Setups available in TouchGFX Designer that demonstrate how to set up JPEG decoding with different types of DMAs:
  • STM32U5G9J-DK2: JPEG + GPDMA
  • STM32H750B-DK: JPEG + MDMA
  • STM32H7S78-DK: JPEG + HPDMA

STM32H750 example​

Here is an example of how to set up the DMA on STM32H750 for JPEG decoding. For the STM32H750, the JPEG peripheral must be configured to use the MDMA peripheral instead of the regular DMA peripherals. Add an MDMA configuration for both the input and output FIFO threshold signals, as shown in the image below. The configuration is identical for both the input and output buffers.

JPEG DMA settings for STM32H750

FileReader接口​

当在非内存映射的存储器中存储MJPEG视频时，开发人员可以指定touchgfx::VideoDataReader的实现，TouchGFX视频控制器可以使用它将需要解码的数据传输到配置的解码器（软件/硬件）。 下面是此接口的简单示例，它将视频数据从一个缓冲区复制到另一个缓冲区。

VideoView.cpp
class MyReader : public touchgfx::VideoDataReader
{
public:
    MyReader() : position(0) { }
    virtual uint32_t getDataLength() { return video_len; }
    virtual void seek(uint32_t pos) { position = pos; }
    virtual bool readData(void* dst, uint32_t bytes)
    {
        memcpy(dst, &video_data[position], bytes);
        position += bytes;
        return true;
    }
    private:
        uint32_t position;
} myReader;

开发人员可以配置控件以使用数据读取器，而不是将一个Video控件指向映射内存中的视频开始部分：

VideoView.cpp
video.setVideoData(myReader);

迁移TBS，以支持视频解码​

如果您想在支持视频解码之前，将TouchGFX Board Setup（TBS）（带有支持视频解码的MCU）创建的项目从TouchGFX 设计器版本迁移，并且希望能够在TouchGFX-Designer中使用“Run Target（运行目标）”，则需要对GCC Makefile进行一些手动变更。

以下部分将列出所需的变更、以及进行变更的原因。 根据开发人员应用程序使用的内容，必须始终应用一些常规变更以及一些LibJPEG（软件解码）和JPEG（硬件解码）特定变更。 这些更改是对旧TBS中已经存在的GCC Makefile的扩展。

如上述场景所述，除了更新Makefile，您还必须在STM32CubeMX中设置视频解码。

常规变更​

定义您项目中的LIBJPEG路径：

# LibJPEG path
libjpeg_path := $(cubemx_middlewares_path)/Third_Party/LibJPEG

然后必须定义视频资源输入路径：

asset_texts_input  := TouchGFX/assets/texts

asset_videos_input := TouchGFX/assets/videos

视频资源输出路径也必须定义在现有资源输出路径下面：

asset_images_output := $(asset_root_path)/images
asset_fonts_output := $(asset_root_path)/fonts
asset_texts_output := $(asset_root_path)/texts
asset_videos_output := $(asset_root_path)/videos

将视频输出资源添加到组件列表中：

all_components := $(components) \
    $(asset_fonts_output) \
    $(asset_images_output) \
    $(asset_texts_output)
    $(asset_texts_output) \
    $(asset_videos_output)

必须定义视频对象文件。 视频对象文件与已经存在的对象分离：

c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)

video_object_files := $(call find, $(asset_videos_output),*.o)

必须定义视频转换器脚本的路径：

textconvert_script_path := $(touchgfx_path)/framework/tools/textconvert
textconvert_executable := $(call find, $(textconvert_script_path), *.rb)
videoconvert_script_path := $(touchgfx_path)/framework/tools/videoconvert

可以添加一个可选的Echo，以查看所有视频对象文件。 视频对象文件必须添加到链接阶段。 该行将$(video_object_files)和其他对象文件加在一起：

$(binary_output_path)/$(target_executable): $(object_files) $(object_asm_files)
    @echo Video Objects: $(video_object_files)
    @echo Linking $(@)
    @mkdir -p $(@D)
    @mkdir -p $(object_output_path)
    @$(file >$(build_root_path)/objects.tmp) $(foreach F,$(object_files) $(video_object_files),$(file >>$(build_root_path)/objects.tmp,$F))

将视频规则添加到现有资源和.PHONY：

_assets_: BitmapDatabase TextKeysAndLanguages Videos

.PHONY: BitmapDatabase TextKeysAndLanguages Videos

添加视频转换后，接着添加视频规则：

Videos:
    @ruby $(videoconvert_script_path)/videoconvert.rb $(asset_videos_input) $(asset_videos_output)

最后更新清除规则，也删除与视频相关的输出。

_clean_:
    @echo Cleaning: $(board_name)
    @rm -rf $(build_root_path)
    # Do not remove gui_generated
    @rm -rf $(asset_images_output)
    @rm -rf $(asset_fonts_output)
    @rm -rf $(asset_texts_output)
    @rm -rf $(asset_videos_output)
    # Create directory to avoid error if it does not exist
    @mkdir -p $(asset_root_path)
    # Remove assets folder if it is empty (i.e. no gui_generated folder)
    @rmdir --ignore-fail-on-non-empty $(asset_root_path)
    # Clean bootloader project
    @$(MAKE) -r -f ExtMem_Boot/gcc/Makefile -s $(MFLAGS) clean

软件变化​

添加所有LIBJPEG路径到include路径：

include_paths := $(library_includes) \
    $(foreach comp, $(all_components), $(comp)/include) \
    $(foreach comp, $(cubemx_components), $(comp)/Inc) \
    $(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
    $(framework_includes) \
    $(cubemx_middlewares_path) \
    $(touchgfx_middlewares_path) \
    $(touchgfx_generator_components) \
    LIBJPEG/Target \
    $(libjpeg_path)/include \
    LIBJPEG/App

必须定义LIBJPEG源路径：

c_source_files := $(call find, $(source_paths),*.c) $(os_source_files) $(board_c_files)
source_files += $(board_cpp_files)

libjpeg_source_path = Middlewares/Third_Party/LibJPEG/source

接下来，所有LIBJPEG源文件必须添加到board_c_files：

board_c_files := \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
    $(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
    $(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
    $(libjpeg_source_path)/jaricom.c \
    $(libjpeg_source_path)/jcomapi.c \
    $(libjpeg_source_path)/jdapimin.c \
    $(libjpeg_source_path)/jdapistd.c \
    $(libjpeg_source_path)/jdarith.c \
    $(libjpeg_source_path)/jdatasrc.c \
    $(libjpeg_source_path)/jdcoefct.c \
    $(libjpeg_source_path)/jdcolor.c \
    $(libjpeg_source_path)/jddctmgr.c \
    $(libjpeg_source_path)/jdhuff.c \
    $(libjpeg_source_path)/jdinput.c \
    $(libjpeg_source_path)/jdmainct.c \
    $(libjpeg_source_path)/jdmarker.c \
    $(libjpeg_source_path)/jdmaster.c \
    $(libjpeg_source_path)/jdmerge.c \
    $(libjpeg_source_path)/jdpostct.c \
    $(libjpeg_source_path)/jdsample.c \
    $(libjpeg_source_path)/jdtrans.c \
    $(libjpeg_source_path)/jerror.c \
    $(libjpeg_source_path)/jidctflt.c \
    $(libjpeg_source_path)/jidctfst.c \
    $(libjpeg_source_path)/jidctint.c \
    $(libjpeg_source_path)/jmemmgr.c \
    $(libjpeg_source_path)/jmemnobs.c \
    $(libjpeg_source_path)/jquant1.c \
    $(libjpeg_source_path)/jquant2.c \
    $(libjpeg_source_path)/jutils.c \
    LIBJPEG/App/libjpeg.c

然后将LIBJPEG源文件添加到现有的目标文件中，方法与其他中间件源文件相同：

# Start converting paths
object_files := $(object_files:$(touchgfx_path)/%.cpp=$(object_output_path)/touchgfx/%.o)
object_files := $(object_files:%.cpp=$(object_output_path)/%.o)
object_files := $(object_files:$(touchgfx_middlewares_path)/%.c=$(object_output_path)/$(touchgfx_middlewares_path)/%.o)
object_files := $(object_files:$(cubemx_middlewares_path)/%.c=$(object_output_path)/$(cubemx_middlewares_path)/%.o)
object_files := $(object_files:$(libjpeg_source_path)/%.c=$(object_output_path)/$(libjpeg_source_path)/%.o)
object_files := $(object_files:$(Drivers_path)/%.c=$(object_output_path)/Drivers/%.o)
object_files := $(object_files:%.c=$(object_output_path)/%.o)

硬件变化​

添加所有JPEG路径到include 路径：

include_paths := $(library_includes) \
    $(foreach comp, $(all_components), $(comp)/include) \
    $(foreach comp, $(cubemx_components), $(comp)/Inc) \
    $(foreach comp, $(touchgfx_generator_components), $(comp)/generated) \
    $(framework_includes) \
    $(cubemx_middlewares_path) \
    $(touchgfx_middlewares_path) \
    $(touchgfx_generator_components) \
    Utilities/JPEG

接下来，所有JPEG源文件必须添加到board_c_files：

board_c_files := \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_bus.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_qspi.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_sdram.c \
    $(Drivers_path)/BSP/STM32H750B-DK/stm32h750b_discovery_ts.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336.c \
    $(Drivers_path)/BSP/Components/ft5336/ft5336_reg.c \
    $(Drivers_path)/BSP/Components/mt25tl01g/mt25tl01g.c \
    $(Drivers_path)/BSP/Components/mt48lc4m32b2/mt48lc4m32b2.c \
    Utilities/JPEG/jpeg_utils.c

将视频缓冲区置于外部存储器中​

When developers create new projects with STM32CubeMX, the linker scripts associated with the generated projects do not contain default regions used by TouchGFX. Because of this, 'ExtFlashSection' where the video is stored is placed in internal flash by default. It will often be required to place 'ExtFlashSection' in external flash memory to avoid running out of internal flash memory.

Tip
The following TBSs are examples of preconfigured TBSs that are ready to decode videos from external memory:
  1. STM32F746-DISCO
  2. STM32H735G-DK
  3. STM32H750-DK
  4. STM32U5G9J-DK2

If the video strategy selected is not direct to framebuffer, an RGB buffer dedicated to JPEG decoding is generated by TouchGFX Generator when video is enabled. 该定义是用一个位置指示来实现的，它告诉链接器应该将缓冲区放在哪个部分。 如果链接器在链接器脚本中找不到该存储区域，缓冲区将被放在内部存储中。

LOCATION_PRAGMA_NOLOAD("Video_RGB_Buffer")
uint32_t videoRGBBuffer[57600] LOCATION_ATTRIBUTE_NOLOAD("Video_RGB_Buffer");

The following compiler-specific subsections describe an example of the modifications that developers can make to achieve placing the buffer in SDRAM on STM32F746G-DISCO. Video_RGB_Buffer表示用于视频解码的缓冲区。 链接器脚本示例在SDRAM中为TouchGFX帧缓存区保留了一些空间（从0xC0000000开始）。

  1. EWARM
  2. STM32CubeIDE
  3. MDK-ARM
Further reading
Please read the section in the Generator User Guide on configuring a project for Video Decoding.

下面的例子都在STM32F746G-DISCO板（0xC0000000->>0xC00FF000）上的SDRAM开始部分保留了一些空间，允许应用程序通过地址（例如0xC0000000）引用帧缓存区，而不存在链接器覆盖帧缓存区数据的风险。 每个示例都允许链接器将Video_RGB_Buffer放入已定义的SDRAM区域。

Tip
如果你分配Touchgfx帧缓存区，而不是使用直接寻址，那么你的链接器脚本也应该将TouchGFX_Framebuffer放入SDRAM中。

EWARM (IAR)​

stm32f746xx_flash.icf
define symbol __ICFEDIT_region_SDRAM_start__   = 0xC00FF000;
define symbol __ICFEDIT_region_SDRAM_end__     = 0xC0700FFF;

define region SDRAM_region    = mem:[from __ICFEDIT_region_SDRAM_start__ to __ICFEDIT_region_SDRAM_end__];

place in SDRAM_region    { first section Video_RGB_Buffer };

链接之后，EWARM\STM32F746G_DISCO\List\STM32F746G_DISCO.map包含Video_RGB_Buffer的以下放置信息：

STM32F746G_DISCO.map
Video_RGB_Buffer        zero     0xc00f'f000   0x3'8400  TouchGFXGeneratedHAL.o [2]
                               - 0xc013'7400   0x3'8400

STM32CubeIDE​

STM32F746NGHX_FLASH.ld
MEMORY
{
  ...
  SDRAM      (xrw)    : ORIGIN = 0xC00FF000,   LENGTH = 8M
}

BufferSection (NOLOAD) :
{
  *(Video_RGB_Buffer Video_RGB_Buffer.*)
  *(.gnu.linkonce.r.*)
  . = ALIGN(0x4);
} >SDRAM

编译之后，STM32CubeIDE\Debug\STM32F746G_DISCO.map 包含Video_RGB_Buffer的以下放置信息：

STM32F746G_DISCO.map
BufferSection   0x00000000c00ff000    0x1c200
 *(Video_RGB_Buffer Video_RGB_Buffer.*)
 Video_RGB_Buffer
                0x00000000c00ff000    0x1c200 Application/User/TouchGFX/target/generated/TouchGFXGeneratedHAL.o
                0x00000000c00ff000                videoRGBBuffer

MDK-ARM (Keil)​

STM32F746G_DISCO.sct
LR_IROM1 0x08000000 0x00200000  {    ; load region size_region
  ER_IROM1 0x08000000 0x00200000  {  ; load address = execution address
    *.o (RESET, +First)
    *(InRoot$$Sections)
    .ANY (+RO)
  }
  RW_IRAM1 0x20000000 0x00050000  {  ; RW data
    .ANY (+RW +ZI)
  }
  RW_SDRAM 0xC00FF000 UNINIT 0xC0700FFF  {
    *.o (Video_RGB_Buffer)
  }
}

确保在保存视频缓冲区的段中包含UNINIT属性 这确保了将内存区域保留为未初始化的数据。 链接之后，MDK-ARM\STM32F746G_DISCO\STM32F746G_DISCO.map包含Video_RGB_Buffer的以下放置信息：

STM32F746G_DISCO.map
Video_RGB_Buffer                         0xc00ff000   Section    115200  touchgfxgeneratedhal.o(Video_RGB_Buffer)