STM32_AI_HandPosture_Demo

Hand Motion Recognition Demo

Description

This project is a demonstration of hand posture, motion recognition using VL53L8 ToF sensors and X-CUBE-AI.
Projects referenced STM32 Model Zoo’s handposture and STSW-IMG035_F401 Gesture sample code.

Development targets are stm32F769i disco board Nucleo F401RF board and X-NUCLEO-53L8A1 board.

Nucleo F401RE connected to X-NUCLEO-53L8A1 board, receives VL53L8 ToF sensor data and recognizes hand posture, motions through CNN model processing using X-CUBE-AI and GesturesMZ library in Gestures sample code.
Processed hand motion values are transmitted to STM32F769I-DISCO board via Uart com, and results are displayed GUI.

There are three menus in GUI.

1. Recognizes posture of hand(Using X-CUBE-AI)
2. Recognizes hand height and heigh(Using GesturesMZ lib)
3. Recognizes position of hand(Using GesturesMZ lib)

GUI is built with Touchgfx, and you can move menu with touch of screen.

–Version–
STM32CUBE F7 v1.17.2
STM32CUBE MX 6.11.1
STM32CUBE IDE 1.15.1
X-CUBE-AI 9.0.0
TouchGFX 4.24

Project Structure

STM32_AI_HandPosture_Demo

├─ NucleoF401
│  ├─ .ai
│  ├─ .cproject
│  ├─ .mxproject
│  ├─ .project -------------------------------------------> Project file
│  ├─ .settings
│  ├─ CNN2D_ST_HandPosture_8classes.h5 -------------------> STM32 Model Zoo CNN Model 
│  ├─ Core -----------------------------------------------> Main App
│  │  ├─ Inc 
│  │  └─ Src
│  ├─ Drivers --------------------------------------------> BSP, HAL Driver
│  │  ├─ BSP
│  │  │  ├─ Components
│  │  │  │  └─ VL53LMZ -----------------------------------> VL53LX ToF BSP
│  │  │  ├─ platform.c
│  │  │  └─ platform.h
│  │  ├─ CMSIS
│  │  └─ STM32F4xx_HAL_Driver
│  ├─ GesturesMZ -----------------------------------------> GestureMZ lib
│  ├─ HandPosture_Demo_F4 Debug.launch
│  ├─ HandPosture_Demo_F4.ioc ----------------------------> CubeMX .ioc file
│  ├─ Middlewares
│  │  └─ ST
│  │     └─ AI -------------------------------------------> X-CUBE-AI middleware
│  ├─ STM32F401RETX_FLASH.ld
│  ├─ STM32F401RETX_RAM.ld
│  ├─ X-CUBE-AI
│  └─ startup
├─ README.md
==================================================================================
├─ STM32F769I_DISCO
├─ .extSettings
├─ .mxproject
├─ Core -------------------------------------------------> Main App
│  ├─ Inc
│  └─ Src
├─ Drivers ----------------------------------------------> BSP, HAL Driver
│  ├─ BSP
│  │  └─ Components
│  │     ├─ Common
│  │     ├─ ft6x06
│  │     ├─ mx25l512
│  │     └─ otm8009a
│  ├─ CMSIS
│  └─ STM32F7xx_HAL_Driver
├─ HandPosture_Demo_F7.ioc ----------------------------> CubeMX .ioc file
├─ LIBJPEG
├─ Middlewares
│  ├─ ST
│  │  └─ touchgfx -------------------------------------> TouchGFX middleware
│  └─ Third_Party
│     ├─ FreeRTOS
│     └─ LibJPEG
├─ STM32CubeIDE
│  ├─ .cproject
│  ├─ .project -----------------------------------------> Project file
│  ├─ .settings
│  ├─ Application -----------------------> Empty dir, Use as a project link
│  ├─ Drivers ---------------------------> Empty dir, Use as a project link
│  ├─ HandPosture_Demo_F7 Debug.launch
│  ├─ Middlewares -----------------------> Empty dir, Use as a project link
│  ├─ STM32F769I_DISCO.launch
│  ├─ STM32F769NIHX_FLASH.ld
│  └─ STM32F769NIHX_RAM.ld
├─ TouchGFX
│  ├─ App
│  ├─ ApplicationTemplate.touchgfx.part
│  ├─ HandPosture_Demo.touchgfx -------------------------> TouchGFX designer file
│  ├─ MATERIAL-ICONS-LICENSE
│  ├─ application.config
│  ├─ assets --------------------------------------------> GUI image and fonts
│  ├─ build
│  ├─ config
│  ├─ generated -----------------------------------------> File generated by TouchGFX designer
│  │  ├─ gui_generated
│  │  ├─ images
│  │  ├─ simulator
│  │  ├─ texts
│  │  └─ videos
│  ├─ gui -----------------------------------------------> GUI App code
│  │  ├─ include
│  │  └─ src
│  ├─ simulator
│  ├─ target.config
│  └─ target
├─ changelog.txt
└─ readme.md

AI Model Information

AI model uses a pre-trained CNN 2D model in STM32 Model Zoo.
(ref. https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/hand_posture)

How to run

1. Run .project file in F401RE and F769I dir to add a project to CubeIDE.
   (Merging may be necessary due to differences in program versions.)
2. Build project. (Target: Nucleo-F401RE, STM32F769i disco board)
3. Connect target board and RUN it.
4. Connect X-NUCLEO-53L8A1 board to Nucleo-F401RE and Connect UART line STM32F769I DISCO board as shown in picture below.

7. Press desired menu in GUI to move screen, and then detect hand gesture on sensor, result is output.

Menu Preview

1. Hand Posture

Recognize a total of 7 hand motions (Flat, Fist, Like, Dislike, Love, Time, Cross).

Video.
https://github.com/user-attachments/assets/0ec3457b-cf05-4496-bd4a-7ae7c6f7803c

2. Light Dim Control

Change Dimming from 0 to 100 depending on hand height of sensor (up to about 30 cm).

Video.
https://github.com/user-attachments/assets/426a58dd-b0b9-40c4-8a46-94bd820900e8

3. Select Target

Select 0-360 degree menu depending on position of hand of sensor.

Video.
https://github.com/user-attachments/assets/3dfbdc69-15ec-4241-8792-f95f8ef60050

Issue

1. Due to fast sensor operation speed, communication loss between boards occurs occasionally.
2. I don’t know why, syscalls.c file disappears when code is regenerated through CubeMX in NucleoF401 project.
3. This project is intended for personal study and may be of low quality. :worried::worried: