nihui/ncnn-android-yolov8：ncnn android yolov 8实时检测、分割、姿态

技术解读

以下内容由 GitRead 中文社区编辑团队基于项目 README 与官方资料整理，结合中文开发者的常见使用场景补充技术选型建议。具体以项目官方文档为准。

项目背景与痛点

ncnn-android-yolov8 是一款基于 ncnn 深度计算框架的 Android 优化版, 提供了 YOLOv8 系列模型的实时目标检测、分割、姿态估计和分类等功能，为移动设备提供了高性能的机器视觉解决方案。在当今移动设备日益复杂的功能需求面前，特别是涉及机器视觉的实时应用时，对模型的性能和资源占用提出了极高的要求。ncnn-android-yolov8 的出现，正是为了解决这些痛点，它在保证一定精度的情况下，大幅提升了模型的运行效率，显著降低了延迟。

引入这类项目的原因在于，传统的机器视觉应用在较低端的移动设备上往往会遇到性能瓶颈，导致体验不佳。同时，移动设备在带宽和实时性要求上也对模型提出了更高的挑战。因此，市场需要一个能够在手机、平板等有限资源的设备上实现高性能视觉任务的框架，ncnn-android-yolov8 应运而生。

核心技术揭秘

ncnn-android-yolov8 在核心架构和技术实现上，采用了轻量级模型设计和高效的优化策略。它基于 ncnn 框架，这个框架本身就被优化为在低功耗设备上运行，提高了计算密集型任务的能效比。在 YOLOv8 模型方面，其核心设计思想结合了轻量化网络结构和高分辨率输入的权衡，确保了模型在精度和速度上的良好平衡。

技术架构图

利用mermaid绘制数据流图，可以一目了然地理解ncnn-android-yolov8的架构实现原理。以下展示了一个简化的核心数据流图：

graph TD A[用户图片输入] --> B("预处理") B --> C("特征提取网络") C --> D{"目标检测"} D -->|"检测结果"| E[后处理] B --> F("特征图") F --> G{"分割，分类"} G -->|"结果合并"| E E --> H[输出展示]

从图中可以看到，数据流从用户输入图片开始，依次通过预处理、特征提取网络，再进入到目标检测和分类等多个阶段的处理，最终将各个阶段的结果进行合并，输出到展示层。这一设计极大程度地优化了每一阶段的计算负载，确保了实时性和资源的有效利用。

功能亮点与差异

ncnn-android-yolov8 提供了实时目标检测、分割、姿态估计和分类四大功能，相比同类竞品有着独特的设计优势。最为显著的是，它通过将轻量级模型的特性与海量目标检测任务实际应用相结合，实现了精度和速度之间的最佳平衡。这一平衡不仅体现在对模型速度的大幅提高上，同时也保证了模型在实际应用中的鲁棒性和较为宽泛的适应范围。

此外，ncnn-android-yolov8 还支持多种目标检测任务的组合，其独特的多任务学习机制使得单个模型即可处理多个视觉相关任务，这为开发者提供了极大的灵活性，同时减少了模型训练和部署的时间成本，更适合复杂的实际生产环境。

应用场景与落地建议

在生产环境中使用ncnn-android-yolov8时，需注意几点：

资源管理: 移动设备资源有限，合理分配资源，防止过度消耗 CPU/GPU 等硬件资源。
模型精度: 根据实际应用需求调整模型精度要求，在保持可接受的精度水平下尽量降低计算负担。
兼容性: 测试不同机型的适配性，确保模型在多种设备上稳定运行。

此外，该项目适用于需要实时处理大量视图信息的应用场景，比如智能摄像头监控系统、增强现实应用等。在部署过程中，建议开发者首先进行小规模测试验证，再逐步扩大应用范围，以确保系统的稳定性和可靠性。

综合评价

ncnn-android-yolov8是一个技术成熟度高且具备广泛应用前景的项目。其显著优势包括高能效、高效的实时处理能力以及对多种视觉任务的支持，有效应对了移动设备上机器视觉任务的挑战。与此同时，它也存在一些不足，比如在复杂场景中的实际效果还需进一步优化，并且可能存在硬件兼容性问题。总之，对于寻求高性能视觉任务处理解决方案的开发者而言，不失为一个优质选择。

关于

GitHub 原始描述（英文）

ncnn android yolov8 realtime detection, segmentation, pose estimation, classification and obb

android classification detection ncnn obb pose-estimation realtime segmentation yolo yolov8

233 Stars
47 Forks
5 Watching

描述取自 GitHub 官方 API，保持仓库作者书写的原文。

简要分析

热度分

332

价值分

153

活跃状态

活跃

GitHub

语言占比

C++

CMake

Java

Release

贡献者

nihui

ncnn-android-yolov8

download

The YOLOv8 object detection

This is a sample ncnn android project, it depends on ncnn library and opencv

https://github.com/Tencent/ncnn

https://github.com/nihui/opencv-mobile

https://github.com/nihui/mesa-turnip-android-driver (mesa turnip driver)

https://github.com/nihui/momoten (momoten driver)

android apk file download

https://github.com/nihui/ncnn-android-yolov8/releases/latest

how to build and run

step1

https://github.com/Tencent/ncnn/releases

Download ncnn-YYYYMMDD-android-vulkan.zip or build ncnn for android yourself
Extract ncnn-YYYYMMDD-android-vulkan.zip into app/src/main/jni and change the ncnn_DIR path to yours in app/src/main/jni/CMakeLists.txt

step2

https://github.com/nihui/opencv-mobile

Download opencv-mobile-XYZ-android.zip
Extract opencv-mobile-XYZ-android.zip into app/src/main/jni and change the OpenCV_DIR path to yours in app/src/main/jni/CMakeLists.txt

step3

https://github.com/nihui/mesa-turnip-android-driver

Download mesa-turnip-android-XYZ.zip
Create directory app/src/main/jniLibs/arm64-v8a if not exists
Extract libvulkan_freedreno.so from mesa-turnip-android-XYZ.zip into app/src/main/jniLibs/arm64-v8a

step4

https://github.com/nihui/momoten/releases/latest

Download momoten-YYYYMMDD-android-<target>.zip for arm64-v8a-api21, armeabi-v7a-api14, x86-api14 and x86_64-api21
Extract lib/libmomoten.so into app/src/main/jniLibs/<abi> for each corresponding ABI (create directories if needed)
Select momoten in the CPU/GPU selector; a usable vendor OpenCL driver is required

step5

Open this project with Android Studio, build it and enjoy!

some notes

Android 5.0 (API 21) or newer is required
NDK Camera is preferred, with Camera1 used on older Android versions and legacy devices
All models are manually modified to accept dynamic input shape
Most small models run slower on GPU than on CPU, this is common
FPS may be lower in dark environment because of longer camera exposure time

screenshot

guidelines for converting YOLOv8 models

1. install

pip3 install -U ultralytics pnnx ncnn

2. export yolov8 torchscript

yolo export model=yolov8n.pt format=torchscript
yolo export model=yolov8n-seg.pt format=torchscript
yolo export model=yolov8n-pose.pt format=torchscript
yolo export model=yolov8n-cls.pt format=torchscript
yolo export model=yolov8n-obb.pt format=torchscript

3. convert torchscript with static shape

For classification models, step 1-3 is enough.

pnnx yolov8n.torchscript
pnnx yolov8n-seg.torchscript
pnnx yolov8n-pose.torchscript
pnnx yolov8n-cls.torchscript
pnnx yolov8n-obb.torchscript

4. modify pnnx model script for dynamic shape inference manually

Edit yolov8n_pnnx.py / yolov8n_seg_pnnx.py / yolov8n_pose_pnnx.py / yolov8n_obb_pnnx.py

modify reshape to support dynamic image sizes
permute tensor before concat and adjust concat axis (permutations are faster on smaller tensors)
drop post-process part (we implement the post-process externally to avoid invalid bounding box coordinate calculations below the threshold, which is faster)

model	before	after
det	`v_165 = v_142.view(1, 144, 6400) v_166 = v_153.view(1, 144, 1600) v_167 = v_164.view(1, 144, 400) v_168 = torch.cat((v_165, v_166, v_167), dim=2) # ...`	`v_165 = v_142.view(1, 144, -1).transpose(1, 2) v_166 = v_153.view(1, 144, -1).transpose(1, 2) v_167 = v_164.view(1, 144, -1).transpose(1, 2) v_168 = torch.cat((v_165, v_166, v_167), dim=1) return v_168`
seg	`v_144 = v_143.view(1, 32, 6400) v_150 = v_149.view(1, 32, 1600) v_156 = v_155.view(1, 32, 400) v_157 = torch.cat((v_144, v_150, v_156), dim=2) # ... v_191 = v_168.view(1, 144, 6400) v_192 = v_179.view(1, 144, 1600) v_193 = v_190.view(1, 144, 400) v_194 = torch.cat((v_191, v_192, v_193), dim=2) # ... v_215 = (v_214, v_138, ) return v_215`	`v_144 = v_143.view(1, 32, -1).transpose(1, 2) v_150 = v_149.view(1, 32, -1).transpose(1, 2) v_156 = v_155.view(1, 32, -1).transpose(1, 2) v_157 = torch.cat((v_144, v_150, v_156), dim=1) # ... v_191 = v_168.view(1, 144, -1).transpose(1, 2) v_192 = v_179.view(1, 144, -1).transpose(1, 2) v_193 = v_190.view(1, 144, -1).transpose(1, 2) v_194 = torch.cat((v_191, v_192, v_193), dim=1) return v_194, v_157, v_138`
pose	`v_137 = v_136.view(1, 51, 6400) v_143 = v_142.view(1, 51, 1600) v_149 = v_148.view(1, 51, 400) v_150 = torch.cat((v_137, v_143, v_149), dim=-1) # ... v_184 = v_161.view(1, 65, 6400) v_185 = v_172.view(1, 65, 1600) v_186 = v_183.view(1, 65, 400) v_187 = torch.cat((v_184, v_185, v_186), dim=2) # ...`	`v_137 = v_136.view(1, 51, -1).transpose(1, 2) v_143 = v_142.view(1, 51, -1).transpose(1, 2) v_149 = v_148.view(1, 51, -1).transpose(1, 2) v_150 = torch.cat((v_137, v_143, v_149), dim=1) # ... v_184 = v_161.view(1, 65, -1).transpose(1, 2) v_185 = v_172.view(1, 65, -1).transpose(1, 2) v_186 = v_183.view(1, 65, -1).transpose(1, 2) v_187 = torch.cat((v_184, v_185, v_186), dim=1) return v_187, v_150`
obb	`v_137 = v_136.view(1, 1, 16384) v_143 = v_142.view(1, 1, 4096) v_149 = v_148.view(1, 1, 1024) v_150 = torch.cat((v_137, v_143, v_149), dim=2) # ... v_186 = v_163.view(1, 79, 16384) v_187 = v_174.view(1, 79, 4096) v_188 = v_185.view(1, 79, 1024) v_189 = torch.cat((v_186, v_187, v_188), dim=2) # ...`	`v_137 = v_136.view(1, 1, -1).transpose(1, 2) v_143 = v_142.view(1, 1, -1).transpose(1, 2) v_149 = v_148.view(1, 1, -1).transpose(1, 2) v_150 = torch.cat((v_137, v_143, v_149), dim=1) # ... v_186 = v_163.view(1, 79, -1).transpose(1, 2) v_187 = v_174.view(1, 79, -1).transpose(1, 2) v_188 = v_185.view(1, 79, -1).transpose(1, 2) v_189 = torch.cat((v_186, v_187, v_188), dim=1) return v_189, v_150`

5. re-export yolov8 torchscript

python3 -c 'import yolov8n_pnnx; yolov8n_pnnx.export_torchscript()'
python3 -c 'import yolov8n_seg_pnnx; yolov8n_seg_pnnx.export_torchscript()'
python3 -c 'import yolov8n_pose_pnnx; yolov8n_pose_pnnx.export_torchscript()'
python3 -c 'import yolov8n_obb_pnnx; yolov8n_obb_pnnx.export_torchscript()'

6. convert new torchscript with dynamic shape

Note the shape difference for obb model

pnnx yolov8n_pnnx.py.pt inputshape=[1,3,640,640] inputshape2=[1,3,320,320]
pnnx yolov8n_seg_pnnx.py.pt inputshape=[1,3,640,640] inputshape2=[1,3,320,320]
pnnx yolov8n_pose_pnnx.py.pt inputshape=[1,3,640,640] inputshape2=[1,3,320,320]
pnnx yolov8n_obb_pnnx.py.pt inputshape=[1,3,1024,1024] inputshape2=[1,3,512,512]

7. now you get ncnn model files

mv yolov8n_pnnx.py.ncnn.param yolov8n.ncnn.param
mv yolov8n_pnnx.py.ncnn.bin yolov8n.ncnn.bin
mv yolov8n_seg_pnnx.py.ncnn.param yolov8n_seg.ncnn.param
mv yolov8n_seg_pnnx.py.ncnn.bin yolov8n_seg.ncnn.bin
mv yolov8n_pose_pnnx.py.ncnn.param yolov8n_pose.ncnn.param
mv yolov8n_pose_pnnx.py.ncnn.bin yolov8n_pose.ncnn.bin
mv yolov8n_obb_pnnx.py.ncnn.param yolov8n_obb.ncnn.param
mv yolov8n_obb_pnnx.py.ncnn.bin yolov8n_obb.ncnn.bin

暂无评论

ncnn-android-yolov8

技术解读

项目背景与痛点

核心技术揭秘

技术架构图

功能亮点与差异

应用场景与落地建议

综合评价

相关仓库

关于

简要分析

语言占比

Release

贡献者

ncnn-android-yolov8

android apk file download

how to build and run

step1

step2

step3

step4

step5

some notes

screenshot

guidelines for converting YOLOv8 models

1. install

2. export yolov8 torchscript

3. convert torchscript with static shape

4. modify pnnx model script for dynamic shape inference manually

5. re-export yolov8 torchscript

6. convert new torchscript with dynamic shape

7. now you get ncnn model files

评论