Toggle navigation
首页
问答
文章
积分商城
专家
专区
更多专区...
文档中心
返回主站
搜索
提问
会员
中心
登录
注册
MicroPython
MQTT
RA8-M85-vision-board
[Vision Board创客营]AIOT demo
发布于 2024-06-22 17:29:37 浏览:771
订阅该版
[tocm] #项目简介: 本文通过edge impulse来训练猫咪的图像,将模型部署到vision board上,将目标检测的结果上传到mqtt服务器上,最终在客户端查看目标检测的结果;从而实现一个AIOT的demo; **本文最终的实现的代码地址为:** [ https://github.com/8-rtt-study/sdk-bsp-ra8d1-vision-board.git]( https://github.com/8-rtt-study/sdk-bsp-ra8d1-vision-board.git) **演示视频地址为:** https://www.bilibili.com/video/BV1gagveNEup/?spm_id_from=333.999.0.0&vd_source=6d49f1af36ee59119860204ad2fae52f 本文内容主要分为六个章节: ##第一章:edge impulse模型训练 本章讲述使用edge impulse平台训练的过程;将拍摄到的照片,上传到edge impulse,选择模型和参数后进行训练,最终输出部署的文件备用; ##第二章:mqtt软件包 本章讲述如何添加mqtt软件包到openmv工程中,使得openmv具备在c侧使用mqtt接口的功能; ##第三章:micro python映射 本章讲述如何将c侧的mqtt接口映射到micro python中,使得在python中调用mqtt的功能; ##第四章:联调实验 本章讲述如何联调edge impulse训练的脚本与python端的mqtt接口,将识别的结果上传到服务器端,通过桌面的客户端接收信息; ##第五章:总结和展望 本章简述一些本项目的不足支持,以及后期可以扩展的方向; ##第六章:致谢 本章主要致谢rtt导师rb不厌其烦的指导; ##第七章:总结和展望 本章介绍在整个项目的过程中,参考的一些连接和资料,大家都是不断站在前人的肩膀上继续前行; # 第一章. edge impulse的模型训练 ## 1. 准备照片 提着猫条,诱惑猫咪到处走动,随后用手机抓拍;咔咔咔,这里拍了98张; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/400440ec4a65ece578415c8d750e7648.png.webp) ## 2. 创建与配置项目info 进入官网:[https://edgeimpulse.com/](https://edgeimpulse.com/) 注册后创建一个项目,接着就是对项目的配置了; 这里有一个需要注意的地方,就是设置项目导入标签的方式,有两种方式,一种是导入图片后,自己手动标记一张一张标记;另一种导入图片的时候,直接给本次导入统一添加一个标记; 我认为这两者的区别有两点, **(1)是否需要精确的识别效果;这里我选择的是手动标记(默认就是这个); (2)会影响本章3.2步骤中,选择learning block的种类;** ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/407155eb66fdf1b734ea843de1cd92a1.png.webp) ##3. 将图像导入到impluse中 由于手机拍摄的照片比较大,98张图片,大概是404M,上传了大约半个小时; 可以看到有大量的图片上传失败了,不知道啥原因;这里最终上传成功是58张; 因为上节中选择了手动标记,所以这里的label并没有起到作用; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/0fa77cee2524a06dc16c5f402471d0b9.png.webp) ##4. 对图像进行标记 稍稍漫长的标记过程,其实也挺好玩的; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/91264ab3a878540b81846cb0bc1e4c5f.png.webp) ##5. create impulse 这里需要注意:如果在上面第2步选择不同的标签方式,则下面的add a learning block的可选项将有所不同; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/e9f6478d79fda08f8da40e1fc35e7122.png.webp) ##6. generate feature 这里开始训练生成特征 ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/fdfe455256a8f167af72b7061719c3db.png.webp) 可以在右侧观察到特征情况;(别问我,我也不知道这些点是什么意思,评论区要是有知道的大佬,麻烦科普一下哈) ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/a07be52e06192cccc2f46edf2ce675fe.png.webp) ##7. 开始训练 选择模型,开始训练 ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/f20256059d3a6a4ce98970b74c7e6949.png.webp) ##8. 选择部署库,下载库文件备用 选择openmv lib进行构建; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/e5bd5f8b36815400b00e30837a88a4b2.png.webp) 将构建生成的文件下载到本地,解压后备用; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/c183f6a09ce5a441ab4adac5e27b2023.png) #第二章:mqtt软件包 ##1. 软件配置说明 这里的主要操作的软件就两个; ###(1).MDK 一定要使用rtt 文档中提供的mdk 5.38 这里贴一下连接[百度网盘:https://pan.baidu.com/s/1O38xjFGV2K1RP7tP1WlcvA?pwd=ra8v 提取码:ra8v](百度网盘:https://pan.baidu.com/s/1O38xjFGV2K1RP7tP1WlcvA?pwd=ra8v 提取码:ra8v) ###(2).env的下载和使用 [https://edgeimpulse.com/](https://edgeimpulse.com/)[使用说明](https://www.rt-thread.org/document/site/#/development-tools/env/env "使用说明") 这里最好直接从git上下载; https://github.com/RT-Thread/env-windows[工具下载地址](https://github.com/RT-Thread/env-windows "工具下载地址") ## 2. 使用env导入mqtt包 ###(1).下载vision board的项目code [git地址](https://github.com/RT-Thread-Studio/sdk-bsp-ra8d1-vision-board.git "git地址") ### (2). 点击bat脚本 这一步经常忘记操作,导致后面配置和编译都会报错; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/64fe8fde4fbea0ac8f238f9696413442.png) ### (3)打开env的menuconfig 进入项目的目录,右键进入env之后,输入menuconfig ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/76ec237b00c56b1912b798f3b01036dd.png) 此处按照路径找到
这个软件包,使用空格键选中; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/52dc1f5f96a7747212d5ff1217d1b377.png.webp) 进入
的配置项,选择mqtt sample,因为测试基于这个sample修改而来; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/805a035fe05c49420d939b7048da603d.png.webp) 最后保存后,输入: scons --target=mdk5 生成mdk工程 ### 4. code修改 这里需要输入通信猫的测试服务器地址(因为域名无法解析) ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/f418f3a1780836750bb3083109f08394.png) ### 5. mdk编译下载和调试 可以看到,进入msh客户端之后,输入help,出现mqtt相关调试命令; 实际上,我们主要使用mqtt_start和mqtt_publish这两个命令; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/44a3f7303dbabf8e409eca3e626e215d.png) #第三章:micro python映射 ## 1、micro python软件包使用说明 上个章节中,我们添加了mqtt软件包,但是mqtt的功能只能在c侧使用,也就是无法在micro python的命令行中调用,本章就是介绍如何将mqtt功能映射到micro python上,直接使用python的脚本调用mqtt的功能; 这里先贴一下micro python软件包的参考资料: [https://edgeimpulse.com/](https://edgeimpulse.com/)[micro python使用手册](https://www.rt-thread.org/document/site/#/rt-thread-version/rt-thread-standard/packages-manual/micropython-docs/micropython-librarys "micro python使用手册") ## 2.code修改 这里主要使用micro python的自动代码生成网页; [https://summergift.github.io/RT-MicroPython-Generator/](https://summergift.github.io/RT-MicroPython-Generator/) ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/f8b8f84687f5a1228384fb50685b1cdd.png) ## 3. 引入mqtt接口 还记得上一个章的结尾的两个mqtt接口吗,就是mqtt_start和mqtt_publish; 这里就是要将这个两个接口映射到micro python生成的code中; 下面给出最终的moduserfunc.c的源码; ```c /* * This file is part of the MicroPython project, http://micropython.org/ * * The MIT License (MIT) * * Copyright (c) 2019 SummerGift
* * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal * in the Software without restriction, including without limitation the rights * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell * copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN * THE SOFTWARE. */ #include "py/runtime.h" #include "mqtt_sample.h" STATIC mp_obj_t add( mp_obj_t arg_1_obj, mp_obj_t arg_2_obj) { mp_int_t arg_1 = mp_obj_get_int(arg_1_obj); mp_int_t arg_2 = mp_obj_get_int(arg_2_obj); mp_int_t ret_val; /* Your code start! */ ret_val = arg_1 + arg_2; /* Your code end! */ return mp_obj_new_int(ret_val); } MP_DEFINE_CONST_FUN_OBJ_2(add_obj, add); STATIC mp_obj_t mqtt_start() { mp_int_t ret_val; /* Your code start! */ ret_val = mqtt_start_py(); if(ret_val == 0) { rt_kprintf("mqtt_start ok\n"); } else { rt_kprintf("mqtt_start fail\n"); } /* Your code end! */ return mp_obj_new_int(ret_val); } MP_DEFINE_CONST_FUN_OBJ_0(mqtt_start_obj, mqtt_start); STATIC mp_obj_t mqtt_publish( mp_obj_t arg_1_obj) { const char* arg_1 = mp_obj_str_get_str(arg_1_obj); bool ret_val; /* Your code start! */ rt_kprintf("get str: %s\n",arg_1); if(mqtt_publish_py(arg_1)) { rt_kprintf("mqtt_send_fail\n"); ret_val = false; } else { rt_kprintf("mqtt_send_ok\n"); ret_val = true; } /* Your code end! */ return mp_obj_new_bool(ret_val); } MP_DEFINE_CONST_FUN_OBJ_1(mqtt_publish_obj, mqtt_publish); STATIC const mp_rom_map_elem_t mp_module_userfunc_globals_table[] = { { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_userfunc) }, { MP_ROM_QSTR(MP_QSTR_add), MP_ROM_PTR(&add_obj) }, { MP_ROM_QSTR(MP_QSTR_mqtt_start), MP_ROM_PTR(&mqtt_start_obj) }, { MP_ROM_QSTR(MP_QSTR_mqtt_publish), MP_ROM_PTR(&mqtt_publish_obj) }, }; STATIC MP_DEFINE_CONST_DICT(mp_module_userfunc_globals, mp_module_userfunc_globals_table); const mp_obj_module_t mp_module_userfunc = { .base = { &mp_type_module }, .globals = (mp_obj_dict_t*)&mp_module_userfunc_globals, }; ``` ## 4. mqtt_sample接口引出 上一节中,我在micro python的接口上引入了mqtt的两个函数,那这个函数实际上从mqtt_sample上引出的,所以这里要把函数和头文件添加进去; 实际上,下面的代码就是复制了原有的mqtt_publish和mqtt_start,然后改了一个名字; 当然,要把这两个函数做一个头文件,然后把头文件放入moduserfunc.c中; ```c int mqtt_publish_py(const char *msg_str) { if (is_started == 0) { LOG_E("mqtt client is not connected."); return -1; } paho_mqtt_publish(&client, QOS1, MQTT_PUBTOPIC, msg_str); return 0; } int mqtt_start_py(void) { /* init condata param by using MQTTPacket_connectData_initializer */ MQTTPacket_connectData condata = MQTTPacket_connectData_initializer; static char cid[20] = { 0 }; if (is_started) { LOG_E("mqtt client is already connected."); return -2; } /* config MQTT context param */ { client.isconnected = 0; client.uri = MQTT_URI; /* generate the random client ID */ rt_snprintf(cid, sizeof(cid), "rtthread%d", rt_tick_get()); /* config connect param */ rt_memcpy(&client.condata, &condata, sizeof(condata)); client.condata.clientID.cstring = cid; client.condata.keepAliveInterval = 30; client.condata.cleansession = 1; /* config MQTT will param. */ client.condata.willFlag = 1; client.condata.will.qos = 1; client.condata.will.retained = 0; client.condata.will.topicName.cstring = MQTT_PUBTOPIC; client.condata.will.message.cstring = MQTT_WILLMSG; /* rt_malloc buffer. */ client.buf_size = client.readbuf_size = 1024; client.buf = rt_calloc(1, client.buf_size); client.readbuf = rt_calloc(1, client.readbuf_size); if (!(client.buf && client.readbuf)) { LOG_E("no memory for MQTT client buffer!"); return -1; } /* set event callback function */ client.connect_callback = mqtt_connect_callback; client.online_callback = mqtt_online_callback; client.offline_callback = mqtt_offline_callback; /* set subscribe table and event callback */ client.messageHandlers[0].topicFilter = rt_strdup(MQTT_SUBTOPIC); client.messageHandlers[0].callback = mqtt_sub_callback; client.messageHandlers[0].qos = QOS1; /* set default subscribe event callback */ client.defaultMessageHandler = mqtt_sub_default_callback; } /* run mqtt client */ paho_mqtt_start(&client); is_started = 1; return 0; } ``` ## 5. micro python接口验证实验 下面写了一个验证的脚本,主要就是使用micro python去调用接口连接wifi,然后连接mqtt服务器,使用mqtt发送数据; ```c import network import time import userfunc SSID = "rtthread" # Network SSID KEY = "12345678" # Network key # Init wlan module and connect to network wlan = network.WLAN(network.STA_IF) wlan.active(True) wlan.connect(SSID, KEY) while not wlan.isconnected(): print('Trying to connect to "{:s}"...'.format(SSID)) time.sleep_ms(1000) # We should have a valid IP now via DHCP print("WiFi Connected ", wlan.ifconfig()) userfunc.mqtt_start() count = 0 while True: time.sleep_ms(1000) count = count +1 send_data = str(count) ret = userfunc.mqtt_publish(send_data) print(send_data) ``` 可以看到,数据发送成功! ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/b72a0fc40e99cacecafaae239e4b98e1.png.webp) #第四章:联调实验 ##1.本章说明 经过前面几章的努力,现获取了micro python调用模型识别的接口和mqtt的接口;所有本章就是将两者的功能合二为一,也就是AI + IOT; ##2.复制部署文件 找到第一个章中生成的文件,主要将这个两个文件移到sd卡中,并插到vision board的卡槽中; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/2392893b05cd620a4e4a99a8a6f7cbaa.png) ##3. 最终脚本 ```python # Edge Impulse - OpenMV Object Detection Example import sensor, image, time, os, tf, math, uos, gc import network import time import userfunc SSID = "rtthread" # Network SSID KEY = "12345678" # Network key # Init wlan module and connect to network wlan = network.WLAN(network.STA_IF) wlan.active(True) wlan.connect(SSID, KEY) while not wlan.isconnected(): print('Trying to connect to "{:s}"...'.format(SSID)) time.sleep_ms(1000) # We should have a valid IP now via DHCP print("WiFi Connected ", wlan.ifconfig()) userfunc.mqtt_start() sensor.reset() # Reset and initialize the sensor. sensor.set_pixformat(sensor.RGB565) # Set pixel format to RGB565 (or GRAYSCALE) sensor.set_framesize(sensor.QVGA) # Set frame size to QVGA (320x240) sensor.set_windowing((240, 240)) # Set 240x240 window. sensor.skip_frames(time=2000) # Let the camera adjust. net = None labels = None min_confidence = 0.5 try: # load the model, alloc the model file on the heap if we have at least 64K free after loading net = tf.load("trained.tflite", load_to_fb=uos.stat('trained.tflite')[6] > (gc.mem_free() - (64*1024))) except Exception as e: raise Exception('Failed to load "trained.tflite", did you copy the .tflite and labels.txt file onto the mass-storage device? (' + str(e) + ')') try: labels = [line.rstrip('\n') for line in open("labels.txt")] except Exception as e: raise Exception('Failed to load "labels.txt", did you copy the .tflite and labels.txt file onto the mass-storage device? (' + str(e) + ')') colors = [ # Add more colors if you are detecting more than 7 types of classes at once. (255, 0, 0), ( 0, 255, 0), (255, 255, 0), ( 0, 0, 255), (255, 0, 255), ( 0, 255, 255), (255, 255, 255), ] discover_count = 0 send_times = 0 clock = time.clock() while(True): clock.tick() img = sensor.snapshot() # detect() returns all objects found in the image (splitted out per class already) # we skip class index 0, as that is the background, and then draw circles of the center # of our objects for i, detection_list in enumerate(net.detect(img, thresholds=[(math.ceil(min_confidence * 255), 255)])): if (i == 0): continue # background class if (len(detection_list) == 0): continue # no detections for this class? print("********** %s **********" % labels[i]) for d in detection_list: [x, y, w, h] = d.rect() center_x = math.floor(x + (w / 2)) center_y = math.floor(y + (h / 2)) print('x %d\ty %d' % (center_x, center_y)) img.draw_circle((center_x, center_y, 12), color=colors[i], thickness=2) discover_count = discover_count +1 if discover_count > 10: discover_count = 0 send_times = send_times + 1 send_times_str = str(send_times) userfunc.mqtt_publish('discover the cat,this is '+ send_times_str + ' times') print(clock.fps(), "fps", end="\n\n") ``` ##4.成果展示 可以看到,猫咪出现的时候,被vision board识别;然后在mqtt客户端上收到了发现猫咪的信息; ![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/45926d3486be82dcd2b60163d8951c84.png.webp) #第五章:总结和展望 很明显,这个项目还有些许的不足之处,主要有以下几点; ##1.容易误识别 由于家里的猫咪是纯白色的,实际测试的过程中,假如vision board一直在运动,则容易将稍微白色的地方识别成猫咪, ##2.应用场景待发掘 本文最终的效果是,发现猫咪,然后就发送一个msg到mqtt上;但是这个应用其实用其他色块等非神经网络的方式也可以实现;还没有完全体现神经网络的优势;rb提议可以将获取到的猫咪的坐标进行利用,加入运动控制等玩法; #第六章:致谢 感谢Rb导师耐心的指导,基本上有问必答,非常感谢; 感谢rtt此次的创客营活动,这是一个很棒的创意,希望以后可以越办越好; 感谢论坛上大量开源爱好者的无私分享; #第七章:参考资料 最早看到使用edge impulse结合vision board的demo https://club.rt-thread.org/ask/article/69ef73018d63deef.html 鸟巢的识别,本文的第一章基本参考这篇文章; [https://club.rt-thread.org/ask/article/aa1fd8a02231c785.html](https://club.rt-thread.org/ask/article/aa1fd8a02231c785.html) 这篇文章里面学到了,当编译结果太大,如何去删除驱动,让项目瘦身; https://club.rt-thread.org/ask/article/69ef73018d63deef.html
1
条评论
默认排序
按发布时间排序
登录
注册新账号
关于作者
Vangaoven
大道之行,天下为公
文章
3
回答
3
被采纳
1
关注TA
发私信
相关文章
1
请问rt-thread有没有移植micropython呢
2
micropython import 文件名的方式执行脚本问题
3
第一篇:Micropython 的起源和发展
4
第二篇:RT-Thread Micropython 简介
5
第三篇:RT-Thread Micropython 快速入门
6
第四篇:Micropython DIY 项目汇总
7
第五篇:Micropython 教程和资源
8
第六篇: RT-Thread MicroPython 学习经验和学习路线
9
RT-Thread MicroPython 最新开发板固件汇总【已失效】
10
有Mpy专门的板块啦~
推荐文章
1
RT-Thread应用项目汇总
2
玩转RT-Thread系列教程
3
国产MCU移植系列教程汇总,欢迎查看!
4
机器人操作系统 (ROS2) 和 RT-Thread 通信
5
五分钟玩转RT-Thread新社区
6
【技术三千问】之《玩转ART-Pi》,看这篇就够了!干货汇总
7
关于STM32H7开发板上使用SDIO接口驱动SD卡挂载文件系统的问题总结
8
STM32的“GPU”——DMA2D实例详解
9
RT-Thread隐藏的宝藏之completion
10
【ART-PI】RT-Thread 开启RTC 与 Alarm组件
热门标签
RT-Thread Studio
串口
Env
LWIP
SPI
AT
Bootloader
Hardfault
CAN总线
FinSH
ART-Pi
USB
DMA
文件系统
RT-Thread
SCons
RT-Thread Nano
线程
MQTT
STM32
RTC
FAL
rt-smart
ESP8266
I2C_IIC
UART
WIZnet_W5500
ota在线升级
freemodbus
PWM
flash
cubemx
packages_软件包
BSP
潘多拉开发板_Pandora
定时器
ADC
GD32
flashDB
socket
中断
编译报错
Debug
rt_mq_消息队列_msg_queue
SFUD
msh
keil_MDK
ulog
C++_cpp
MicroPython
本月问答贡献
踩姑娘的小蘑菇
4
个答案
1
次被采纳
红枫
4
个答案
1
次被采纳
张世争
4
个答案
1
次被采纳
Ryan_CW
4
个答案
1
次被采纳
xiaorui
1
个答案
1
次被采纳
本月文章贡献
catcatbing
3
篇文章
5
次点赞
qq1078249029
2
篇文章
2
次点赞
xnosky
2
篇文章
1
次点赞
Woshizhapuren
1
篇文章
5
次点赞
YZRD
1
篇文章
2
次点赞
回到
顶部
发布
问题
投诉
建议
回到
底部