RT-Thread-[Vision Board创客营]AIOT demoRT-Thread问答社区

[Vision Board创客营]AIOT demo

发布于 2024-06-22 17:29:37 浏览：774 订阅该版

[tocm]

#项目简介：
  本文通过edge impulse来训练猫咪的图像，将模型部署到vision board上，将目标检测的结果上传到mqtt服务器上，最终在客户端查看目标检测的结果；从而实现一个AIOT的demo；
**本文最终的实现的代码地址为：**
[ https://github.com/8-rtt-study/sdk-bsp-ra8d1-vision-board.git]( https://github.com/8-rtt-study/sdk-bsp-ra8d1-vision-board.git)
**演示视频地址为：**
https://www.bilibili.com/video/BV1gagveNEup/?spm_id_from=333.999.0.0&vd_source=6d49f1af36ee59119860204ad2fae52f
本文内容主要分为六个章节:
##第一章：edge impulse模型训练
  本章讲述使用edge impulse平台训练的过程；将拍摄到的照片，上传到edge impulse，选择模型和参数后进行训练，最终输出部署的文件备用；

##第二章：mqtt软件包
本章讲述如何添加mqtt软件包到openmv工程中，使得openmv具备在c侧使用mqtt接口的功能；

##第三章：micro python映射
  本章讲述如何将c侧的mqtt接口映射到micro python中，使得在python中调用mqtt的功能；

##第四章：联调实验
  本章讲述如何联调edge impulse训练的脚本与python端的mqtt接口，将识别的结果上传到服务器端，通过桌面的客户端接收信息；

##第五章：总结和展望
 本章简述一些本项目的不足支持，以及后期可以扩展的方向；

##第六章：致谢
 本章主要致谢rtt导师rb不厌其烦的指导；

##第七章：总结和展望
 本章介绍在整个项目的过程中，参考的一些连接和资料，大家都是不断站在前人的肩膀上继续前行；

# 第一章. edge impulse的模型训练
## 1. 准备照片
提着猫条，诱惑猫咪到处走动，随后用手机抓拍；咔咔咔，这里拍了98张；
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/400440ec4a65ece578415c8d750e7648.png.webp)
## 2. 创建与配置项目info
进入官网：[https://edgeimpulse.com/](https://edgeimpulse.com/)
注册后创建一个项目，接着就是对项目的配置了；
这里有一个需要注意的地方，就是设置项目导入标签的方式，有两种方式，一种是导入图片后，自己手动标记一张一张标记；另一种导入图片的时候，直接给本次导入统一添加一个标记；
我认为这两者的区别有两点，
**（1）是否需要精确的识别效果；这里我选择的是手动标记（默认就是这个）；
（2）会影响本章3.2步骤中，选择learning block的种类；**

![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/407155eb66fdf1b734ea843de1cd92a1.png.webp)

##3. 将图像导入到impluse中
由于手机拍摄的照片比较大，98张图片，大概是404M,上传了大约半个小时；
可以看到有大量的图片上传失败了，不知道啥原因；这里最终上传成功是58张；
因为上节中选择了手动标记，所以这里的label并没有起到作用；
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/0fa77cee2524a06dc16c5f402471d0b9.png.webp)

##4. 对图像进行标记
稍稍漫长的标记过程，其实也挺好玩的；
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/91264ab3a878540b81846cb0bc1e4c5f.png.webp)

##5. create impulse
这里需要注意：如果在上面第2步选择不同的标签方式，则下面的add a learning block的可选项将有所不同；
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/e9f6478d79fda08f8da40e1fc35e7122.png.webp)

##6. generate feature
这里开始训练生成特征
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/fdfe455256a8f167af72b7061719c3db.png.webp)

可以在右侧观察到特征情况；（别问我，我也不知道这些点是什么意思，评论区要是有知道的大佬，麻烦科普一下哈）
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/a07be52e06192cccc2f46edf2ce675fe.png.webp)

##7. 开始训练
选择模型，开始训练

![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/f20256059d3a6a4ce98970b74c7e6949.png.webp)

##8. 选择部署库，下载库文件备用
选择openmv lib进行构建；
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/e5bd5f8b36815400b00e30837a88a4b2.png.webp)
将构建生成的文件下载到本地，解压后备用；
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240621/c183f6a09ce5a441ab4adac5e27b2023.png)

#第二章：mqtt软件包
##1. 软件配置说明
这里的主要操作的软件就两个；
###(1).MDK
一定要使用rtt 文档中提供的mdk 5.38
这里贴一下连接[百度网盘：https://pan.baidu.com/s/1O38xjFGV2K1RP7tP1WlcvA?pwd=ra8v 提取码：ra8v](百度网盘：https://pan.baidu.com/s/1O38xjFGV2K1RP7tP1WlcvA?pwd=ra8v 提取码：ra8v)
###(2).env的下载和使用
 [https://edgeimpulse.com/](https://edgeimpulse.com/)[使用说明](https://www.rt-thread.org/document/site/#/development-tools/env/env "使用说明")
 这里最好直接从git上下载；
https://github.com/RT-Thread/env-windows[工具下载地址](https://github.com/RT-Thread/env-windows "工具下载地址")
## 2. 使用env导入mqtt包
###（1).下载vision board的项目code
[git地址](https://github.com/RT-Thread-Studio/sdk-bsp-ra8d1-vision-board.git "git地址")
### (2). 点击bat脚本
这一步经常忘记操作，导致后面配置和编译都会报错；

![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/64fe8fde4fbea0ac8f238f9696413442.png)
### （3）打开env的menuconfig
进入项目的目录，右键进入env之后，输入menuconfig
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/76ec237b00c56b1912b798f3b01036dd.png)
此处按照路径找到<paho mqtt>这个软件包，使用空格键选中；
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/52dc1f5f96a7747212d5ff1217d1b377.png.webp)
进入<paho mqtt>的配置项，选择mqtt sample，因为测试基于这个sample修改而来；

![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/805a035fe05c49420d939b7048da603d.png.webp)
最后保存后，输入： scons --target=mdk5
生成mdk工程
### 4. code修改
这里需要输入通信猫的测试服务器地址（因为域名无法解析）
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/f418f3a1780836750bb3083109f08394.png)
### 5. mdk编译下载和调试
可以看到，进入msh客户端之后，输入help,出现mqtt相关调试命令；
实际上，我们主要使用mqtt_start和mqtt_publish这两个命令；
![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/44a3f7303dbabf8e409eca3e626e215d.png)

#第三章：micro python映射
## 1、micro python软件包使用说明
上个章节中，我们添加了mqtt软件包，但是mqtt的功能只能在c侧使用，也就是无法在micro python的命令行中调用，本章就是介绍如何将mqtt功能映射到micro python上，直接使用python的脚本调用mqtt的功能；

这里先贴一下micro python软件包的参考资料：
[https://edgeimpulse.com/](https://edgeimpulse.com/)[micro python使用手册](https://www.rt-thread.org/document/site/#/rt-thread-version/rt-thread-standard/packages-manual/micropython-docs/micropython-librarys "micro python使用手册")

## 2.code修改
这里主要使用micro python的自动代码生成网页；
[https://summergift.github.io/RT-MicroPython-Generator/](https://summergift.github.io/RT-MicroPython-Generator/)

![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/f8b8f84687f5a1228384fb50685b1cdd.png)

## 3. 引入mqtt接口
还记得上一个章的结尾的两个mqtt接口吗，就是mqtt_start和mqtt_publish；
这里就是要将这个两个接口映射到micro python生成的code中；
下面给出最终的moduserfunc.c的源码；
```c
/*
 * This file is part of the MicroPython project, http://micropython.org/
 *
 * The MIT License (MIT)
 *
 * Copyright (c) 2019 SummerGift <summergift2019@gmail.com>
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 */

#include "py/runtime.h"
#include "mqtt_sample.h"

STATIC mp_obj_t add(
        mp_obj_t arg_1_obj,
        mp_obj_t arg_2_obj) {
    mp_int_t arg_1 = mp_obj_get_int(arg_1_obj);
    mp_int_t arg_2 = mp_obj_get_int(arg_2_obj);
    mp_int_t ret_val;

/* Your code start! */

ret_val = arg_1 + arg_2;

/* Your code end! */

return mp_obj_new_int(ret_val);
}
MP_DEFINE_CONST_FUN_OBJ_2(add_obj, add);

STATIC mp_obj_t mqtt_start() {
    mp_int_t ret_val;

/* Your code start! */
    ret_val = mqtt_start_py();
    if(ret_val == 0)
    {
        rt_kprintf("mqtt_start ok\n");
    }
    else 
    {
        rt_kprintf("mqtt_start fail\n");
    }
    /* Your code end! */

return mp_obj_new_int(ret_val);
}
MP_DEFINE_CONST_FUN_OBJ_0(mqtt_start_obj, mqtt_start);
    
STATIC mp_obj_t mqtt_publish(
        mp_obj_t arg_1_obj) {
    const char* arg_1 = mp_obj_str_get_str(arg_1_obj);
    bool ret_val;

/* Your code start! */
    rt_kprintf("get str: %s\n",arg_1);

if(mqtt_publish_py(arg_1))
    {
        rt_kprintf("mqtt_send_fail\n");
        ret_val = false;
    } else {
        rt_kprintf("mqtt_send_ok\n");
        ret_val = true;
    }
    /* Your code end! */

return mp_obj_new_bool(ret_val);
}
MP_DEFINE_CONST_FUN_OBJ_1(mqtt_publish_obj, mqtt_publish);

STATIC const mp_rom_map_elem_t mp_module_userfunc_globals_table[] = {
    { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_userfunc) },
    { MP_ROM_QSTR(MP_QSTR_add), MP_ROM_PTR(&add_obj) },
    { MP_ROM_QSTR(MP_QSTR_mqtt_start), MP_ROM_PTR(&mqtt_start_obj) },
    { MP_ROM_QSTR(MP_QSTR_mqtt_publish), MP_ROM_PTR(&mqtt_publish_obj) },
};

STATIC MP_DEFINE_CONST_DICT(mp_module_userfunc_globals, mp_module_userfunc_globals_table);

const mp_obj_module_t mp_module_userfunc = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t*)&mp_module_userfunc_globals,
};

```

## 4. mqtt_sample接口引出

上一节中，我在micro python的接口上引入了mqtt的两个函数，那这个函数实际上从mqtt_sample上引出的，所以这里要把函数和头文件添加进去；
实际上，下面的代码就是复制了原有的mqtt_publish和mqtt_start，然后改了一个名字；
当然，要把这两个函数做一个头文件，然后把头文件放入moduserfunc.c中；

```c
int mqtt_publish_py(const char *msg_str)
{
    if (is_started == 0)
    {
        LOG_E("mqtt client is not connected.");
        return -1;
    }

paho_mqtt_publish(&client, QOS1, MQTT_PUBTOPIC, msg_str);
    return 0;
}

int mqtt_start_py(void)
{
    /* init condata param by using MQTTPacket_connectData_initializer */
    MQTTPacket_connectData condata = MQTTPacket_connectData_initializer;
    static char cid[20] = { 0 };

if (is_started)
    {
        LOG_E("mqtt client is already connected.");
        return -2;
    }
    /* config MQTT context param */
    {
        client.isconnected = 0;
        client.uri = MQTT_URI;

/* generate the random client ID */
        rt_snprintf(cid, sizeof(cid), "rtthread%d", rt_tick_get());
        /* config connect param */
        rt_memcpy(&client.condata, &condata, sizeof(condata));
        client.condata.clientID.cstring = cid;
        client.condata.keepAliveInterval = 30;
        client.condata.cleansession = 1;

/* config MQTT will param. */
        client.condata.willFlag = 1;
        client.condata.will.qos = 1;
        client.condata.will.retained = 0;
        client.condata.will.topicName.cstring = MQTT_PUBTOPIC;
        client.condata.will.message.cstring = MQTT_WILLMSG;

/* rt_malloc buffer. */
        client.buf_size = client.readbuf_size = 1024;
        client.buf = rt_calloc(1, client.buf_size);
        client.readbuf = rt_calloc(1, client.readbuf_size);
        if (!(client.buf && client.readbuf))
        {
            LOG_E("no memory for MQTT client buffer!");
            return -1;
        }

/* set event callback function */
        client.connect_callback = mqtt_connect_callback;
        client.online_callback = mqtt_online_callback;
        client.offline_callback = mqtt_offline_callback;

/* set subscribe table and event callback */
        client.messageHandlers[0].topicFilter = rt_strdup(MQTT_SUBTOPIC);
        client.messageHandlers[0].callback = mqtt_sub_callback;
        client.messageHandlers[0].qos = QOS1;

/* set default subscribe event callback */
        client.defaultMessageHandler = mqtt_sub_default_callback;
    }

/* run mqtt client */
    paho_mqtt_start(&client);
    is_started = 1;

return 0;
}
```

## 5. micro python接口验证实验
下面写了一个验证的脚本，主要就是使用micro python去调用接口连接wifi，然后连接mqtt服务器，使用mqtt发送数据；

```c

import network
import time
import userfunc

SSID = "rtthread"  # Network SSID
KEY = "12345678"  # Network key

# Init wlan module and connect to network
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(SSID, KEY)

while not wlan.isconnected():
    print('Trying to connect to "{:s}"...'.format(SSID))
    time.sleep_ms(1000)

# We should have a valid IP now via DHCP
print("WiFi Connected ", wlan.ifconfig())

userfunc.mqtt_start()

count = 0
while True:
    time.sleep_ms(1000)
    count = count +1
    send_data = str(count)
    ret = userfunc.mqtt_publish(send_data)
    print(send_data)

```
可以看到，数据发送成功！

![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/b72a0fc40e99cacecafaae239e4b98e1.png.webp)

#第四章：联调实验
##1.本章说明
经过前面几章的努力，现获取了micro python调用模型识别的接口和mqtt的接口；所有本章就是将两者的功能合二为一，也就是AI + IOT；
##2.复制部署文件
找到第一个章中生成的文件，主要将这个两个文件移到sd卡中，并插到vision board的卡槽中；

![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/2392893b05cd620a4e4a99a8a6f7cbaa.png)
##3. 最终脚本

```python
# Edge Impulse - OpenMV Object Detection Example

import sensor, image, time, os, tf, math, uos, gc
import network
import time
import userfunc

SSID = "rtthread"  # Network SSID
KEY = "12345678"  # Network key

# Init wlan module and connect to network
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(SSID, KEY)

while not wlan.isconnected():
    print('Trying to connect to "{:s}"...'.format(SSID))
    time.sleep_ms(1000)

# We should have a valid IP now via DHCP
print("WiFi Connected ", wlan.ifconfig())

userfunc.mqtt_start()

sensor.reset()                         # Reset and initialize the sensor.
sensor.set_pixformat(sensor.RGB565)    # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)      # Set frame size to QVGA (320x240)
sensor.set_windowing((240, 240))       # Set 240x240 window.
sensor.skip_frames(time=2000)          # Let the camera adjust.

net = None
labels = None
min_confidence = 0.5

try:
    # load the model, alloc the model file on the heap if we have at least 64K free after loading
    net = tf.load("trained.tflite", load_to_fb=uos.stat('trained.tflite')[6] > (gc.mem_free() - (64*1024)))
except Exception as e:
    raise Exception('Failed to load "trained.tflite", did you copy the .tflite and labels.txt file onto the mass-storage device? (' + str(e) + ')')

try:
    labels = [line.rstrip('\n') for line in open("labels.txt")]
except Exception as e:
    raise Exception('Failed to load "labels.txt", did you copy the .tflite and labels.txt file onto the mass-storage device? (' + str(e) + ')')

colors = [ # Add more colors if you are detecting more than 7 types of classes at once.
    (255,   0,   0),
    (  0, 255,   0),
    (255, 255,   0),
    (  0,   0, 255),
    (255,   0, 255),
    (  0, 255, 255),
    (255, 255, 255),
]

discover_count = 0
send_times = 0
clock = time.clock()
while(True):
    clock.tick()

img = sensor.snapshot()

# detect() returns all objects found in the image (splitted out per class already)
    # we skip class index 0, as that is the background, and then draw circles of the center
    # of our objects

for i, detection_list in enumerate(net.detect(img, thresholds=[(math.ceil(min_confidence * 255), 255)])):
        if (i == 0): continue # background class
        if (len(detection_list) == 0): continue # no detections for this class?

print("********** %s **********" % labels[i])
        for d in detection_list:
            [x, y, w, h] = d.rect()
            center_x = math.floor(x + (w / 2))
            center_y = math.floor(y + (h / 2))
            print('x %d\ty %d' % (center_x, center_y))
            img.draw_circle((center_x, center_y, 12), color=colors[i], thickness=2)
            discover_count = discover_count +1
            if discover_count > 10:
                discover_count = 0
                send_times = send_times + 1
                send_times_str = str(send_times)
                userfunc.mqtt_publish('discover the cat,this is '+ send_times_str + ' times')

print(clock.fps(), "fps", end="\n\n")

```

##4.成果展示
可以看到，猫咪出现的时候，被vision board识别；然后在mqtt客户端上收到了发现猫咪的信息；

![screenshot_image.png](https://oss-club.rt-thread.org/uploads/20240622/45926d3486be82dcd2b60163d8951c84.png.webp)

#第五章：总结和展望
很明显，这个项目还有些许的不足之处，主要有以下几点；
##1.容易误识别
由于家里的猫咪是纯白色的，实际测试的过程中，假如vision board一直在运动，则容易将稍微白色的地方识别成猫咪，
##2.应用场景待发掘
本文最终的效果是，发现猫咪，然后就发送一个msg到mqtt上；但是这个应用其实用其他色块等非神经网络的方式也可以实现；还没有完全体现神经网络的优势；rb提议可以将获取到的猫咪的坐标进行利用，加入运动控制等玩法；

#第六章：致谢
感谢Rb导师耐心的指导，基本上有问必答，非常感谢；
感谢rtt此次的创客营活动，这是一个很棒的创意，希望以后可以越办越好；
感谢论坛上大量开源爱好者的无私分享；

#第七章：参考资料
最早看到使用edge impulse结合vision board的demo
https://club.rt-thread.org/ask/article/69ef73018d63deef.html

鸟巢的识别，本文的第一章基本参考这篇文章；
[https://club.rt-thread.org/ask/article/aa1fd8a02231c785.html](https://club.rt-thread.org/ask/article/aa1fd8a02231c785.html)

这篇文章里面学到了，当编译结果太大，如何去删除驱动，让项目瘦身；
https://club.rt-thread.org/ask/article/69ef73018d63deef.html