使用SAL_AT组件下载文件，运行过程中内存占用太大

发布于 2019-05-29 00:25:48 浏览：3719 订阅该版

现象：使用RTT的SAL_AT组件，利用sim800c, 在阿里物联网平台进行OTA下载过程中，占用内存超大（700K的文件下载，动态内存使用最高超过600K）。下载完毕后内存全部释放。

查找问题：
看了下RTT实现机理是AT组件检查到+RECEIVE标志后，通过URC表调用回调函数，回调函数再执行at_socket.c中的at_recv_notice_cb()回调函数，使用at_recvpkt_put()将接收到的数据写入list链表上，然后使用workqueue唤醒等待接收数据的线程使用at_recvpkt_get()数据读取, 该函数读完后使用at_recvpkt_node_delete（）删除链表节点并且释放内存。

在这个过程中内存申请就在urc_recv_func()函数，没接收到一个“+RECEIVE”b标志就会申请内存，因为没有释放，造成动态内存使用超高。

请教：
只要内存足够大，整个下载过程非常完整，但是过程中动态内存申请巨大。同样的应用程序，使用netdev组件转换成以太网，则没有这么大的内存消耗。

所以请教下这个占用内存超大的原理。以下是运行日志：

```
[172703] I/ali-ota: receive 4096 bytes, total recieve: 0 bytes
[172944] I/ali-ota: receive 4096 bytes, total recieve: 4096 bytes
[174964] I/ali-ota: receive 4096 bytes, total recieve: 8192 bytes
[175205] I/ali-ota: receive 4096 bytes, total recieve: 12288 bytes
free
memheap   pool size  max used size available size
-------- ---------- ------------- --------------
ext-sram 1048576    <font color="#ff0000">63012</font>         985564
heap     62816      62804         12
msh />[177267] I/ali-ota: receive 4096 bytes, total recieve: 16384 bytes
[177508] I/ali-ota: receive 4096 bytes, total recieve: 20480 bytes
[180038] I/ali-ota: receive 4096 bytes, total recieve: 24576 bytes
msh />free
memheap   pool size  max used size available size
-------- ---------- ------------- --------------
ext-sram 1048576    <font color="#ff0000">77200</font>         971376
heap     62816      62804         12
msh />[181498] I/ali-ota: receive 4096 bytes, total recieve: 28672 bytes
[181739] I/ali-ota: receive 4096 bytes, total recieve: 32768 bytes
msh />free
memheap   pool size  max used size available size
-------- ---------- ------------- --------------
ext-sram 1048576    <font color="#ff0000">84220</font>         964356
heap     62816      62804         12
msh />[183979] I/ali-ota: receive 4096 bytes, total recieve: 36864 bytes
[184220] I/ali-ota: receive 4096 bytes, total recieve: 40960 bytes
[186718] I/ali-ota: receive 4096 bytes, total recieve: 45056 bytes
[188782] I/ali-ota: receive 4096 bytes, total recieve: 49152 bytes
[189024] I/ali-ota: receive 4096 bytes, total recieve: 53248 bytes
[191181] I/ali-ota: receive 4096 bytes, total recieve: 57344 bytes
[191422] I/ali-ota: receive 4096 bytes, total recieve: 61440 bytes
[193701] I/ali-ota: receive 4096 bytes, total recieve: 65536 bytes
msh />free
memheap   pool size  max used size available size
-------- ---------- ------------- --------------
ext-sram 1048576    126600        921976
heap     62816      62804         12
```
可以看见，从固件开始下载后，动态内存的使用往上飙升，直至下载完毕后，所有内存释放得干干净净。

24 个回答

Cheney_Chen 2019-05-29

这家伙很懒，什么也没写！

armink 2019-05-29

这家伙很懒，什么也没写！

Spunky 2019-05-29

这家伙很懒，什么也没写！

Spunky 2019-05-29

这家伙很懒，什么也没写！

armink 2019-05-30

这家伙很懒，什么也没写！

Spunky 2019-05-30

这家伙很懒，什么也没写！

armink 2019-05-30

这家伙很懒，什么也没写！

Spunky 2019-05-30

这家伙很懒，什么也没写！

>加大 固件接收的频率及包尺寸试试呢

---

我贴出我的应用代码请帮忙看一下:

```
/*
 * Copyright (c) 2006-2018 RT-Thread Development Team. All rights reserved.
 * License-Identifier: Apache-2.0
 *
 * Licensed under the Apache License, Version 2.0 (the "License"); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>

#include "iot_import.h"
#include "iot_export.h"

#include "exports/iot_export_ota.h"

#include "rtthread.h"
#include "cJSON.h"
#include "fal.h"
#include "easyflash.h"
#include "mqtt-def.h"

#define LOG_TAG              "ali-ota"    
#define LOG_LVL              LOG_LVL_DBG
#include <ulog.h>

#define MQTT_OTA_RECV_BUFF_LEN					(4097)

static void *ota_hd = RT_NULL;
static char ota_recv_buff[MQTT_OTA_RECV_BUFF_LEN];
<div>
</div><div>/* 这个TIMER是控制MQTT周期性publish设备状态 */</div>void mqtt_start_period_timer(void)
{
    rt_timer_t period_timer;
	
    period_timer = (rt_timer_t)rt_object_find("ali_timer", RT_Object_Class_Timer);
    if (period_timer)
    {
        rt_timer_start(period_timer);
    }
}

<span style='display: inline !important; float: none; background-color: transparent; color: rgb(51, 51, 51); font-family: arial,"Microsoft Yahei","Hiragino Sans GB",sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px; word-wrap: break-word;'>/* 这个TIMER是控制MQTT周期性publish设备状态 */</span>
void mqtt_stop_period_timer(void)
{
    rt_timer_t period_timer;
	
    period_timer = (rt_timer_t)rt_object_find("ali_timer", RT_Object_Class_Timer);
    if (period_timer)
    {
        rt_timer_stop(period_timer);
    }
}

/* 该函数在MQTT上线后, 循环调用 */
rt_err_t mqtt_ota_init(void *mqtt_ota_hd, char *product_key, char *device_name)
{
    if ((mqtt_ota_hd == RT_NULL) || (product_key == RT_NULL) || (device_name == RT_NULL))
        return RT_ERROR;

if (ota_hd)
        return RT_EOK;

ota_hd = IOT_OTA_Init(product_key, device_name, mqtt_ota_hd);
    if (RT_NULL == ota_hd)
    {
        LOG_D("initialize OTA failed");
        return RT_ERROR;
    }
    
    char firmware_verison[FIRMWARE_VERSION_MAXLEN];
    rt_memset(firmware_verison, 0, sizeof(firmware_verison));
    if (ef_get_env_blob(MQTT_OTA_FIRMWARE_VERSION, firmware_verison, sizeof(firmware_verison) - 1, RT_NULL) <= 0)
    {
        LOG_D("get device firmware version failed");
        if (ota_hd)
        {
            IOT_OTA_Deinit(ota_hd);
            ota_hd = RT_NULL;
        }
        return RT_ERROR;
     }   
    
     if (0 != IOT_OTA_ReportVersion(ota_hd, firmware_verison)) 
     {
         LOG_D("report OTA version failed");
         if (ota_hd)
         {
             IOT_OTA_Deinit(ota_hd);
             ota_hd = RT_NULL;
         }
         return RT_ERROR;
     }
	
     return RT_EOK;
}

void mqtt_ota_deinit(void)
{
    if (ota_hd)
    {
        IOT_OTA_Deinit(ota_hd);
        ota_hd = RT_NULL;
    }
}

rt_err_t mqtt_ota(void *mqtt_ota_hd, char *product_key, char *device_name)
{
    rt_err_t result = RT_ERROR;
	
    if ((mqtt_ota_hd == RT_NULL) || (product_key == RT_NULL) || (device_name == RT_NULL))
        goto __mqtt_ota_exit;

if (mqtt_ota_init(mqtt_ota_hd, product_key, device_name) != RT_EOK)
        goto __mqtt_ota_exit;  
	
    if (IOT_OTA_IsFetching(ota_hd))
    {
         char fm_ver[FIRMWARE_VERSION_MAXLEN], md5sum[33];		
         IOT_OTA_Ioctl(ota_hd, IOT_OTAG_VERSION, fm_ver, FIRMWARE_VERSION_MAXLEN);
         IOT_OTA_Ioctl(ota_hd, IOT_OTAG_MD5SUM, md5sum, 33);
		
         char firmware_verison[FIRMWARE_VERSION_MAXLEN];
         rt_memset(firmware_verison, 0, sizeof(firmware_verison));
         if (ef_get_env_blob(MQTT_OTA_FIRMWARE_VERSION, firmware_verison, sizeof(firmware_verison) - 1, RT_NULL) > 0)
         {
             if (!rt_strcasecmp(firmware_verison, fm_ver))
             {
                 IOT_OTA_ReportVersion(ota_hd, fm_ver);
                 result = RT_EOK;
                 goto __mqtt_ota_exit;
            }
         }
		
         const struct fal_partition *dl_partition;
         dl_partition = fal_partition_find(MQTT_OTA_DOWNLOAD_PARTITION_NAME);
         if (dl_partition == RT_NULL)
         {
             LOG_I("can not find %s partition", MQTT_OTA_DOWNLOAD_PARTITION_NAME);
             goto __mqtt_ota_exit;
         }

if (fal_partition_erase_all(dl_partition) < 0)
        {
            LOG_I("can not erase %s partition", dl_partition->name);
            goto __mqtt_ota_exit;
        }
<div>
</div><div>        /* 防止at_client接收粘包, 关闭周期性publish设备状态 */</div>        mqtt_stop_period_timer();

int fetch_len;
        rt_uint32_t last_percent = 0, percent = 0;
        rt_uint32_t size_of_download = 0, size_of_file;
        rt_uint32_t content_write_sz;

/* 循环条件:未下载完成 */
        while (!IOT_OTA_IsFetchFinish(ota_hd))
        {
            fetch_len = IOT_OTA_FetchYield(ota_hd, ota_recv_buff, MQTT_OTA_RECV_BUFF_LEN, 1);						
            if (fetch_len > 0)
            {
                content_write_sz = fal_partition_write(dl_partition, size_of_download, (uint8_t *)ota_recv_buff, fetch_len);
                if (content_write_sz !=  fetch_len)
                {
                    LOG_I("Write OTA data to file failed");
					
                    IOT_OTA_ReportProgress(ota_hd, IOT_OTAP_BURN_FAILED, RT_NULL);	
                    mqtt_ota_deinit();

mqtt_start_period_timer();
                    goto __mqtt_ota_exit;
               }
               else
               {
                   LOG_I("receive %d bytes, total recieve: %d bytes", size_of_download, size_of_file);
               }
           }
           else 
           {
                LOG_I("ota fetch failed.");				
                IOT_OTA_ReportProgress(ota_hd, IOT_OTAP_FETCH_FAILED, NULL);

if (fetch_len < 0)
                {                		                
                    mqtt_ota_deinit();
                    mqtt_start_period_timer();
                    goto __mqtt_ota_exit;
                }				
            }

/* get OTA information */
            IOT_OTA_Ioctl(ota_hd, IOT_OTAG_FETCHED_SIZE, &size_of_download, 4);
            IOT_OTA_Ioctl(ota_hd, IOT_OTAG_FILE_SIZE, &size_of_file, 4);

last_percent = percent;
            percent = (size_of_download * 100) / size_of_file;
            if (percent - last_percent > 0) 
            {
                 IOT_OTA_ReportProgress(ota_hd, (IOT_OTA_Progress_t)percent, RT_NULL);
                 IOT_OTA_ReportProgress(ota_hd, (IOT_OTA_Progress_t)percent, "hello");
            }
			
            IOT_MQTT_Yield(mqtt_ota_hd, 100);
        } /* end of while */
		
       IOT_OTA_Ioctl(ota_hd, IOT_OTAG_MD5SUM, md5sum, 33);
       IOT_OTA_Ioctl(ota_hd, IOT_OTAG_VERSION, fm_ver, FIRMWARE_VERSION_MAXLEN);

uint32_t firmware_valid;
       IOT_OTA_Ioctl(ota_hd, IOT_OTAG_CHECK_FIRMWARE, &firmware_valid, 4);
       if ((firmware_valid) && (size_of_download == size_of_file) && (size_of_file > 0))
       {
            LOG_D("The firmware is valid!  Download firmware successfully.");

LOG_D("OTA FW version: %s", fm_ver);
            LOG_D("OTA FW MD5 Sum: %s", md5sum);

ef_set_env_blob(MQTT_OTA_FIRMWARE_VERSION, fm_ver, rt_strlen(fm_ver));
            IOT_OTA_ReportVersion(ota_hd, fm_ver);
            result = RT_EOK;			
       }

mqtt_ota_deinit();
       mqtt_start_period_timer();
   }
	
__mqtt_ota_exit:
<div>	return result;</div><div>}
</div>
```

这上面的代码中,  每次调用IOT_OTA_FetchYield()后, 都会调用一次IOT_MQTT_Yield(mqtt_ota_hd, 100); 这里会不会有问题, 阿里平台OAT流程是:
1. OTA平台通过版本号判断是否下载固件, 这个通道为MQTT通道. 我在每次OAT初始化后发送一次给平台
2. OTA平台下载固件前会通过MQTT通道下发一个升级命令(包含固件的URL地址), 然后设备通过HTTPS主动连接指定的URL.
3. 建立TLS连接后, 平台向设备下发固件数据.

由上面代码所示,我通过IOT_OTA_FetchYield()接收完一帧数据后, 会调用IOT_MQTT_Yield(mqtt_ota_hd, 100),  在给出的官方例程中是这样做的,目的应该在下载过程中更新下MQTT通道,包括reconnect, keepalive, 还有一些订阅和发布的数据处理. 阿里的SDK在这不分是固定将线程挂起100个tick(这个是关键).  实际执行时间是应该会大于100个tick的.  这里就有个问题:
1. 由于MQTT与HTTPS是两个通道(独立socket), HTTPS的下发是不会停的, 就造成数据的堆积. 比如在那100个tick中, OTA下发了两帧的数据, 留在动态内存中. 
2. 每执行1次IOT_OTA_FetchYield()只在动态内存中读取1帧固件数据, 然后释放1帧数据, 留了1帧数据在动态内存中下个循环再读取. 
3. 应用程序将读取的1帧数据写入FLASH, 在执行1次IOT_MQTT_Yield(mqtt_ota_hd, 100), 又会延迟至少100个tick+写入FLASH的时间, 这个过程中OTA平台可能又下发了2帧数据. 这样动态内存中就有3帧数据了;
4. 下一个循环在动态内存中就缓存了4帧, 再一个循环就是5帧数据,  然后恶性循环下去, 知道动态内存分配失败.

这个推断可以解释两个现象:
1. 以太网下载, 动态内存几乎没有太多增长.   除了以太网的延迟低外, TCP的实现是由LWIP自己控制, 也就是可以控制数据流, 所以不会造成数据堆积.
2. AT_SAL下载, 延迟大且不稳定, 而TCP协议栈的实现是GPRS模块实现, 模块收到数据就会往串口上发,所以MCU不能控制数据流, 一旦MCU的recv有延迟就会造成数据堆积.

按照[https://www.rt-thread.org/qa/space-uid-6104.html](armink提的建议是加大接收块size和减少recv的延迟的确是解决问题的方向, 我后续把)IOT_MQTT_Yield(mqtt_ota_hd, 100)去掉, 将每一次接收数据块的长度加大试试.

为了方便请高手帮我分析下,我把OTA在mqtt实现的部分代码贴上来, 我是在MQTT连上服务器后循环调用mqtt_ota()的. 由于代码太多,贴出关键代码

```
        while (IOT_MQTT_CheckStateNormal(mqtt_device_hd))
        {
    /* handle the MQTT packet received from TCP or SSL connection */
    IOT_MQTT_Yield(mqtt_device_hd, 200);

/* OTA周期执行 */
    extern rt_err_t mqtt_ota(void *mqtt_ota_hd, char *product_key, char *device_name);
    mqtt_ota(mqtt_device_hd, product_key, device_name);
			
    HAL_SleepMs(200);
        }
```