64*64*3小圖（12KB），batchSize=128，訓(xùn)練樣本100萬(wàn)，

全部load進(jìn)來(lái)內(nèi)存受不了，load一次需要大半天

訓(xùn)練時(shí)讀入一個(gè)batch，ali云服務(wù)器上每個(gè)batch讀入時(shí)間1.9~3.2s不等，迭代一次2s多

由于有多個(gè)label不能用caffe自帶的lmdb轉(zhuǎn)了，輸入是自己寫的python層，試著用pickel

import os, sys import cv2 import numpy as np import numpy.random as npr import cPickle as pickle wk_dir = "/Users/xxx/wkspace/caffe_space/detection/caffe/data/1103reg64/" InputSize = int(sys.argv[1]) BatchSize = int(sys.argv[2]) trainfile = "train.txt" testfile = "test.txt" print "gen imdb with for net input:", InputSize, "batchSize:", BatchSizewith open(wk_dir+trainfile, 'r') as f:trainlines = f.readlines() with open(wk_dir+testfile, 'r') as f:testlines = f.readlines() ####################################### # we seperate train data by batchsize # ####################################### to_dir = wk_dir + "/trainIMDB/" if not os.path.isdir(to_dir):os.makedirs(to_dir)train_list = [] cur_ = 0 sum_ = len(trainlines) for line in trainlines:cur_ += 1words = line.split()image_file_name = words[0]im = cv2.imread(wk_dir + image_file_name)h,w,ch = im.shapeif h!=InputSize or w!=InputSize:im = cv2.resize(im,(InputSize,InputSize))roi = [float(words[2]),float(words[3]),float(words[4]),float(words[5])]train_list.append([im, roi])if (cur_ % BatchSize == 0):print "write batch:" , cur_/BatchSizefid = open(to_dir +'train'+ str(BatchSize) + '_'+str(cur_/BatchSize),'w')pickle.dump(train_list, fid)fid.close()train_list[:] = []print len(train_list), "train data generated\n"########################### # tests # ########################### to_dir = wk_dir + "/testIMDB/" if not os.path.isdir(to_dir):os.makedirs(to_dir) test_list = [] cur_ = 0 sum_ = len(testlines) for line in testlines:cur_ += 1words = line.split()image_file_name = words[0]im = cv2.imread(wk_dir + image_file_name)h,w,ch = im.shapeif h!=InputSize or w!=InputSize:im = cv2.resize(im,(InputSize,InputSize))roi = [float(words[2]),float(words[3]),float(words[4]),float(words[5])]test_list.append([im, roi])if (cur_ % BatchSize == 0):print "write batch:", cur_ / BatchSizefid = open(to_dir +'test'+ str(BatchSize) + '_'+str(cur_/BatchSize), 'w')pickle.dump(test_list, fid)fid.close()test_list[:] = [] print len(test_list), "test data generated\n"

每個(gè)batch生成4.8MB的塊（約比128張?jiān)瓐D占3倍磁盤空間）：

訓(xùn)練時(shí)讀入，ali云訓(xùn)練每個(gè)batch時(shí)間變?yōu)?.2s，可加速10倍

?

mac上是ssd硬盤，本來(lái)讀圖就很快，一個(gè)batch 0.05s, 改成pickel后反而變慢了，load一個(gè)batch需要0.2s。

?

轉(zhuǎn)載于:https://www.cnblogs.com/zhengmeisong/p/9903539.html

總結(jié)

以上是生活随笔為你收集整理的pickel加速caffe读图的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。