當前位置：首頁 > 运维知识 > windows >内容正文

windows

caffe使用过程+digits在windows下的安装和运行

發布時間：2024/9/21 windows 32 豆豆

生活随笔收集整理的這篇文章主要介紹了 caffe使用过程+digits在windows下的安装和运行小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

一。模型基本組成

想要訓練一個caffe模型，需要配置兩個文件，包含兩個部分：網絡模型，參數配置，分別對應*.prototxt , ****_solver.prototxt文件。

Caffe模型文件解析：

預處理圖像的leveldb構建

輸入：一批圖像和label （2和3）?
輸出：leveldb （4）?
指令里包含如下信息：

conver_imageset （構建leveldb的可運行程序）

train/ （此目錄放處理的jpg或者其他格式的圖像)

label.txt (圖像文件名及其label信息)

輸出的leveldb文件夾的名字

CPU/GPU (指定是在cpu上還是在gpu上運行code)

CNN網絡配置文件

Imagenet_solver.prototxt （包含全局參數的配置的文件）

Imagenet.prototxt （包含訓練網絡的配置的文件）

Imagenet_val.prototxt （包含測試網絡的配置文件）

網絡模型：

DATA:一般包括訓練數據和測試數據層兩種類型。一般指輸入層，包含source：數據路徑，批處理數據大小batch_size，scale表示數據表示在[0,1]，0.00390625即 1/255

訓練數據層：

layer {name: "mnist"type: "Data"top: "data"top: "label"include {phase: TRAIN}transform_param {scale: 0.00390625}data_param {source: "examples/mnist/mnist_train_lmdb"batch_size: 64backend: LMDB} }

測試數據層：

layer {name: "mnist"type: "Data"top: "data"top: "label"include {phase: TEST}transform_param {scale: 0.00390625}data_param {source: "examples/mnist/mnist_test_lmdb"batch_size: 100backend: LMDB} }

CONVOLUATION：卷積層，blobs_lr:1 , blobs_lr:2分別表示weight 及bias更新時的學習率，這里權重的學習率為solver.prototxt文件中定義的學習率真，bias的學習率真是權重學習率的2倍，這樣一般會得到很好的收斂速度。

num_output表示濾波的個數，kernelsize表示濾波的大小，stride表示步長，weight_filter表示濾波的類型

layer {name: "conv1"type: "Convolution"bottom: "data"top: "conv1"param {lr_mult: 1 //weight學習率}param {lr_mult: 2 //bias學習率，一般為weight的兩倍}convolution_param {num_output: 20 //濾波器個數kernel_size: 5stride: 1 //步長weight_filler {type: "xavier"}bias_filler {type: "constant"}} }

POOLING: 池化層

layer {name: "pool1"type: "Pooling"bottom: "conv1"top: "pool1"pooling_param {pool: MAXkernel_size: 2 stride: 2} }

INNER_PRODUCT: 其實表示全連接，不要被名字誤導

layer {name: "ip1"type: "InnerProduct"bottom: "pool2"top: "ip1"param {lr_mult: 1}param {lr_mult: 2}inner_product_param {num_output: 500 weight_filler {type: "xavier"}bias_filler {type: "constant"}} }

RELU：激活函數，非線性變化層 max( 0 ,x )，一般與CONVOLUTION層成對出現

layer {name: "relu1"type: "ReLU"bottom: "ip1"top: "ip1" }

SOFTMAX:

layer {name: "loss"type: "SoftmaxWithLoss"bottom: "ip2"bottom: "label"top: "loss" }

參數配置文件：

***_solver.prototxt文件定義一些模型訓練過程中需要到的參數，比較學習率，權重衰減系數，迭代次數，使用GPU還是CPU等等.

# The train/test net protocol buffer definition net: "examples/mnist/lenet_train_test.prototxt"# test_iter specifies how many forward passes the test should carry out. # In the case of MNIST, we have test batch size 100 and 100 test iterations, # covering the full 10,000 testing images. test_iter: 100# Carry out testing every 500 training iterations. test_interval: 500# The base learning rate, momentum and the weight decay of the network. base_lr: 0.01 momentum: 0.9 weight_decay: 0.0005# The learning rate policy lr_policy: "inv" gamma: 0.0001 power: 0.75# Display every 100 iterations display: 100# The maximum number of iterations max_iter: 10000# snapshot intermediate results snapshot: 5000 snapshot_prefix: "examples/mnist/lenet"# solver mode: CPU or GPU solver_mode: GPU device_id: 0 #在cmdcaffe接口下，GPU序號從0開始，如果有一個GPU，則device_id:0

訓練出的模型被存為*.caffemodel，可供以后使用。?
一個完整的網絡應該是：?

步驟

數據準備?
準備三組數據：?
Training Set：用于訓練網絡
Validation Set：用于訓練時測試網絡準確率
Test Set：用于測試網絡訓練完成后的最終正確率
構建lmdb/leveldb文件，caffe支持三種數據格式輸入：images, levelda, lmdb

雖然lmdb的內存消耗是leveldb的1.1倍，但是lmdb的速度比leveldb快10%至15%，更重要的是lmdb允許多種訓練模型同時讀取同一組數據集。?
因此lmdb取代了leveldb成為Caffe默認的數據集生成格式。

定義name.prototxt , name_solver.prototxt文件
訓練模型

在windows下訓練巨麻煩，要在win下使用.sh文件才行。

在windows使用.sh

安裝一波?
cygwin?
在軟件下可以安裝，如果出現package不存在的情況可以重新打開setup執行包下載，一般沒問題碰到什么問題解決什么問題。

用.bat來測試

去官網http://yann.lecun.com/exdb/mnist/下載mnist數據集。下載后解壓到C:\caffe-master\data\mnist

在caffe根目錄下，新建一個create_mnist.bat，里面寫入如下的腳本。此處可能出錯，因為train-images.idx3-ubyte?在解壓的時候可能是train-images-idx3-ubyte要注意修改。

.\Build\x64\Release\convert_mnist_data.exe .\data\mnist\mnist_train_lmdb\train-images.idx3-ubyte .\data\mnist\mnist_train_lmdb\train-labels.idx1-ubyte .\examples\mnist\mnist_train_lmdb?
echo.?
.\Build\x64\Release\convert_mnist_data.exe .\data\mnist\mnist_test_lmdb\t10k-images.idx3-ubyte .\data\mnist\mnist_test_lmdb\t10k-labels.idx1-ubyte .\examples\mnist\mnist_test_lmdb?
pause?
`

然后雙擊該腳本運行，即可在E:\caffe\examples\mnist下面生成相應的lmdb數據文件。

在caffe根目錄下，新建train_mnist.bat，然后輸入如下的腳本，

.\Build\x64\Release\caffe.exe train –solver=.\examples\mnist\lenet_solver.prototxt?
pause

然后雙擊運行，就會開始訓練，訓練完畢后會得到相應的準確率和損失率。?
、

接下來安裝digits：

按照這里裝就好了

https://github.com/NVIDIA/DIGITS/blob/digits-5.0/docs/BuildDigitsWindows.md

最后在digits目錄下執行python -m digits就可以了?
出現了不少bug

bug1

出現找不到pycaffe的情況，?
這種情況一般是因為python沒有導入caffe的包只需要將CAFFE_ROOT\Build\x64\Release\pycaffe\caffe文件夾復制到anaconda的sitepackages中就可以了。

bug2

出現pkg_resources._vendor.packaging.version.InvalidVersion: Invalid version: 'CAFFE_VERSION'?
找到\DIGITS-master\digits\config下的caffe.py?
按照下面的中文部分修改。

from __future__ import absolute_importimport imp import os import platform import re import subprocess import sysfrom . import option_list from digits import device_query from digits.utils import parse_versiondef load_from_envvar(envvar):"""Load information from an installation indicated by an environment variable"""value = os.environ[envvar].strip().strip("\"' ") #此處需要修改路徑，于CAFFE_HOME對應if platform.system() == 'Windows':#executable_dir = os.path.join(value, 'install', 'bin')executable_dir = os.path.join(value)#python_dir = os.path.join(value, 'install', 'python')python_dir = os.path.join(value, 'pycaffe')else:executable_dir = os.path.join(value, 'build', 'tools')python_dir = os.path.join(value, 'python')try:executable = find_executable_in_dir(executable_dir)if executable is None:raise ValueError('Caffe executable not found at "%s"'% executable_dir)if not is_pycaffe_in_dir(python_dir):raise ValueError('Pycaffe not found in "%s"'% python_dir)import_pycaffe(python_dir)version, flavor = get_version_and_flavor(executable)except:print ('"%s" from %s does not point to a valid installation of Caffe.'% (value, envvar))print 'Use the envvar CAFFE_ROOT to indicate a valid installation.'raisereturn executable, version, flavordef load_from_path():"""Load information from an installation on standard paths (PATH and PYTHONPATH)"""try:executable = find_executable_in_dir()if executable is None:raise ValueError('Caffe executable not found in PATH')if not is_pycaffe_in_dir():raise ValueError('Pycaffe not found in PYTHONPATH')import_pycaffe()version, flavor = get_version_and_flavor(executable)except:print 'A valid Caffe installation was not found on your system.'print 'Use the envvar CAFFE_ROOT to indicate a valid installation.'raisereturn executable, version, flavordef find_executable_in_dir(dirname=None):"""Returns the path to the caffe executable at dirnameIf dirname is None, search all directories in sys.pathReturns None if not found"""if platform.system() == 'Windows':exe_name = 'caffe.exe'else:exe_name = 'caffe'if dirname is None:dirnames = [path.strip("\"' ") for path in os.environ['PATH'].split(os.pathsep)]else:dirnames = [dirname]for dirname in dirnames:path = os.path.join(dirname, exe_name)if os.path.isfile(path) and os.access(path, os.X_OK):return pathreturn Nonedef is_pycaffe_in_dir(dirname=None):"""Returns True if you can "import caffe" from dirnameIf dirname is None, search all directories in sys.path"""old_path = sys.pathif dirname is not None:sys.path = [dirname] # temporarily replace sys.pathtry:imp.find_module('caffe')except ImportError:return Falsefinally:sys.path = old_pathreturn Truedef import_pycaffe(dirname=None):"""Imports caffeIf dirname is not None, prepend it to sys.path first"""if dirname is not None:sys.path.insert(0, dirname)# Add to PYTHONPATH so that build/tools/caffe is aware of python layers thereos.environ['PYTHONPATH'] = '%s%s%s' % (dirname, os.pathsep, os.environ.get('PYTHONPATH'))# Suppress GLOG output for python bindingsGLOG_minloglevel = os.environ.pop('GLOG_minloglevel', None)# Show only "ERROR" and "FATAL"os.environ['GLOG_minloglevel'] = '2'# for Windows environment, loading h5py before caffe solves the issue mentioned in# https://github.com/NVIDIA/DIGITS/issues/47#issuecomment-206292824import h5py # noqatry:import caffeexcept ImportError:print 'Did you forget to "make pycaffe"?'raise# Strange issue with protocol buffers and pickle - see issue #32sys.path.insert(0, os.path.join(os.path.dirname(caffe.__file__), 'proto'))# Turn GLOG output back on for subprocess callsif GLOG_minloglevel is None:del os.environ['GLOG_minloglevel']else:os.environ['GLOG_minloglevel'] = GLOG_minlogleveldef get_version_and_flavor(executable):"""Returns (version, flavor)Should be called after import_pycaffe()"""version_string = get_version_from_pycaffe()if version_string is None:version_string = get_version_from_cmdline(executable)if version_string is None:version_string = get_version_from_soname(executable)if version_string is None:raise ValueError('Could not find version information for Caffe build ' +'at "%s". Upgrade your installation' % executable)#這部分代碼沒用，但是會出現bug，我就注釋了#version = parse_version(version_string)#if parse_version(0, 99, 0) > version > parse_version(0, 9, 0):# flavor = 'NVIDIA'# minimum_version = '0.11.0'# if version < parse_version(minimum_version):# raise ValueError(# 'Required version "%s" is greater than "%s". Upgrade your installation.'# % (minimum_version, version_string))#else:# flavor = 'BVLC'flavor = 'BVLC'return version_string, flavordef get_version_from_pycaffe():try:from caffe import __version__ as versionreturn versionexcept ImportError:return Nonedef get_version_from_cmdline(executable):command = [executable, '-version']p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)if p.wait():print p.stderr.read().strip()raise RuntimeError('"%s" returned error code %s' % (command, p.returncode))pattern = 'version'for line in p.stdout:if pattern in line:return line[line.find(pattern) + len(pattern) + 1:].strip()return Nonedef get_version_from_soname(executable):command = ['ldd', executable]p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)if p.wait():print p.stderr.read().strip()raise RuntimeError('"%s" returned error code %s' % (command, p.returncode))# Search output for caffe librarylibname = 'libcaffe'caffe_line = Nonefor line in p.stdout:if libname in line:caffe_line = linebreakif caffe_line is None:raise ValueError('libcaffe not found in linked libraries for "%s"'% executable)# Read the symlink for libcaffe from ldd outputsymlink = caffe_line.split()[2]filename = os.path.basename(os.path.realpath(symlink))# parse the version stringmatch = re.match(r'%s(.*)\.so\.(\S+)$' % (libname), filename)if match:return match.group(2)else:return None #看這里，看這里，一個路徑問題 #我們需要在環境變量里聲明一下，CAFFE_ROOT 或者 CAFFE_HOME都可以，指向caffe編譯后的 ./Build/x64/Release if 'CAFFE_ROOT' in os.environ:executable, version, flavor = load_from_envvar('CAFFE_ROOT') elif 'CAFFE_HOME' in os.environ:executable, version, flavor = load_from_envvar('CAFFE_HOME') else:executable, version, flavor = load_from_path()option_list['caffe'] = {'executable': executable,'version': version,'flavor': flavor,'multi_gpu': (flavor == 'BVLC' or parse_version(version) >= parse_version(0, 12)),'cuda_enabled': (len(device_query.get_devices()) > 0), }

再次運行：?

訓練：?
官方教程

完美~

與50位技術專家面對面20年技術見證，附贈技術全景圖

總結

以上是生活随笔為你收集整理的caffe使用过程+digits在windows下的安装和运行的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇：安装Windows digits问题列表
下一篇： CS231n课程笔记翻译系列之目录汇总