文章目錄

• • 1. 梯度下降
  • 2. mini-Batch 梯度下降
  • 3. 動(dòng)量
  • 4. Adam
  • 5. 不同優(yōu)化算法下的模型
  • • 5.1 Mini-batch梯度下降
    • 5.2 帶動(dòng)量的Mini-batch梯度下降
    • 5.3 帶Adam的Mini-batch梯度下降
    • 5.4 對(duì)比總結(jié)

測(cè)試題：參考博文

筆記：02.改善深層神經(jīng)網(wǎng)絡(luò)：超參數(shù)調(diào)試、正則化以及優(yōu)化 W2.優(yōu)化算法

• 導(dǎo)入一些包

import numpy as np import matplotlib.pyplot as plt import scipy.io import math import sklearn import sklearn.datasetsfrom opt_utils import load_params_and_grads, initialize_parameters, forward_propagation, backward_propagation from opt_utils import compute_cost, predict, predict_dec, plot_decision_boundary, load_dataset from testCases import *%matplotlib inline plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots plt.rcParams['image.interpolation'] = 'nearest' plt.rcParams['image.cmap'] = 'gray'

1. 梯度下降

（Batch）Gradient Descent 梯度下降 對(duì)每一層都進(jìn)行：

$W^{[l]}=W^{[l]}?\alpha ?d{W}^{[l]}$

$b^{[l]}=b^{[l]}?\alpha ?d{b}^{[l]}$

$l$ 是層號(hào)，$\alpha$ 是學(xué)習(xí)率

# GRADED FUNCTION: update_parameters_with_gddef update_parameters_with_gd(parameters, grads, learning_rate):"""Update parameters using one step of gradient descentArguments:parameters -- python dictionary containing your parameters to be updated:parameters['W' + str(l)] = Wlparameters['b' + str(l)] = blgrads -- python dictionary containing your gradients to update each parameters:grads['dW' + str(l)] = dWlgrads['db' + str(l)] = dbllearning_rate -- the learning rate, scalar.Returns:parameters -- python dictionary containing your updated parameters """L = len(parameters) // 2 # number of layers in the neural networks# Update rule for each parameterfor l in range(L):### START CODE HERE ### (approx. 2 lines)parameters["W" + str(l+1)] = parameters['W'+str(l+1)] - learning_rate * grads['dW'+str(l+1)]parameters["b" + str(l+1)] = parameters['b'+str(l+1)] - learning_rate * grads['db'+str(l+1)]### END CODE HERE ###return parameters

Stochastic Gradient Descent 隨機(jī)梯度下降

• 每次只用1個(gè)樣本來(lái)更新梯度，當(dāng)訓(xùn)練集很大的時(shí)候，SGD 很快
• 其尋優(yōu)過(guò)程有震蕩
  

代碼差異：

• (Batch) Gradient Descent:

X = data_input Y = labels parameters = initialize_parameters(layers_dims) for i in range(0, num_iterations):# Forward propagationa, caches = forward_propagation(X, parameters)# Compute cost.cost = compute_cost(a, Y)# Backward propagation.grads = backward_propagation(a, caches, parameters)# Update parameters.parameters = update_parameters(parameters, grads)

• Stochastic Gradient Descent:

X = data_input Y = labels parameters = initialize_parameters(layers_dims) for i in range(0, num_iterations):for j in range(0, m):# Forward propagationa, caches = forward_propagation(X[:,j], parameters)# Compute costcost = compute_cost(a, Y[:,j])# Backward propagationgrads = backward_propagation(a, caches, parameters)# Update parameters.parameters = update_parameters(parameters, grads)

3者的差別在于，一次梯度更新時(shí)，用到的樣本數(shù)量不同

調(diào)好參數(shù)的 mini-batch 梯度下降，通常優(yōu)于梯度下降或隨機(jī)梯度下降（特別是當(dāng)訓(xùn)練集很大時(shí)）

2. mini-Batch 梯度下降

如何從訓(xùn)練集 $(X,Y)$ 建立 mini-batches

步驟1：隨機(jī)打亂數(shù)據(jù)，X 和 Y 是同步進(jìn)行的，保持對(duì)應(yīng)關(guān)系

步驟2：切分?jǐn)?shù)據(jù)集（每個(gè)子集大小為 mini_batch_size，最后一個(gè)可能不夠，沒(méi)關(guān)系）

# GRADED FUNCTION: random_mini_batchesdef random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):"""Creates a list of random minibatches from (X, Y)Arguments:X -- input data, of shape (input size, number of examples)Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)mini_batch_size -- size of the mini-batches, integerReturns:mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)"""np.random.seed(seed) # To make your "random" minibatches the same as oursm = X.shape[1] # number of training examplesmini_batches = []# Step 1: Shuffle (X, Y)permutation = list(np.random.permutation(m))shuffled_X = X[:, permutation]shuffled_Y = Y[:, permutation].reshape((1,m))# Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionningfor k in range(0, num_complete_minibatches):### START CODE HERE ### (approx. 2 lines)mini_batch_X = X[:, k*mini_batch_size : (k+1)*mini_batch_size]mini_batch_Y = Y[:, k*mini_batch_size : (k+1)*mini_batch_size]### END CODE HERE ###mini_batch = (mini_batch_X, mini_batch_Y)mini_batches.append(mini_batch)# Handling the end case (last mini-batch < mini_batch_size)if m % mini_batch_size != 0:### START CODE HERE ### (approx. 2 lines)mini_batch_X = X[:, num_complete_minibatches*mini_batch_size : ]mini_batch_Y = Y[:, num_complete_minibatches*mini_batch_size : ]### END CODE HERE ###mini_batch = (mini_batch_X, mini_batch_Y)mini_batches.append(mini_batch)return mini_batches

3. 動(dòng)量

帶動(dòng)量 的 梯度下降可以降低 mini-batch 梯度下降時(shí)的震蕩

原因：Momentum 考慮過(guò)去的梯度對(duì)當(dāng)前的梯度進(jìn)行平滑，梯度不會(huì)劇烈變化

• 初始化梯度的 初速度為 0

# GRADED FUNCTION: initialize_velocitydef initialize_velocity(parameters):"""Initializes the velocity as a python dictionary with:- keys: "dW1", "db1", ..., "dWL", "dbL" - values: numpy arrays of zeros of the same shape as the corresponding gradients/parameters.Arguments:parameters -- python dictionary containing your parameters.parameters['W' + str(l)] = Wlparameters['b' + str(l)] = blReturns:v -- python dictionary containing the current velocity.v['dW' + str(l)] = velocity of dWlv['db' + str(l)] = velocity of dbl"""L = len(parameters) // 2 # number of layers in the neural networksv = {}# Initialize velocityfor l in range(L):### START CODE HERE ### (approx. 2 lines)v["dW" + str(l+1)] = np.zeros(parameters['W'+str(l+1)].shape)v["db" + str(l+1)] = np.zeros(parameters['b'+str(l+1)].shape)### END CODE HERE ###return v

• 對(duì)每一層，更新動(dòng)量
  $$\{\begin{array}{l}{v}_{d{W}^{[l]}}=\beta v_{d{W}^{[l]}}+(1?\beta )d{W}^{[l]}\\ W^{[l]}=W^{[l]}?\alpha v_{d{W}^{[l]}}\end{array}$$

$$\{\begin{array}{l}{v}_{d{b}^{[l]}}=\beta v_{d{b}^{[l]}}+(1?\beta )d{b}^{[l]}\\ b^{[l]}=b^{[l]}?\alpha v_{d{b}^{[l]}}\end{array}$$

# GRADED FUNCTION: update_parameters_with_momentumdef update_parameters_with_momentum(parameters, grads, v, beta, learning_rate):"""Update parameters using MomentumArguments:parameters -- python dictionary containing your parameters:parameters['W' + str(l)] = Wlparameters['b' + str(l)] = blgrads -- python dictionary containing your gradients for each parameters:grads['dW' + str(l)] = dWlgrads['db' + str(l)] = dblv -- python dictionary containing the current velocity:v['dW' + str(l)] = ...v['db' + str(l)] = ...beta -- the momentum hyperparameter, scalarlearning_rate -- the learning rate, scalarReturns:parameters -- python dictionary containing your updated parameters v -- python dictionary containing your updated velocities"""L = len(parameters) // 2 # number of layers in the neural networks# Momentum update for each parameterfor l in range(L):### START CODE HERE ### (approx. 4 lines)# compute velocitiesv["dW" + str(l+1)] = beta* v["dW" + str(l+1)] + (1-beta)*grads['dW' + str(l+1)]v["db" + str(l+1)] = beta* v["db" + str(l+1)] + (1-beta)*grads['db' + str(l+1)]# update parametersparameters["W" + str(l+1)] = parameters["W" + str(l+1)] - learning_rate*v["dW" + str(l+1)]parameters["b" + str(l+1)] = parameters["b" + str(l+1)] - learning_rate*v["db" + str(l+1)]### END CODE HERE ###return parameters, v

注意：

• 速度 v 初始化為 0，算法需要幾次迭代后才能把 v 加上來(lái)，然后開(kāi)始采用大的步長(zhǎng)
• $\beta =0$ 就是不帶動(dòng)量的標(biāo)準(zhǔn)梯度下降

如何選擇 $\beta$：

• $\beta$ 越大，考慮的過(guò)去的梯度越多，梯度輸出也更光滑，太大也不行，過(guò)度光滑
• 經(jīng)常取值為 0.8 - 0.999 之間，如果不確定，0.9 是個(gè)合理的默認(rèn)值
• 參數(shù)驗(yàn)證選取，看其如何影響損失函數(shù)

4. Adam

參看筆記

對(duì)每一層：
$$?\begin{array}{l}{v}_{d{W}^{[l]}}={\beta }_1{v}_{d{W}^{[l]}}+(1?{\beta }_1)\frac{?\mathcal{J}}{?W^{[l]}}\\ v_{d{W}^{[l]}}^{corrected}=\frac{v_{d{W}^{[l]}}}{1?({\beta }_1{)}^t}\\ s_{d{W}^{[l]}}={\beta }_2{s}_{d{W}^{[l]}}+(1?{\beta }_2)(\frac{?\mathcal{J}}{?W^{[l]}}{)}^2\\ s_{d{W}^{[l]}}^{corrected}=\frac{s_{d{W}^{[l]}}}{1?({\beta }_2{)}^t}\\ W^{[l]}=W^{[l]}?\alpha \frac{v_{d{W}^{[l]}}^{corrected}}{\sqrt{s_{d{W}^{[l]}}^{corrected}}+\epsilon }\end{array}$$

• 初始化為 0

# GRADED FUNCTION: initialize_adamdef initialize_adam(parameters) :"""Initializes v and s as two python dictionaries with:- keys: "dW1", "db1", ..., "dWL", "dbL" - values: numpy arrays of zeros of the same shape as the corresponding gradients/parameters.Arguments:parameters -- python dictionary containing your parameters.parameters["W" + str(l)] = Wlparameters["b" + str(l)] = blReturns: v -- python dictionary that will contain the exponentially weighted average of the gradient.v["dW" + str(l)] = ...v["db" + str(l)] = ...s -- python dictionary that will contain the exponentially weighted average of the squared gradient.s["dW" + str(l)] = ...s["db" + str(l)] = ..."""L = len(parameters) // 2 # number of layers in the neural networksv = {}s = {}# Initialize v, s. Input: "parameters". Outputs: "v, s".for l in range(L):### START CODE HERE ### (approx. 4 lines)v["dW" + str(l+1)] = np.zeros(parameters["W" + str(l+1)].shape)v["db" + str(l+1)] = np.zeros(parameters["b" + str(l+1)].shape)s["dW" + str(l+1)] = np.zeros(parameters["W" + str(l+1)].shape)s["db" + str(l+1)] = np.zeros(parameters["b" + str(l+1)].shape)### END CODE HERE ###return v, s

• 迭代更新

# GRADED FUNCTION: update_parameters_with_adamdef update_parameters_with_adam(parameters, grads, v, s, t, learning_rate = 0.01,beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8):"""Update parameters using AdamArguments:parameters -- python dictionary containing your parameters:parameters['W' + str(l)] = Wlparameters['b' + str(l)] = blgrads -- python dictionary containing your gradients for each parameters:grads['dW' + str(l)] = dWlgrads['db' + str(l)] = dblv -- Adam variable, moving average of the first gradient, python dictionarys -- Adam variable, moving average of the squared gradient, python dictionarylearning_rate -- the learning rate, scalar.beta1 -- Exponential decay hyperparameter for the first moment estimates beta2 -- Exponential decay hyperparameter for the second moment estimates epsilon -- hyperparameter preventing division by zero in Adam updatesReturns:parameters -- python dictionary containing your updated parameters v -- Adam variable, moving average of the first gradient, python dictionarys -- Adam variable, moving average of the squared gradient, python dictionary"""L = len(parameters) // 2 # number of layers in the neural networksv_corrected = {} # Initializing first moment estimate, python dictionarys_corrected = {} # Initializing second moment estimate, python dictionary# Perform Adam update on all parametersfor l in range(L):# Moving average of the gradients. Inputs: "v, grads, beta1". Output: "v".### START CODE HERE ### (approx. 2 lines)v["dW" + str(l+1)] = beta1*v["dW" + str(l+1)] + (1-beta1)*grads['dW' + str(l+1)]v["db" + str(l+1)] = beta1*v["db" + str(l+1)] + (1-beta1)*grads['db' + str(l+1)]### END CODE HERE #### Compute bias-corrected first moment estimate. Inputs: "v, beta1, t". Output: "v_corrected".### START CODE HERE ### (approx. 2 lines)v_corrected["dW" + str(l+1)] = v["dW" + str(l+1)]/(1-np.power(beta1,t))v_corrected["db" + str(l+1)] = v["db" + str(l+1)]/(1-np.power(beta1,t))### END CODE HERE #### Moving average of the squared gradients. Inputs: "s, grads, beta2". Output: "s".### START CODE HERE ### (approx. 2 lines)s["dW" + str(l+1)] = beta2*s["dW" + str(l+1)] + (1-beta2)*grads['dW' + str(l+1)]**2s["db" + str(l+1)] = beta2*s["db" + str(l+1)] + (1-beta2)*grads['db' + str(l+1)]**2### END CODE HERE #### Compute bias-corrected second raw moment estimate. Inputs: "s, beta2, t". Output: "s_corrected".### START CODE HERE ### (approx. 2 lines)s_corrected["dW" + str(l+1)] = s["dW" + str(l+1)]/(1-np.power(beta2,t))s_corrected["db" + str(l+1)] = s["db" + str(l+1)]/(1-np.power(beta2,t))### END CODE HERE #### Update parameters. Inputs: "parameters, learning_rate, v_corrected, s_corrected, epsilon". Output: "parameters".### START CODE HERE ### (approx. 2 lines)parameters["W" + str(l+1)] = parameters["W" + str(l+1)] - learning_rate*v_corrected["dW" + str(l+1)]/(np.sqrt(s_corrected["dW" + str(l+1)])+epsilon)parameters["b" + str(l+1)] = parameters["b" + str(l+1)] - learning_rate*v_corrected["db" + str(l+1)]/(np.sqrt(s_corrected["db" + str(l+1)])+epsilon)### END CODE HERE ###return parameters, v, s

5. 不同優(yōu)化算法下的模型

數(shù)據(jù)集：使用以下數(shù)據(jù)集進(jìn)行測(cè)試

3層神經(jīng)網(wǎng)絡(luò)模型：

• Mini-batch Gradient Descent:：
  使用函數(shù) update_parameters_with_gd()
• Mini-batch Momentum:：
  使用函數(shù) initialize_velocity() 和 update_parameters_with_momentum()
• Mini-batch Adam：
  使用函數(shù) initialize_adam() 和 update_parameters_with_adam()

def model(X, Y, layers_dims, optimizer, learning_rate = 0.0007, mini_batch_size = 64, beta = 0.9,beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8, num_epochs = 10000, print_cost = True):"""3-layer neural network model which can be run in different optimizer modes.Arguments:X -- input data, of shape (2, number of examples)Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)layers_dims -- python list, containing the size of each layerlearning_rate -- the learning rate, scalar.mini_batch_size -- the size of a mini batchbeta -- Momentum hyperparameterbeta1 -- Exponential decay hyperparameter for the past gradients estimates beta2 -- Exponential decay hyperparameter for the past squared gradients estimates epsilon -- hyperparameter preventing division by zero in Adam updatesnum_epochs -- number of epochsprint_cost -- True to print the cost every 1000 epochsReturns:parameters -- python dictionary containing your updated parameters """L = len(layers_dims) # number of layers in the neural networkscosts = [] # to keep track of the costt = 0 # initializing the counter required for Adam updateseed = 10 # For grading purposes, so that your "random" minibatches are the same as ours# Initialize parametersparameters = initialize_parameters(layers_dims)# Initialize the optimizerif optimizer == "gd":pass # no initialization required for gradient descentelif optimizer == "momentum":v = initialize_velocity(parameters)elif optimizer == "adam":v, s = initialize_adam(parameters)# Optimization loopfor i in range(num_epochs):# Define the random minibatches. We increment the seed to reshuffle differently the dataset after each epochseed = seed + 1minibatches = random_mini_batches(X, Y, mini_batch_size, seed)for minibatch in minibatches:# Select a minibatch(minibatch_X, minibatch_Y) = minibatch# Forward propagationa3, caches = forward_propagation(minibatch_X, parameters)# Compute costcost = compute_cost(a3, minibatch_Y)# Backward propagationgrads = backward_propagation(minibatch_X, minibatch_Y, caches)# Update parametersif optimizer == "gd":parameters = update_parameters_with_gd(parameters, grads, learning_rate)elif optimizer == "momentum":parameters, v = update_parameters_with_momentum(parameters, grads, v, beta, learning_rate)elif optimizer == "adam":t = t + 1 # Adam counterparameters, v, s = update_parameters_with_adam(parameters, grads, v, s,t, learning_rate, beta1, beta2, epsilon)# Print the cost every 1000 epochif print_cost and i % 1000 == 0:print ("Cost after epoch %i: %f" %(i, cost))if print_cost and i % 100 == 0:costs.append(cost)# plot the costplt.plot(costs)plt.ylabel('cost')plt.xlabel('epochs (per 100)')plt.title("Learning rate = " + str(learning_rate))plt.show()return parameters

5.1 Mini-batch梯度下降

# train 3-layer model layers_dims = [train_X.shape[0], 5, 2, 1] parameters = model(train_X, train_Y, layers_dims, optimizer = "gd")# Predict predictions = predict(train_X, train_Y, parameters)# Plot decision boundary plt.title("Model with Gradient Descent optimization") axes = plt.gca() axes.set_xlim([-1.5,2.5]) axes.set_ylim([-1,1.5]) plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)

表現(xiàn)：

Accuracy: 0.79 （Mini-batch梯度下降）

5.2 帶動(dòng)量的Mini-batch梯度下降

# train 3-layer model layers_dims = [train_X.shape[0], 5, 2, 1] parameters = model(train_X, train_Y, layers_dims, beta = 0.9, optimizer = "momentum")# Predict predictions = predict(train_X, train_Y, parameters)# Plot decision boundary plt.title("Model with Momentum optimization") axes = plt.gca() axes.set_xlim([-1.5,2.5]) axes.set_ylim([-1,1.5]) plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)

表現(xiàn)：

Accuracy: 0.79（帶動(dòng)量的Mini-batch梯度下降）

本例子由于太過(guò)簡(jiǎn)單，所以動(dòng)量的優(yōu)勢(shì)沒(méi)有體現(xiàn)出來(lái)，在大的數(shù)據(jù)集上會(huì)較不帶動(dòng)量的模型更好

5.3 帶Adam的Mini-batch梯度下降

# train 3-layer model layers_dims = [train_X.shape[0], 5, 2, 1] parameters = model(train_X, train_Y, layers_dims, optimizer = "adam")# Predict predictions = predict(train_X, train_Y, parameters)# Plot decision boundary plt.title("Model with Adam optimization") axes = plt.gca() axes.set_xlim([-1.5,2.5]) axes.set_ylim([-1,1.5]) plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y)

表現(xiàn)：

Accuracy: 0.9366666666666666（帶Adam的Mini-batch梯度下降）

5.4 對(duì)比總結(jié)

優(yōu)化方法準(zhǔn)確率cost shape

Gradient descent	79.7%	振蕩（我的結(jié)果是光滑）
Momentum	79.7%	振蕩 （我的結(jié)果是光滑，求指點(diǎn)）
Adam	94%	更光滑

• 動(dòng)量Momentum 通常是有幫助的，但是 較小的學(xué)習(xí)率 和 過(guò)于簡(jiǎn)單的數(shù)據(jù)集，優(yōu)勢(shì)體現(xiàn)不出來(lái)
• Adam，明顯優(yōu)于 mini-batch梯度下降 和 動(dòng)量
• 如果運(yùn)行更多的迭代次數(shù)，三種方法都會(huì)產(chǎn)生非常好的結(jié)果。但是 Adam 收斂更快
• Adam優(yōu)點(diǎn)：
  相對(duì)較低的內(nèi)存要求（雖然比 梯度下降 和 動(dòng)量梯度下降更高）
  即使很少調(diào)整超參數(shù)（除了𝛼 學(xué)習(xí)率），通常也能很好地工作

---

我的CSDN博客地址 https://michael.blog.csdn.net/

長(zhǎng)按或掃碼關(guān)注我的公眾號(hào)（Michael阿明），一起加油、一起學(xué)習(xí)進(jìn)步！

總結(jié)

以上是生活随笔為你收集整理的02.改善深层神经网络：超参数调试、正则化以及优化 W2.优化算法（作业：优化方法）的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。