學(xué)習(xí)率周期性變化，能后解決陷入鞍點(diǎn)的問(wèn)題，更多的方式請(qǐng)參考https://github.com/bckenstler/CLR?

base_lr:最低的學(xué)習(xí)率

max_lr:最高的學(xué)習(xí)率

step_size：（2-8）倍的每個(gè)epoch的訓(xùn)練次數(shù)。

scale_fn(x)：自定義用來(lái)減少最高的學(xué)習(xí)率。

cycle = np.floor(1+iterations/(2*step_size))
x = np.abs(iterations/step_size - 2*cycle + 1)
lr= base_lr + (max_lr-base_lr)*np.maximum(0, (1-x))*scale_fn(x)?

一，學(xué)習(xí)率周期性變化 global_steps=tf.placeholder(shape=[1],dtype=tf.int64) cycle = tf.cast(tf.floor(1. + tf.cast(global_steps, dtype=tf.float32) /(2 * 1000.)), dtype=tf.float32)x = tf.cast(tf.abs(tf.cast(global_steps, dtype=tf.float32) / 1000. - 2. * cycle + 1.), dtype=tf.float32)learning_rate = 1e-6 + (1e-3 - 1e-6) * tf.maximum(0., (1 - x)) with tf.Session() as sess:lr_list = []cycle_list=[]for i in range(8000):lr=sess.run(learning_rate,feed_dict={global_steps:[i]})lr_list.append(lr)cl = sess.run(cycle, feed_dict={global_steps: [i]})cycle_list.append(cl)plt.plot(lr_list)plt.show()print(cycle_list) def lr_change(base_lr=1e-6, max_lr=1e-3, step_size=1000.,):lr_list=[]for clr_iterations in range(8000):cycle = np.floor(1 + clr_iterations / (2 * step_size))x = np.abs(clr_iterations / step_size - 2 * cycle + 1)lr=base_lr + (max_lr - base_lr) * np.maximum(0, (1 - x))#/(2 ** (cycle - 1))lr_list.append(lr)plt.plot(lr_list,'r')plt.xlabel('iterations')plt.ylabel('learning rate')plt.savefig('./data/learning_rate.jpg')plt.show()

二，學(xué)習(xí)率周期性衰減

global_steps=tf.placeholder(shape=[1],dtype=tf.int64) cycle = tf.cast(tf.floor(1. + tf.cast(global_steps, dtype=tf.float32) /(2 * 1000.)), dtype=tf.float32)x = tf.cast(tf.abs(tf.cast(global_steps, dtype=tf.float32) / 1000. - 2. * cycle + 1.), dtype=tf.float32)learning_rate = 1e-6 + (1e-3 - 1e-6) * tf.maximum(0., (1 - x))/tf.cast(2**(cycle-1),dtype=tf.float32) with tf.Session() as sess:lr_list = []cycle_list=[]for i in range(8000):lr=sess.run(learning_rate,feed_dict={global_steps:[i]})lr_list.append(lr)cl = sess.run(cycle, feed_dict={global_steps: [i]})cycle_list.append(cl)plt.plot(lr_list)plt.show()print(cycle_list) def lr_change(base_lr=1e-6, max_lr=1e-3, step_size=1000.,):lr_list=[]for clr_iterations in range(8000):cycle = np.floor(1 + clr_iterations / (2 * step_size))x = np.abs(clr_iterations / step_size - 2 * cycle + 1)lr=base_lr + (max_lr - base_lr) * np.maximum(0, (1 - x))/(2 ** (cycle - 1))lr_list.append(lr)plt.plot(lr_list,'r')plt.xlabel('iterations')plt.ylabel('learning rate')plt.savefig('./data/learning_rate.jpg')plt.show()?

總結(jié)

以上是生活随笔為你收集整理的学习率周期性变化的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。