[Watermelon_book] Chapter 3 Linear Model
生活随笔
收集整理的這篇文章主要介紹了
[Watermelon_book] Chapter 3 Linear Model
小編覺得挺不錯的,現(xiàn)在分享給大家,幫大家做個參考.
- Linear Model
- 基本定義
- 線性模型簡單形式的實際編碼
- Task
- Generate data
- Cost function
- Gradient descent
- Training
- Model evaluation
- Experiment 1
- Experiment 2
- Task 2
- Generate data
- Normalization
- Prediction format
- Cost function
- Gradient descent
- Training
- Model evaluation
- 由回歸到分類
- Logistic Regression ( Binary Classification）的實際編碼
- Dataset
- Visulization
- Prediction Format
- Cost function
- Gradient descent
- Training
- Model evaluation
- Visulization
- Mapping probabilities to classes
Linear Model
基本定義
線性模型簡單形式的實際編碼
Task
We have derived the formula of Linear Model, here I want to write codes for it by hand. The specific task is to predict sales based on radios shown as the table below.
The original problem is from https://ml-cheatsheet.readthedocs.io/en/latest/linear_regression.html
| Amazon | 37.8 | 22.1 |
| 39.3 | 10.4 |
Generate data
import numpy as np import scipy import seaborn as sns import matplotlib.pyplot as plt sample_size = 200 np.random.seed(5) radio_sample = 60 * np.random.rand(sample_size) weight_truth = 0.4 bias_truth = -3 np.random.seed(10) nosie_sample = 10 * (np.random.normal(0, 0.1, sample_size)) sales_sample = radio_sample * weight_truth + bias_truth + nosie_sample sales_no_noise = radio_sample * weight_truth + bias_truth plt.scatter(radio_sample, sales_sample, alpha=0.5) plt.plot(radio_sample, sales_no_noise, c="r") plt.xlabel("radio") plt.ylabel("sales") Text(0, 0.5, 'sales')Cost function
f(m,b)=1N∑i=1n(yi?(mxi+b))2f(m, b)=\frac{1}{N} \sum_{i=1}^{n}\left(y_{i}-\left(m x_{i}+b\right)\right)^{2}f(m,b)=N1?∑i=1n?(yi??(mxi?+b))2
def cost_function(radio, sales, weight, bias):'''cost_function for linear modelArgs: radio,sales: is numpy arrayweight,bias: is scalar'''sample_size = len(radio)error = 0.0for i in range(sample_size):error += (sales[i] - (radio[i]*weight + bias))**2error_avg = error/sample_sizereturn error_avgGradient descent
\begin{aligned} f^{\prime}(m, b)=\left[ \begin{array}{c}{\frac{d f}{d m}} \ {\frac{d f}{d b}}\end{array}\right] &=\left[ \begin{array}{c}{\frac{1}{N} \sum-2 x_{i}\left(y_{i}-\left(m x_{i}+b\right)\right)} \ {\frac{1}{N} \sum-2\left(y_{i}-\left(m x_{i}+b\right)\right)}\end{array}\right] \end{aligned}
def update_weight(radio, sales, weight, bias, learning_rate):# initial valueweight_deriv = 0bias_deriv = 0sample_size = len(radio)for i in range(sample_size):# calculate partial derivativesweight_deriv += -2*radio[i]*(sales[i] - (weight*radio[i] + bias))bias_deriv += -2*(sales[i] - (weight*radio[i] + bias))#gradient descentweight = weight - (weight_deriv/sample_size)*learning_ratebias = bias - (bias_deriv/sample_size)*learning_ratereturn weight,biasTraining
def train(radio, sales, weight, bias, learning_rate, iters):cost_history = []for i in range(iters):weight, bias = update_weight(radio, sales, weight, bias, learning_rate)cost = cost_function(radio, sales, weight, bias)cost_history.append(cost)if i % 5 == 0:print("Iter: %d \t Weight: %2f \t Bias: %2f \t Cost: %4f" %(i,weight, bias, cost))return weight, bias, cost_historyModel evaluation
Experiment 1
weight_final, bias_final, cost_history = train(radio_sample, sales_sample, 0, 0, 0.0001, 2000) Iter: 0 Weight: 0.080657 Bias: 0.001851 Cost: 77.703045 Iter: 5 Weight: 0.267103 Bias: 0.005752 Cost: 7.320940 Iter: 10 Weight: 0.312288 Bias: 0.006165 Cost: 3.188519 Iter: 15 Weight: 0.323249 Bias: 0.005734 Cost: 2.945016 Iter: 20 Weight: 0.325918 Bias: 0.005099 Cost: 2.929796 Iter: 25 Weight: 0.326578 Bias: 0.004414 Cost: 2.927978 Iter: 30 Weight: 0.326751 Bias: 0.003717 Cost: 2.926946 Iter: 35 Weight: 0.326806 Bias: 0.003017 Cost: 2.925961 Iter: 40 Weight: 0.326832 Bias: 0.002317 Cost: 2.924980 Iter: 45 Weight: 0.326852 Bias: 0.001617 Cost: 2.923999 Iter: 50 Weight: 0.326869 Bias: 0.000917 Cost: 2.923018 Iter: 55 Weight: 0.326887 Bias: 0.000217 Cost: 2.922038 Iter: 60 Weight: 0.326904 Bias: -0.000482 Cost: 2.921058 Iter: 65 Weight: 0.326921 Bias: -0.001182 Cost: 2.920079 Iter: 70 Weight: 0.326939 Bias: -0.001881 Cost: 2.919101 Iter: 75 Weight: 0.326956 Bias: -0.002580 Cost: 2.918123 Iter: 80 Weight: 0.326973 Bias: -0.003279 Cost: 2.917145 Iter: 85 Weight: 0.326990 Bias: -0.003978 Cost: 2.916168 Iter: 90 Weight: 0.327008 Bias: -0.004677 Cost: 2.915191 Iter: 95 Weight: 0.327025 Bias: -0.005375 Cost: 2.914215 Iter: 100 Weight: 0.327042 Bias: -0.006073 Cost: 2.913240 Iter: 105 Weight: 0.327059 Bias: -0.006771 Cost: 2.912264 Iter: 110 Weight: 0.327077 Bias: -0.007469 Cost: 2.911290 Iter: 115 Weight: 0.327094 Bias: -0.008167 Cost: 2.910315 Iter: 120 Weight: 0.327111 Bias: -0.008864 Cost: 2.909342 Iter: 125 Weight: 0.327128 Bias: -0.009562 Cost: 2.908368 Iter: 130 Weight: 0.327145 Bias: -0.010259 Cost: 2.907396 Iter: 135 Weight: 0.327163 Bias: -0.010956 Cost: 2.906423 Iter: 140 Weight: 0.327180 Bias: -0.011653 Cost: 2.905452 Iter: 145 Weight: 0.327197 Bias: -0.012350 Cost: 2.904480 Iter: 150 Weight: 0.327214 Bias: -0.013046 Cost: 2.903509 Iter: 155 Weight: 0.327231 Bias: -0.013742 Cost: 2.902539 Iter: 160 Weight: 0.327249 Bias: -0.014439 Cost: 2.901569 Iter: 165 Weight: 0.327266 Bias: -0.015135 Cost: 2.900600 Iter: 170 Weight: 0.327283 Bias: -0.015830 Cost: 2.899631 Iter: 175 Weight: 0.327300 Bias: -0.016526 Cost: 2.898663 Iter: 180 Weight: 0.327317 Bias: -0.017222 Cost: 2.897695 Iter: 185 Weight: 0.327335 Bias: -0.017917 Cost: 2.896727 Iter: 190 Weight: 0.327352 Bias: -0.018612 Cost: 2.895760 Iter: 195 Weight: 0.327369 Bias: -0.019307 Cost: 2.894794 Iter: 200 Weight: 0.327386 Bias: -0.020002 Cost: 2.893828 Iter: 205 Weight: 0.327403 Bias: -0.020696 Cost: 2.892862 Iter: 210 Weight: 0.327420 Bias: -0.021391 Cost: 2.891897 Iter: 215 Weight: 0.327437 Bias: -0.022085 Cost: 2.890932 Iter: 220 Weight: 0.327455 Bias: -0.022779 Cost: 2.889968 Iter: 225 Weight: 0.327472 Bias: -0.023473 Cost: 2.889005 Iter: 230 Weight: 0.327489 Bias: -0.024167 Cost: 2.888042 Iter: 235 Weight: 0.327506 Bias: -0.024860 Cost: 2.887079 Iter: 240 Weight: 0.327523 Bias: -0.025554 Cost: 2.886117 Iter: 245 Weight: 0.327540 Bias: -0.026247 Cost: 2.885155 Iter: 250 Weight: 0.327557 Bias: -0.026940 Cost: 2.884194 Iter: 255 Weight: 0.327574 Bias: -0.027633 Cost: 2.883233 Iter: 260 Weight: 0.327592 Bias: -0.028326 Cost: 2.882273 Iter: 265 Weight: 0.327609 Bias: -0.029018 Cost: 2.881313 Iter: 270 Weight: 0.327626 Bias: -0.029711 Cost: 2.880354 Iter: 275 Weight: 0.327643 Bias: -0.030403 Cost: 2.879395 Iter: 280 Weight: 0.327660 Bias: -0.031095 Cost: 2.878436 Iter: 285 Weight: 0.327677 Bias: -0.031787 Cost: 2.877479 Iter: 290 Weight: 0.327694 Bias: -0.032478 Cost: 2.876521 Iter: 295 Weight: 0.327711 Bias: -0.033170 Cost: 2.875564 Iter: 300 Weight: 0.327728 Bias: -0.033861 Cost: 2.874608 Iter: 305 Weight: 0.327745 Bias: -0.034552 Cost: 2.873652 Iter: 310 Weight: 0.327762 Bias: -0.035243 Cost: 2.872696 Iter: 315 Weight: 0.327779 Bias: -0.035934 Cost: 2.871741 Iter: 320 Weight: 0.327796 Bias: -0.036625 Cost: 2.870787 Iter: 325 Weight: 0.327813 Bias: -0.037315 Cost: 2.869832 Iter: 330 Weight: 0.327830 Bias: -0.038006 Cost: 2.868879 Iter: 335 Weight: 0.327848 Bias: -0.038696 Cost: 2.867926 Iter: 340 Weight: 0.327865 Bias: -0.039386 Cost: 2.866973 Iter: 345 Weight: 0.327882 Bias: -0.040076 Cost: 2.866021 Iter: 350 Weight: 0.327899 Bias: -0.040765 Cost: 2.865069 Iter: 355 Weight: 0.327916 Bias: -0.041455 Cost: 2.864118 Iter: 360 Weight: 0.327933 Bias: -0.042144 Cost: 2.863167 Iter: 365 Weight: 0.327950 Bias: -0.042833 Cost: 2.862217 Iter: 370 Weight: 0.327967 Bias: -0.043522 Cost: 2.861267 Iter: 375 Weight: 0.327984 Bias: -0.044211 Cost: 2.860317 Iter: 380 Weight: 0.328001 Bias: -0.044899 Cost: 2.859369 Iter: 385 Weight: 0.328018 Bias: -0.045588 Cost: 2.858420 Iter: 390 Weight: 0.328035 Bias: -0.046276 Cost: 2.857472 Iter: 395 Weight: 0.328052 Bias: -0.046964 Cost: 2.856525 Iter: 400 Weight: 0.328069 Bias: -0.047652 Cost: 2.855578 Iter: 405 Weight: 0.328086 Bias: -0.048340 Cost: 2.854631 Iter: 410 Weight: 0.328103 Bias: -0.049028 Cost: 2.853685 Iter: 415 Weight: 0.328120 Bias: -0.049715 Cost: 2.852739 Iter: 420 Weight: 0.328137 Bias: -0.050402 Cost: 2.851794 Iter: 425 Weight: 0.328153 Bias: -0.051089 Cost: 2.850850 Iter: 430 Weight: 0.328170 Bias: -0.051776 Cost: 2.849905 Iter: 435 Weight: 0.328187 Bias: -0.052463 Cost: 2.848962 Iter: 440 Weight: 0.328204 Bias: -0.053149 Cost: 2.848018 Iter: 445 Weight: 0.328221 Bias: -0.053836 Cost: 2.847076 Iter: 450 Weight: 0.328238 Bias: -0.054522 Cost: 2.846133 Iter: 455 Weight: 0.328255 Bias: -0.055208 Cost: 2.845191 Iter: 460 Weight: 0.328272 Bias: -0.055894 Cost: 2.844250 Iter: 465 Weight: 0.328289 Bias: -0.056580 Cost: 2.843309 Iter: 470 Weight: 0.328306 Bias: -0.057265 Cost: 2.842369 Iter: 475 Weight: 0.328323 Bias: -0.057951 Cost: 2.841429 Iter: 480 Weight: 0.328340 Bias: -0.058636 Cost: 2.840489 Iter: 485 Weight: 0.328357 Bias: -0.059321 Cost: 2.839550 Iter: 490 Weight: 0.328374 Bias: -0.060006 Cost: 2.838611 Iter: 495 Weight: 0.328390 Bias: -0.060690 Cost: 2.837673 Iter: 500 Weight: 0.328407 Bias: -0.061375 Cost: 2.836736 Iter: 505 Weight: 0.328424 Bias: -0.062059 Cost: 2.835798 Iter: 510 Weight: 0.328441 Bias: -0.062743 Cost: 2.834862 Iter: 515 Weight: 0.328458 Bias: -0.063427 Cost: 2.833925 Iter: 520 Weight: 0.328475 Bias: -0.064111 Cost: 2.832990 Iter: 525 Weight: 0.328492 Bias: -0.064795 Cost: 2.832054 Iter: 530 Weight: 0.328509 Bias: -0.065478 Cost: 2.831119 Iter: 535 Weight: 0.328526 Bias: -0.066162 Cost: 2.830185 Iter: 540 Weight: 0.328542 Bias: -0.066845 Cost: 2.829251 Iter: 545 Weight: 0.328559 Bias: -0.067528 Cost: 2.828318 Iter: 550 Weight: 0.328576 Bias: -0.068211 Cost: 2.827385 Iter: 555 Weight: 0.328593 Bias: -0.068893 Cost: 2.826452 Iter: 560 Weight: 0.328610 Bias: -0.069576 Cost: 2.825520 Iter: 565 Weight: 0.328627 Bias: -0.070258 Cost: 2.824588 Iter: 570 Weight: 0.328644 Bias: -0.070940 Cost: 2.823657 Iter: 575 Weight: 0.328660 Bias: -0.071622 Cost: 2.822727 Iter: 580 Weight: 0.328677 Bias: -0.072304 Cost: 2.821796 Iter: 585 Weight: 0.328694 Bias: -0.072985 Cost: 2.820867 Iter: 590 Weight: 0.328711 Bias: -0.073667 Cost: 2.819937 Iter: 595 Weight: 0.328728 Bias: -0.074348 Cost: 2.819008 Iter: 600 Weight: 0.328744 Bias: -0.075029 Cost: 2.818080 Iter: 605 Weight: 0.328761 Bias: -0.075710 Cost: 2.817152 Iter: 610 Weight: 0.328778 Bias: -0.076391 Cost: 2.816225 Iter: 615 Weight: 0.328795 Bias: -0.077072 Cost: 2.815298 Iter: 620 Weight: 0.328812 Bias: -0.077752 Cost: 2.814371 Iter: 625 Weight: 0.328828 Bias: -0.078432 Cost: 2.813445 Iter: 630 Weight: 0.328845 Bias: -0.079113 Cost: 2.812519 Iter: 635 Weight: 0.328862 Bias: -0.079792 Cost: 2.811594 Iter: 640 Weight: 0.328879 Bias: -0.080472 Cost: 2.810669 Iter: 645 Weight: 0.328896 Bias: -0.081152 Cost: 2.809745 Iter: 650 Weight: 0.328912 Bias: -0.081831 Cost: 2.808821 Iter: 655 Weight: 0.328929 Bias: -0.082511 Cost: 2.807898 Iter: 660 Weight: 0.328946 Bias: -0.083190 Cost: 2.806975 Iter: 665 Weight: 0.328963 Bias: -0.083869 Cost: 2.806053 Iter: 670 Weight: 0.328979 Bias: -0.084547 Cost: 2.805131 Iter: 675 Weight: 0.328996 Bias: -0.085226 Cost: 2.804209 Iter: 680 Weight: 0.329013 Bias: -0.085904 Cost: 2.803288 Iter: 685 Weight: 0.329030 Bias: -0.086583 Cost: 2.802368 Iter: 690 Weight: 0.329046 Bias: -0.087261 Cost: 2.801447 Iter: 695 Weight: 0.329063 Bias: -0.087939 Cost: 2.800528 Iter: 700 Weight: 0.329080 Bias: -0.088616 Cost: 2.799609 Iter: 705 Weight: 0.329097 Bias: -0.089294 Cost: 2.798690 Iter: 710 Weight: 0.329113 Bias: -0.089971 Cost: 2.797772 Iter: 715 Weight: 0.329130 Bias: -0.090649 Cost: 2.796854 Iter: 720 Weight: 0.329147 Bias: -0.091326 Cost: 2.795936 Iter: 725 Weight: 0.329163 Bias: -0.092003 Cost: 2.795019 Iter: 730 Weight: 0.329180 Bias: -0.092679 Cost: 2.794103 Iter: 735 Weight: 0.329197 Bias: -0.093356 Cost: 2.793187 Iter: 740 Weight: 0.329214 Bias: -0.094032 Cost: 2.792271 Iter: 745 Weight: 0.329230 Bias: -0.094708 Cost: 2.791356 Iter: 750 Weight: 0.329247 Bias: -0.095385 Cost: 2.790442 Iter: 755 Weight: 0.329264 Bias: -0.096060 Cost: 2.789527 Iter: 760 Weight: 0.329280 Bias: -0.096736 Cost: 2.788614 Iter: 765 Weight: 0.329297 Bias: -0.097412 Cost: 2.787700 Iter: 770 Weight: 0.329314 Bias: -0.098087 Cost: 2.786787 Iter: 775 Weight: 0.329330 Bias: -0.098762 Cost: 2.785875 Iter: 780 Weight: 0.329347 Bias: -0.099437 Cost: 2.784963 Iter: 785 Weight: 0.329364 Bias: -0.100112 Cost: 2.784052 Iter: 790 Weight: 0.329380 Bias: -0.100787 Cost: 2.783141 Iter: 795 Weight: 0.329397 Bias: -0.101462 Cost: 2.782230 Iter: 800 Weight: 0.329414 Bias: -0.102136 Cost: 2.781320 Iter: 805 Weight: 0.329430 Bias: -0.102810 Cost: 2.780410 Iter: 810 Weight: 0.329447 Bias: -0.103484 Cost: 2.779501 Iter: 815 Weight: 0.329464 Bias: -0.104158 Cost: 2.778592 Iter: 820 Weight: 0.329480 Bias: -0.104832 Cost: 2.777684 Iter: 825 Weight: 0.329497 Bias: -0.105505 Cost: 2.776776 Iter: 830 Weight: 0.329513 Bias: -0.106179 Cost: 2.775869 Iter: 835 Weight: 0.329530 Bias: -0.106852 Cost: 2.774962 Iter: 840 Weight: 0.329547 Bias: -0.107525 Cost: 2.774055 Iter: 845 Weight: 0.329563 Bias: -0.108198 Cost: 2.773149 Iter: 850 Weight: 0.329580 Bias: -0.108871 Cost: 2.772243 Iter: 855 Weight: 0.329597 Bias: -0.109543 Cost: 2.771338 Iter: 860 Weight: 0.329613 Bias: -0.110216 Cost: 2.770433 Iter: 865 Weight: 0.329630 Bias: -0.110888 Cost: 2.769529 Iter: 870 Weight: 0.329646 Bias: -0.111560 Cost: 2.768625 Iter: 875 Weight: 0.329663 Bias: -0.112232 Cost: 2.767722 Iter: 880 Weight: 0.329679 Bias: -0.112903 Cost: 2.766819 Iter: 885 Weight: 0.329696 Bias: -0.113575 Cost: 2.765917 Iter: 890 Weight: 0.329713 Bias: -0.114246 Cost: 2.765015 Iter: 895 Weight: 0.329729 Bias: -0.114918 Cost: 2.764113 Iter: 900 Weight: 0.329746 Bias: -0.115589 Cost: 2.763212 Iter: 905 Weight: 0.329762 Bias: -0.116259 Cost: 2.762311 Iter: 910 Weight: 0.329779 Bias: -0.116930 Cost: 2.761411 Iter: 915 Weight: 0.329795 Bias: -0.117601 Cost: 2.760511 Iter: 920 Weight: 0.329812 Bias: -0.118271 Cost: 2.759612 Iter: 925 Weight: 0.329829 Bias: -0.118941 Cost: 2.758713 Iter: 930 Weight: 0.329845 Bias: -0.119611 Cost: 2.757814 Iter: 935 Weight: 0.329862 Bias: -0.120281 Cost: 2.756916 Iter: 940 Weight: 0.329878 Bias: -0.120951 Cost: 2.756019 Iter: 945 Weight: 0.329895 Bias: -0.121621 Cost: 2.755122 Iter: 950 Weight: 0.329911 Bias: -0.122290 Cost: 2.754225 Iter: 955 Weight: 0.329928 Bias: -0.122959 Cost: 2.753329 Iter: 960 Weight: 0.329944 Bias: -0.123628 Cost: 2.752433 Iter: 965 Weight: 0.329961 Bias: -0.124297 Cost: 2.751538 Iter: 970 Weight: 0.329977 Bias: -0.124966 Cost: 2.750643 Iter: 975 Weight: 0.329994 Bias: -0.125634 Cost: 2.749748 Iter: 980 Weight: 0.330010 Bias: -0.126303 Cost: 2.748854 Iter: 985 Weight: 0.330027 Bias: -0.126971 Cost: 2.747961 Iter: 990 Weight: 0.330043 Bias: -0.127639 Cost: 2.747068 Iter: 995 Weight: 0.330060 Bias: -0.128307 Cost: 2.746175 Iter: 1000 Weight: 0.330076 Bias: -0.128975 Cost: 2.745283 Iter: 1005 Weight: 0.330093 Bias: -0.129642 Cost: 2.744391 Iter: 1010 Weight: 0.330109 Bias: -0.130310 Cost: 2.743500 Iter: 1015 Weight: 0.330126 Bias: -0.130977 Cost: 2.742609 Iter: 1020 Weight: 0.330142 Bias: -0.131644 Cost: 2.741718 Iter: 1025 Weight: 0.330159 Bias: -0.132311 Cost: 2.740828 Iter: 1030 Weight: 0.330175 Bias: -0.132977 Cost: 2.739939 Iter: 1035 Weight: 0.330191 Bias: -0.133644 Cost: 2.739050 Iter: 1040 Weight: 0.330208 Bias: -0.134310 Cost: 2.738161 Iter: 1045 Weight: 0.330224 Bias: -0.134977 Cost: 2.737273 Iter: 1050 Weight: 0.330241 Bias: -0.135643 Cost: 2.736385 Iter: 1055 Weight: 0.330257 Bias: -0.136309 Cost: 2.735497 Iter: 1060 Weight: 0.330274 Bias: -0.136974 Cost: 2.734610 Iter: 1065 Weight: 0.330290 Bias: -0.137640 Cost: 2.733724 Iter: 1070 Weight: 0.330307 Bias: -0.138305 Cost: 2.732838 Iter: 1075 Weight: 0.330323 Bias: -0.138971 Cost: 2.731952 Iter: 1080 Weight: 0.330339 Bias: -0.139636 Cost: 2.731067 Iter: 1085 Weight: 0.330356 Bias: -0.140301 Cost: 2.730182 Iter: 1090 Weight: 0.330372 Bias: -0.140965 Cost: 2.729298 Iter: 1095 Weight: 0.330389 Bias: -0.141630 Cost: 2.728414 Iter: 1100 Weight: 0.330405 Bias: -0.142294 Cost: 2.727531 Iter: 1105 Weight: 0.330421 Bias: -0.142959 Cost: 2.726648 Iter: 1110 Weight: 0.330438 Bias: -0.143623 Cost: 2.725765 Iter: 1115 Weight: 0.330454 Bias: -0.144287 Cost: 2.724883 Iter: 1120 Weight: 0.330471 Bias: -0.144950 Cost: 2.724002 Iter: 1125 Weight: 0.330487 Bias: -0.145614 Cost: 2.723120 Iter: 1130 Weight: 0.330503 Bias: -0.146277 Cost: 2.722240 Iter: 1135 Weight: 0.330520 Bias: -0.146941 Cost: 2.721359 Iter: 1140 Weight: 0.330536 Bias: -0.147604 Cost: 2.720479 Iter: 1145 Weight: 0.330552 Bias: -0.148267 Cost: 2.719600 Iter: 1150 Weight: 0.330569 Bias: -0.148929 Cost: 2.718721 Iter: 1155 Weight: 0.330585 Bias: -0.149592 Cost: 2.717842 Iter: 1160 Weight: 0.330602 Bias: -0.150254 Cost: 2.716964 Iter: 1165 Weight: 0.330618 Bias: -0.150917 Cost: 2.716086 Iter: 1170 Weight: 0.330634 Bias: -0.151579 Cost: 2.715209 Iter: 1175 Weight: 0.330651 Bias: -0.152241 Cost: 2.714332 Iter: 1180 Weight: 0.330667 Bias: -0.152903 Cost: 2.713456 Iter: 1185 Weight: 0.330683 Bias: -0.153564 Cost: 2.712580 Iter: 1190 Weight: 0.330700 Bias: -0.154226 Cost: 2.711704 Iter: 1195 Weight: 0.330716 Bias: -0.154887 Cost: 2.710829 Iter: 1200 Weight: 0.330732 Bias: -0.155548 Cost: 2.709955 Iter: 1205 Weight: 0.330749 Bias: -0.156209 Cost: 2.709080 Iter: 1210 Weight: 0.330765 Bias: -0.156870 Cost: 2.708206 Iter: 1215 Weight: 0.330781 Bias: -0.157530 Cost: 2.707333 Iter: 1220 Weight: 0.330797 Bias: -0.158191 Cost: 2.706460 Iter: 1225 Weight: 0.330814 Bias: -0.158851 Cost: 2.705588 Iter: 1230 Weight: 0.330830 Bias: -0.159511 Cost: 2.704716 Iter: 1235 Weight: 0.330846 Bias: -0.160171 Cost: 2.703844 Iter: 1240 Weight: 0.330863 Bias: -0.160831 Cost: 2.702973 Iter: 1245 Weight: 0.330879 Bias: -0.161491 Cost: 2.702102 Iter: 1250 Weight: 0.330895 Bias: -0.162150 Cost: 2.701232 Iter: 1255 Weight: 0.330911 Bias: -0.162810 Cost: 2.700362 Iter: 1260 Weight: 0.330928 Bias: -0.163469 Cost: 2.699492 Iter: 1265 Weight: 0.330944 Bias: -0.164128 Cost: 2.698623 Iter: 1270 Weight: 0.330960 Bias: -0.164787 Cost: 2.697755 Iter: 1275 Weight: 0.330977 Bias: -0.165445 Cost: 2.696886 Iter: 1280 Weight: 0.330993 Bias: -0.166104 Cost: 2.696019 Iter: 1285 Weight: 0.331009 Bias: -0.166762 Cost: 2.695151 Iter: 1290 Weight: 0.331025 Bias: -0.167420 Cost: 2.694284 Iter: 1295 Weight: 0.331042 Bias: -0.168078 Cost: 2.693418 Iter: 1300 Weight: 0.331058 Bias: -0.168736 Cost: 2.692552 Iter: 1305 Weight: 0.331074 Bias: -0.169394 Cost: 2.691686 Iter: 1310 Weight: 0.331090 Bias: -0.170051 Cost: 2.690821 Iter: 1315 Weight: 0.331107 Bias: -0.170709 Cost: 2.689956 Iter: 1320 Weight: 0.331123 Bias: -0.171366 Cost: 2.689092 Iter: 1325 Weight: 0.331139 Bias: -0.172023 Cost: 2.688228 Iter: 1330 Weight: 0.331155 Bias: -0.172680 Cost: 2.687365 Iter: 1335 Weight: 0.331171 Bias: -0.173336 Cost: 2.686502 Iter: 1340 Weight: 0.331188 Bias: -0.173993 Cost: 2.685639 Iter: 1345 Weight: 0.331204 Bias: -0.174649 Cost: 2.684777 Iter: 1350 Weight: 0.331220 Bias: -0.175306 Cost: 2.683915 Iter: 1355 Weight: 0.331236 Bias: -0.175962 Cost: 2.683054 Iter: 1360 Weight: 0.331252 Bias: -0.176618 Cost: 2.682193 Iter: 1365 Weight: 0.331269 Bias: -0.177273 Cost: 2.681332 Iter: 1370 Weight: 0.331285 Bias: -0.177929 Cost: 2.680472 Iter: 1375 Weight: 0.331301 Bias: -0.178584 Cost: 2.679613 Iter: 1380 Weight: 0.331317 Bias: -0.179239 Cost: 2.678754 Iter: 1385 Weight: 0.331333 Bias: -0.179895 Cost: 2.677895 Iter: 1390 Weight: 0.331349 Bias: -0.180549 Cost: 2.677037 Iter: 1395 Weight: 0.331366 Bias: -0.181204 Cost: 2.676179 Iter: 1400 Weight: 0.331382 Bias: -0.181859 Cost: 2.675321 Iter: 1405 Weight: 0.331398 Bias: -0.182513 Cost: 2.674464 Iter: 1410 Weight: 0.331414 Bias: -0.183167 Cost: 2.673607 Iter: 1415 Weight: 0.331430 Bias: -0.183822 Cost: 2.672751 Iter: 1420 Weight: 0.331446 Bias: -0.184476 Cost: 2.671896 Iter: 1425 Weight: 0.331463 Bias: -0.185129 Cost: 2.671040 Iter: 1430 Weight: 0.331479 Bias: -0.185783 Cost: 2.670185 Iter: 1435 Weight: 0.331495 Bias: -0.186436 Cost: 2.669331 Iter: 1440 Weight: 0.331511 Bias: -0.187090 Cost: 2.668477 Iter: 1445 Weight: 0.331527 Bias: -0.187743 Cost: 2.667623 Iter: 1450 Weight: 0.331543 Bias: -0.188396 Cost: 2.666770 Iter: 1455 Weight: 0.331559 Bias: -0.189049 Cost: 2.665917 Iter: 1460 Weight: 0.331575 Bias: -0.189701 Cost: 2.665065 Iter: 1465 Weight: 0.331591 Bias: -0.190354 Cost: 2.664213 Iter: 1470 Weight: 0.331608 Bias: -0.191006 Cost: 2.663361 Iter: 1475 Weight: 0.331624 Bias: -0.191658 Cost: 2.662510 Iter: 1480 Weight: 0.331640 Bias: -0.192310 Cost: 2.661659 Iter: 1485 Weight: 0.331656 Bias: -0.192962 Cost: 2.660809 Iter: 1490 Weight: 0.331672 Bias: -0.193614 Cost: 2.659959 Iter: 1495 Weight: 0.331688 Bias: -0.194265 Cost: 2.659110 Iter: 1500 Weight: 0.331704 Bias: -0.194917 Cost: 2.658261 Iter: 1505 Weight: 0.331720 Bias: -0.195568 Cost: 2.657412 Iter: 1510 Weight: 0.331736 Bias: -0.196219 Cost: 2.656564 Iter: 1515 Weight: 0.331752 Bias: -0.196870 Cost: 2.655716 Iter: 1520 Weight: 0.331768 Bias: -0.197520 Cost: 2.654869 Iter: 1525 Weight: 0.331784 Bias: -0.198171 Cost: 2.654022 Iter: 1530 Weight: 0.331801 Bias: -0.198821 Cost: 2.653176 Iter: 1535 Weight: 0.331817 Bias: -0.199471 Cost: 2.652330 Iter: 1540 Weight: 0.331833 Bias: -0.200121 Cost: 2.651484 Iter: 1545 Weight: 0.331849 Bias: -0.200771 Cost: 2.650639 Iter: 1550 Weight: 0.331865 Bias: -0.201421 Cost: 2.649794 Iter: 1555 Weight: 0.331881 Bias: -0.202071 Cost: 2.648950 Iter: 1560 Weight: 0.331897 Bias: -0.202720 Cost: 2.648106 Iter: 1565 Weight: 0.331913 Bias: -0.203369 Cost: 2.647262 Iter: 1570 Weight: 0.331929 Bias: -0.204018 Cost: 2.646419 Iter: 1575 Weight: 0.331945 Bias: -0.204667 Cost: 2.645576 Iter: 1580 Weight: 0.331961 Bias: -0.205316 Cost: 2.644734 Iter: 1585 Weight: 0.331977 Bias: -0.205965 Cost: 2.643892 Iter: 1590 Weight: 0.331993 Bias: -0.206613 Cost: 2.643051 Iter: 1595 Weight: 0.332009 Bias: -0.207261 Cost: 2.642210 Iter: 1600 Weight: 0.332025 Bias: -0.207910 Cost: 2.641369 Iter: 1605 Weight: 0.332041 Bias: -0.208557 Cost: 2.640529 Iter: 1610 Weight: 0.332057 Bias: -0.209205 Cost: 2.639689 Iter: 1615 Weight: 0.332073 Bias: -0.209853 Cost: 2.638850 Iter: 1620 Weight: 0.332089 Bias: -0.210500 Cost: 2.638011 Iter: 1625 Weight: 0.332105 Bias: -0.211148 Cost: 2.637172 Iter: 1630 Weight: 0.332121 Bias: -0.211795 Cost: 2.636334 Iter: 1635 Weight: 0.332137 Bias: -0.212442 Cost: 2.635496 Iter: 1640 Weight: 0.332153 Bias: -0.213089 Cost: 2.634659 Iter: 1645 Weight: 0.332169 Bias: -0.213735 Cost: 2.633822 Iter: 1650 Weight: 0.332185 Bias: -0.214382 Cost: 2.632986 Iter: 1655 Weight: 0.332201 Bias: -0.215028 Cost: 2.632150 Iter: 1660 Weight: 0.332217 Bias: -0.215674 Cost: 2.631314 Iter: 1665 Weight: 0.332233 Bias: -0.216320 Cost: 2.630479 Iter: 1670 Weight: 0.332248 Bias: -0.216966 Cost: 2.629644 Iter: 1675 Weight: 0.332264 Bias: -0.217612 Cost: 2.628810 Iter: 1680 Weight: 0.332280 Bias: -0.218258 Cost: 2.627976 Iter: 1685 Weight: 0.332296 Bias: -0.218903 Cost: 2.627142 Iter: 1690 Weight: 0.332312 Bias: -0.219548 Cost: 2.626309 Iter: 1695 Weight: 0.332328 Bias: -0.220193 Cost: 2.625477 Iter: 1700 Weight: 0.332344 Bias: -0.220838 Cost: 2.624644 Iter: 1705 Weight: 0.332360 Bias: -0.221483 Cost: 2.623812 Iter: 1710 Weight: 0.332376 Bias: -0.222128 Cost: 2.622981 Iter: 1715 Weight: 0.332392 Bias: -0.222772 Cost: 2.622150 Iter: 1720 Weight: 0.332408 Bias: -0.223416 Cost: 2.621319 Iter: 1725 Weight: 0.332424 Bias: -0.224060 Cost: 2.620489 Iter: 1730 Weight: 0.332439 Bias: -0.224704 Cost: 2.619659 Iter: 1735 Weight: 0.332455 Bias: -0.225348 Cost: 2.618830 Iter: 1740 Weight: 0.332471 Bias: -0.225992 Cost: 2.618001 Iter: 1745 Weight: 0.332487 Bias: -0.226635 Cost: 2.617172 Iter: 1750 Weight: 0.332503 Bias: -0.227278 Cost: 2.616344 Iter: 1755 Weight: 0.332519 Bias: -0.227922 Cost: 2.615516 Iter: 1760 Weight: 0.332535 Bias: -0.228565 Cost: 2.614689 Iter: 1765 Weight: 0.332551 Bias: -0.229207 Cost: 2.613862 Iter: 1770 Weight: 0.332567 Bias: -0.229850 Cost: 2.613035 Iter: 1775 Weight: 0.332582 Bias: -0.230493 Cost: 2.612209 Iter: 1780 Weight: 0.332598 Bias: -0.231135 Cost: 2.611383 Iter: 1785 Weight: 0.332614 Bias: -0.231777 Cost: 2.610558 Iter: 1790 Weight: 0.332630 Bias: -0.232419 Cost: 2.609733 Iter: 1795 Weight: 0.332646 Bias: -0.233061 Cost: 2.608909 Iter: 1800 Weight: 0.332662 Bias: -0.233703 Cost: 2.608085 Iter: 1805 Weight: 0.332677 Bias: -0.234344 Cost: 2.607261 Iter: 1810 Weight: 0.332693 Bias: -0.234986 Cost: 2.606438 Iter: 1815 Weight: 0.332709 Bias: -0.235627 Cost: 2.605615 Iter: 1820 Weight: 0.332725 Bias: -0.236268 Cost: 2.604793 Iter: 1825 Weight: 0.332741 Bias: -0.236909 Cost: 2.603970 Iter: 1830 Weight: 0.332757 Bias: -0.237550 Cost: 2.603149 Iter: 1835 Weight: 0.332772 Bias: -0.238190 Cost: 2.602328 Iter: 1840 Weight: 0.332788 Bias: -0.238831 Cost: 2.601507 Iter: 1845 Weight: 0.332804 Bias: -0.239471 Cost: 2.600686 Iter: 1850 Weight: 0.332820 Bias: -0.240111 Cost: 2.599866 Iter: 1855 Weight: 0.332836 Bias: -0.240751 Cost: 2.599047 Iter: 1860 Weight: 0.332851 Bias: -0.241391 Cost: 2.598228 Iter: 1865 Weight: 0.332867 Bias: -0.242031 Cost: 2.597409 Iter: 1870 Weight: 0.332883 Bias: -0.242670 Cost: 2.596591 Iter: 1875 Weight: 0.332899 Bias: -0.243309 Cost: 2.595773 Iter: 1880 Weight: 0.332915 Bias: -0.243949 Cost: 2.594955 Iter: 1885 Weight: 0.332930 Bias: -0.244588 Cost: 2.594138 Iter: 1890 Weight: 0.332946 Bias: -0.245226 Cost: 2.593321 Iter: 1895 Weight: 0.332962 Bias: -0.245865 Cost: 2.592505 Iter: 1900 Weight: 0.332978 Bias: -0.246504 Cost: 2.591689 Iter: 1905 Weight: 0.332993 Bias: -0.247142 Cost: 2.590873 Iter: 1910 Weight: 0.333009 Bias: -0.247780 Cost: 2.590058 Iter: 1915 Weight: 0.333025 Bias: -0.248418 Cost: 2.589244 Iter: 1920 Weight: 0.333041 Bias: -0.249056 Cost: 2.588429 Iter: 1925 Weight: 0.333056 Bias: -0.249694 Cost: 2.587615 Iter: 1930 Weight: 0.333072 Bias: -0.250332 Cost: 2.586802 Iter: 1935 Weight: 0.333088 Bias: -0.250969 Cost: 2.585989 Iter: 1940 Weight: 0.333104 Bias: -0.251606 Cost: 2.585176 Iter: 1945 Weight: 0.333119 Bias: -0.252243 Cost: 2.584364 Iter: 1950 Weight: 0.333135 Bias: -0.252880 Cost: 2.583552 Iter: 1955 Weight: 0.333151 Bias: -0.253517 Cost: 2.582740 Iter: 1960 Weight: 0.333167 Bias: -0.254154 Cost: 2.581929 Iter: 1965 Weight: 0.333182 Bias: -0.254790 Cost: 2.581119 Iter: 1970 Weight: 0.333198 Bias: -0.255426 Cost: 2.580308 Iter: 1975 Weight: 0.333214 Bias: -0.256063 Cost: 2.579498 Iter: 1980 Weight: 0.333229 Bias: -0.256699 Cost: 2.578689 Iter: 1985 Weight: 0.333245 Bias: -0.257335 Cost: 2.577880 Iter: 1990 Weight: 0.333261 Bias: -0.257970 Cost: 2.577071 Iter: 1995 Weight: 0.333276 Bias: -0.258606 Cost: 2.576263 iters = np.arange(2000) plt.plot(iters, cost_history) [<matplotlib.lines.Line2D at 0x7fb27cf937f0>]We can see, based on the initial weight=0,bias=0, learning_rate=0.0001,the cost value decrease sharply.
sales_predict = weight_final*radio_sample + bias_final plt.scatter(radio_sample, sales_sample, alpha=0.5) plt.plot(radio_sample, sales_predict, c="r") plt.xlabel("radio") plt.ylabel("sales") Text(0, 0.5, 'sales')Experiment 2
In the last experiment, we set four hyperparameters manually which are w, b, l and i. Let’s change them and see what happen.
weight_final, bias_final, cost_history = train(radio_sample, sales_sample, -1000, 0, 0.0001, 2000) Iter: 0 Weight: -753.190221 Bias: 6.091610 Cost: 700169768.718390 Iter: 5 Weight: -182.698756 Bias: 20.168463 Cost: 41100789.097582 Iter: 10 Weight: -44.478396 Bias: 23.573870 Cost: 2412831.778656 Iter: 15 Weight: -10.989863 Bias: 24.393771 Cost: 141812.898689 Iter: 20 Weight: -2.876041 Bias: 24.587251 Cost: 8501.907289 Iter: 25 Weight: -0.910075 Bias: 24.628960 Cost: 676.341469 Iter: 30 Weight: -0.433628 Bias: 24.633899 Cost: 216.885960 Iter: 35 Weight: -0.318065 Bias: 24.629931 Cost: 189.827967 Iter: 40 Weight: -0.289939 Bias: 24.623806 Cost: 188.152151 Iter: 45 Weight: -0.282997 Bias: 24.617160 Cost: 187.966335 Iter: 50 Weight: -0.281188 Bias: 24.610389 Cost: 187.868027 Iter: 55 Weight: -0.280622 Bias: 24.603589 Cost: 187.774899 Iter: 60 Weight: -0.280358 Bias: 24.596783 Cost: 187.682118 Iter: 65 Weight: -0.280166 Bias: 24.589976 Cost: 187.589402 Iter: 70 Weight: -0.279993 Bias: 24.583172 Cost: 187.496732 Iter: 75 Weight: -0.279823 Bias: 24.576368 Cost: 187.404109 Iter: 80 Weight: -0.279655 Bias: 24.569567 Cost: 187.311531 Iter: 85 Weight: -0.279487 Bias: 24.562767 Cost: 187.218999 Iter: 90 Weight: -0.279319 Bias: 24.555969 Cost: 187.126514 Iter: 95 Weight: -0.279152 Bias: 24.549172 Cost: 187.034074 Iter: 100 Weight: -0.278984 Bias: 24.542377 Cost: 186.941680 Iter: 105 Weight: -0.278816 Bias: 24.535584 Cost: 186.849332 Iter: 110 Weight: -0.278648 Bias: 24.528793 Cost: 186.757030 Iter: 115 Weight: -0.278481 Bias: 24.522003 Cost: 186.664774 Iter: 120 Weight: -0.278313 Bias: 24.515215 Cost: 186.572564 Iter: 125 Weight: -0.278146 Bias: 24.508428 Cost: 186.480399 Iter: 130 Weight: -0.277978 Bias: 24.501644 Cost: 186.388280 Iter: 135 Weight: -0.277811 Bias: 24.494861 Cost: 186.296207 Iter: 140 Weight: -0.277643 Bias: 24.488079 Cost: 186.204179 Iter: 145 Weight: -0.277476 Bias: 24.481300 Cost: 186.112198 Iter: 150 Weight: -0.277309 Bias: 24.474522 Cost: 186.020261 Iter: 155 Weight: -0.277141 Bias: 24.467745 Cost: 185.928371 Iter: 160 Weight: -0.276974 Bias: 24.460971 Cost: 185.836526 Iter: 165 Weight: -0.276807 Bias: 24.454198 Cost: 185.744727 Iter: 170 Weight: -0.276640 Bias: 24.447426 Cost: 185.652973 Iter: 175 Weight: -0.276473 Bias: 24.440657 Cost: 185.561265 Iter: 180 Weight: -0.276306 Bias: 24.433889 Cost: 185.469603 Iter: 185 Weight: -0.276138 Bias: 24.427123 Cost: 185.377986 Iter: 190 Weight: -0.275971 Bias: 24.420358 Cost: 185.286414 Iter: 195 Weight: -0.275805 Bias: 24.413595 Cost: 185.194888 Iter: 200 Weight: -0.275638 Bias: 24.406834 Cost: 185.103407 Iter: 205 Weight: -0.275471 Bias: 24.400075 Cost: 185.011972 Iter: 210 Weight: -0.275304 Bias: 24.393317 Cost: 184.920582 Iter: 215 Weight: -0.275137 Bias: 24.386561 Cost: 184.829238 Iter: 220 Weight: -0.274970 Bias: 24.379806 Cost: 184.737939 Iter: 225 Weight: -0.274804 Bias: 24.373053 Cost: 184.646685 Iter: 230 Weight: -0.274637 Bias: 24.366302 Cost: 184.555477 Iter: 235 Weight: -0.274470 Bias: 24.359553 Cost: 184.464314 Iter: 240 Weight: -0.274304 Bias: 24.352805 Cost: 184.373196 Iter: 245 Weight: -0.274137 Bias: 24.346059 Cost: 184.282123 Iter: 250 Weight: -0.273971 Bias: 24.339315 Cost: 184.191096 Iter: 255 Weight: -0.273804 Bias: 24.332572 Cost: 184.100113 Iter: 260 Weight: -0.273638 Bias: 24.325831 Cost: 184.009176 Iter: 265 Weight: -0.273472 Bias: 24.319091 Cost: 183.918284 Iter: 270 Weight: -0.273305 Bias: 24.312354 Cost: 183.827438 Iter: 275 Weight: -0.273139 Bias: 24.305618 Cost: 183.736636 Iter: 280 Weight: -0.272973 Bias: 24.298883 Cost: 183.645879 Iter: 285 Weight: -0.272806 Bias: 24.292150 Cost: 183.555168 Iter: 290 Weight: -0.272640 Bias: 24.285419 Cost: 183.464501 Iter: 295 Weight: -0.272474 Bias: 24.278690 Cost: 183.373880 Iter: 300 Weight: -0.272308 Bias: 24.271962 Cost: 183.283303 Iter: 305 Weight: -0.272142 Bias: 24.265236 Cost: 183.192772 Iter: 310 Weight: -0.271976 Bias: 24.258512 Cost: 183.102285 Iter: 315 Weight: -0.271810 Bias: 24.251789 Cost: 183.011844 Iter: 320 Weight: -0.271644 Bias: 24.245068 Cost: 182.921447 Iter: 325 Weight: -0.271478 Bias: 24.238349 Cost: 182.831095 Iter: 330 Weight: -0.271312 Bias: 24.231631 Cost: 182.740788 Iter: 335 Weight: -0.271147 Bias: 24.224915 Cost: 182.650526 Iter: 340 Weight: -0.270981 Bias: 24.218201 Cost: 182.560309 Iter: 345 Weight: -0.270815 Bias: 24.211488 Cost: 182.470136 Iter: 350 Weight: -0.270649 Bias: 24.204777 Cost: 182.380008 Iter: 355 Weight: -0.270484 Bias: 24.198068 Cost: 182.289925 Iter: 360 Weight: -0.270318 Bias: 24.191360 Cost: 182.199887 Iter: 365 Weight: -0.270153 Bias: 24.184654 Cost: 182.109893 Iter: 370 Weight: -0.269987 Bias: 24.177950 Cost: 182.019945 Iter: 375 Weight: -0.269822 Bias: 24.171247 Cost: 181.930040 Iter: 380 Weight: -0.269656 Bias: 24.164546 Cost: 181.840181 Iter: 385 Weight: -0.269491 Bias: 24.157847 Cost: 181.750366 Iter: 390 Weight: -0.269326 Bias: 24.151149 Cost: 181.660595 Iter: 395 Weight: -0.269160 Bias: 24.144453 Cost: 181.570870 Iter: 400 Weight: -0.268995 Bias: 24.137759 Cost: 181.481188 Iter: 405 Weight: -0.268830 Bias: 24.131066 Cost: 181.391552 Iter: 410 Weight: -0.268665 Bias: 24.124375 Cost: 181.301959 Iter: 415 Weight: -0.268499 Bias: 24.117686 Cost: 181.212412 Iter: 420 Weight: -0.268334 Bias: 24.110998 Cost: 181.122908 Iter: 425 Weight: -0.268169 Bias: 24.104312 Cost: 181.033450 Iter: 430 Weight: -0.268004 Bias: 24.097628 Cost: 180.944035 Iter: 435 Weight: -0.267839 Bias: 24.090945 Cost: 180.854665 Iter: 440 Weight: -0.267674 Bias: 24.084264 Cost: 180.765339 Iter: 445 Weight: -0.267509 Bias: 24.077584 Cost: 180.676058 Iter: 450 Weight: -0.267345 Bias: 24.070907 Cost: 180.586821 Iter: 455 Weight: -0.267180 Bias: 24.064231 Cost: 180.497629 Iter: 460 Weight: -0.267015 Bias: 24.057556 Cost: 180.408480 Iter: 465 Weight: -0.266850 Bias: 24.050883 Cost: 180.319376 Iter: 470 Weight: -0.266686 Bias: 24.044212 Cost: 180.230316 Iter: 475 Weight: -0.266521 Bias: 24.037543 Cost: 180.141300 Iter: 480 Weight: -0.266356 Bias: 24.030875 Cost: 180.052329 Iter: 485 Weight: -0.266192 Bias: 24.024209 Cost: 179.963402 Iter: 490 Weight: -0.266027 Bias: 24.017544 Cost: 179.874519 Iter: 495 Weight: -0.265863 Bias: 24.010881 Cost: 179.785680 Iter: 500 Weight: -0.265698 Bias: 24.004220 Cost: 179.696885 Iter: 505 Weight: -0.265534 Bias: 23.997561 Cost: 179.608134 Iter: 510 Weight: -0.265370 Bias: 23.990903 Cost: 179.519427 Iter: 515 Weight: -0.265205 Bias: 23.984247 Cost: 179.430765 Iter: 520 Weight: -0.265041 Bias: 23.977592 Cost: 179.342146 Iter: 525 Weight: -0.264877 Bias: 23.970939 Cost: 179.253571 Iter: 530 Weight: -0.264713 Bias: 23.964288 Cost: 179.165041 Iter: 535 Weight: -0.264548 Bias: 23.957638 Cost: 179.076554 Iter: 540 Weight: -0.264384 Bias: 23.950990 Cost: 178.988111 Iter: 545 Weight: -0.264220 Bias: 23.944344 Cost: 178.899712 Iter: 550 Weight: -0.264056 Bias: 23.937699 Cost: 178.811357 Iter: 555 Weight: -0.263892 Bias: 23.931056 Cost: 178.723046 Iter: 560 Weight: -0.263728 Bias: 23.924415 Cost: 178.634779 Iter: 565 Weight: -0.263564 Bias: 23.917775 Cost: 178.546555 Iter: 570 Weight: -0.263400 Bias: 23.911137 Cost: 178.458376 Iter: 575 Weight: -0.263237 Bias: 23.904501 Cost: 178.370240 Iter: 580 Weight: -0.263073 Bias: 23.897866 Cost: 178.282148 Iter: 585 Weight: -0.262909 Bias: 23.891233 Cost: 178.194099 Iter: 590 Weight: -0.262745 Bias: 23.884601 Cost: 178.106095 Iter: 595 Weight: -0.262582 Bias: 23.877971 Cost: 178.018134 Iter: 600 Weight: -0.262418 Bias: 23.871343 Cost: 177.930216 Iter: 605 Weight: -0.262254 Bias: 23.864717 Cost: 177.842343 Iter: 610 Weight: -0.262091 Bias: 23.858092 Cost: 177.754513 Iter: 615 Weight: -0.261927 Bias: 23.851468 Cost: 177.666726 Iter: 620 Weight: -0.261764 Bias: 23.844847 Cost: 177.578984 Iter: 625 Weight: -0.261600 Bias: 23.838227 Cost: 177.491284 Iter: 630 Weight: -0.261437 Bias: 23.831609 Cost: 177.403629 Iter: 635 Weight: -0.261274 Bias: 23.824992 Cost: 177.316017 Iter: 640 Weight: -0.261110 Bias: 23.818377 Cost: 177.228448 Iter: 645 Weight: -0.260947 Bias: 23.811763 Cost: 177.140923 Iter: 650 Weight: -0.260784 Bias: 23.805152 Cost: 177.053441 Iter: 655 Weight: -0.260621 Bias: 23.798542 Cost: 176.966003 Iter: 660 Weight: -0.260458 Bias: 23.791933 Cost: 176.878608 Iter: 665 Weight: -0.260295 Bias: 23.785326 Cost: 176.791256 Iter: 670 Weight: -0.260131 Bias: 23.778721 Cost: 176.703948 Iter: 675 Weight: -0.259968 Bias: 23.772117 Cost: 176.616683 Iter: 680 Weight: -0.259805 Bias: 23.765516 Cost: 176.529462 Iter: 685 Weight: -0.259642 Bias: 23.758915 Cost: 176.442284 Iter: 690 Weight: -0.259480 Bias: 23.752317 Cost: 176.355149 Iter: 695 Weight: -0.259317 Bias: 23.745720 Cost: 176.268058 Iter: 700 Weight: -0.259154 Bias: 23.739124 Cost: 176.181009 Iter: 705 Weight: -0.258991 Bias: 23.732531 Cost: 176.094004 Iter: 710 Weight: -0.258828 Bias: 23.725939 Cost: 176.007042 Iter: 715 Weight: -0.258666 Bias: 23.719348 Cost: 175.920123 Iter: 720 Weight: -0.258503 Bias: 23.712759 Cost: 175.833248 Iter: 725 Weight: -0.258340 Bias: 23.706172 Cost: 175.746415 Iter: 730 Weight: -0.258178 Bias: 23.699587 Cost: 175.659626 Iter: 735 Weight: -0.258015 Bias: 23.693003 Cost: 175.572880 Iter: 740 Weight: -0.257853 Bias: 23.686420 Cost: 175.486177 Iter: 745 Weight: -0.257690 Bias: 23.679840 Cost: 175.399517 Iter: 750 Weight: -0.257528 Bias: 23.673261 Cost: 175.312899 Iter: 755 Weight: -0.257366 Bias: 23.666683 Cost: 175.226325 Iter: 760 Weight: -0.257203 Bias: 23.660108 Cost: 175.139794 Iter: 765 Weight: -0.257041 Bias: 23.653534 Cost: 175.053306 Iter: 770 Weight: -0.256879 Bias: 23.646961 Cost: 174.966861 Iter: 775 Weight: -0.256716 Bias: 23.640390 Cost: 174.880459 Iter: 780 Weight: -0.256554 Bias: 23.633821 Cost: 174.794099 Iter: 785 Weight: -0.256392 Bias: 23.627253 Cost: 174.707783 Iter: 790 Weight: -0.256230 Bias: 23.620688 Cost: 174.621509 Iter: 795 Weight: -0.256068 Bias: 23.614123 Cost: 174.535278 Iter: 800 Weight: -0.255906 Bias: 23.607561 Cost: 174.449090 Iter: 805 Weight: -0.255744 Bias: 23.600999 Cost: 174.362945 Iter: 810 Weight: -0.255582 Bias: 23.594440 Cost: 174.276843 Iter: 815 Weight: -0.255420 Bias: 23.587882 Cost: 174.190783 Iter: 820 Weight: -0.255258 Bias: 23.581326 Cost: 174.104766 Iter: 825 Weight: -0.255097 Bias: 23.574772 Cost: 174.018792 Iter: 830 Weight: -0.254935 Bias: 23.568219 Cost: 173.932860 Iter: 835 Weight: -0.254773 Bias: 23.561667 Cost: 173.846971 Iter: 840 Weight: -0.254611 Bias: 23.555118 Cost: 173.761125 Iter: 845 Weight: -0.254450 Bias: 23.548570 Cost: 173.675322 Iter: 850 Weight: -0.254288 Bias: 23.542023 Cost: 173.589561 Iter: 855 Weight: -0.254127 Bias: 23.535478 Cost: 173.503842 Iter: 860 Weight: -0.253965 Bias: 23.528935 Cost: 173.418166 Iter: 865 Weight: -0.253804 Bias: 23.522394 Cost: 173.332533 Iter: 870 Weight: -0.253642 Bias: 23.515854 Cost: 173.246942 Iter: 875 Weight: -0.253481 Bias: 23.509316 Cost: 173.161394 Iter: 880 Weight: -0.253319 Bias: 23.502779 Cost: 173.075888 Iter: 885 Weight: -0.253158 Bias: 23.496244 Cost: 172.990425 Iter: 890 Weight: -0.252997 Bias: 23.489710 Cost: 172.905004 Iter: 895 Weight: -0.252835 Bias: 23.483179 Cost: 172.819625 Iter: 900 Weight: -0.252674 Bias: 23.476648 Cost: 172.734289 Iter: 905 Weight: -0.252513 Bias: 23.470120 Cost: 172.648995 Iter: 910 Weight: -0.252352 Bias: 23.463593 Cost: 172.563744 Iter: 915 Weight: -0.252191 Bias: 23.457068 Cost: 172.478535 Iter: 920 Weight: -0.252030 Bias: 23.450544 Cost: 172.393368 Iter: 925 Weight: -0.251869 Bias: 23.444022 Cost: 172.308244 Iter: 930 Weight: -0.251708 Bias: 23.437501 Cost: 172.223161 Iter: 935 Weight: -0.251547 Bias: 23.430983 Cost: 172.138121 Iter: 940 Weight: -0.251386 Bias: 23.424465 Cost: 172.053124 Iter: 945 Weight: -0.251225 Bias: 23.417950 Cost: 171.968168 Iter: 950 Weight: -0.251064 Bias: 23.411436 Cost: 171.883255 Iter: 955 Weight: -0.250904 Bias: 23.404924 Cost: 171.798383 Iter: 960 Weight: -0.250743 Bias: 23.398413 Cost: 171.713554 Iter: 965 Weight: -0.250582 Bias: 23.391904 Cost: 171.628767 Iter: 970 Weight: -0.250421 Bias: 23.385396 Cost: 171.544022 Iter: 975 Weight: -0.250261 Bias: 23.378890 Cost: 171.459320 Iter: 980 Weight: -0.250100 Bias: 23.372386 Cost: 171.374659 Iter: 985 Weight: -0.249940 Bias: 23.365883 Cost: 171.290040 Iter: 990 Weight: -0.249779 Bias: 23.359382 Cost: 171.205464 Iter: 995 Weight: -0.249619 Bias: 23.352883 Cost: 171.120929 Iter: 1000 Weight: -0.249458 Bias: 23.346385 Cost: 171.036436 Iter: 1005 Weight: -0.249298 Bias: 23.339889 Cost: 170.951985 Iter: 1010 Weight: -0.249138 Bias: 23.333394 Cost: 170.867577 Iter: 1015 Weight: -0.248977 Bias: 23.326901 Cost: 170.783210 Iter: 1020 Weight: -0.248817 Bias: 23.320410 Cost: 170.698885 Iter: 1025 Weight: -0.248657 Bias: 23.313920 Cost: 170.614602 Iter: 1030 Weight: -0.248497 Bias: 23.307432 Cost: 170.530360 Iter: 1035 Weight: -0.248337 Bias: 23.300945 Cost: 170.446161 Iter: 1040 Weight: -0.248177 Bias: 23.294460 Cost: 170.362003 Iter: 1045 Weight: -0.248016 Bias: 23.287977 Cost: 170.277887 Iter: 1050 Weight: -0.247856 Bias: 23.281495 Cost: 170.193813 Iter: 1055 Weight: -0.247696 Bias: 23.275015 Cost: 170.109781 Iter: 1060 Weight: -0.247537 Bias: 23.268537 Cost: 170.025790 Iter: 1065 Weight: -0.247377 Bias: 23.262060 Cost: 169.941841 Iter: 1070 Weight: -0.247217 Bias: 23.255585 Cost: 169.857934 Iter: 1075 Weight: -0.247057 Bias: 23.249111 Cost: 169.774068 Iter: 1080 Weight: -0.246897 Bias: 23.242639 Cost: 169.690244 Iter: 1085 Weight: -0.246737 Bias: 23.236169 Cost: 169.606462 Iter: 1090 Weight: -0.246578 Bias: 23.229700 Cost: 169.522721 Iter: 1095 Weight: -0.246418 Bias: 23.223232 Cost: 169.439022 Iter: 1100 Weight: -0.246259 Bias: 23.216767 Cost: 169.355364 Iter: 1105 Weight: -0.246099 Bias: 23.210303 Cost: 169.271748 Iter: 1110 Weight: -0.245939 Bias: 23.203840 Cost: 169.188174 Iter: 1115 Weight: -0.245780 Bias: 23.197380 Cost: 169.104640 Iter: 1120 Weight: -0.245620 Bias: 23.190920 Cost: 169.021149 Iter: 1125 Weight: -0.245461 Bias: 23.184463 Cost: 168.937699 Iter: 1130 Weight: -0.245302 Bias: 23.178007 Cost: 168.854290 Iter: 1135 Weight: -0.245142 Bias: 23.171552 Cost: 168.770923 Iter: 1140 Weight: -0.244983 Bias: 23.165100 Cost: 168.687597 Iter: 1145 Weight: -0.244824 Bias: 23.158648 Cost: 168.604312 Iter: 1150 Weight: -0.244665 Bias: 23.152199 Cost: 168.521069 Iter: 1155 Weight: -0.244505 Bias: 23.145751 Cost: 168.437867 Iter: 1160 Weight: -0.244346 Bias: 23.139304 Cost: 168.354707 Iter: 1165 Weight: -0.244187 Bias: 23.132860 Cost: 168.271588 Iter: 1170 Weight: -0.244028 Bias: 23.126416 Cost: 168.188510 Iter: 1175 Weight: -0.243869 Bias: 23.119975 Cost: 168.105473 Iter: 1180 Weight: -0.243710 Bias: 23.113535 Cost: 168.022477 Iter: 1185 Weight: -0.243551 Bias: 23.107096 Cost: 167.939523 Iter: 1190 Weight: -0.243392 Bias: 23.100660 Cost: 167.856610 Iter: 1195 Weight: -0.243233 Bias: 23.094224 Cost: 167.773738 Iter: 1200 Weight: -0.243074 Bias: 23.087791 Cost: 167.690907 Iter: 1205 Weight: -0.242916 Bias: 23.081359 Cost: 167.608118 Iter: 1210 Weight: -0.242757 Bias: 23.074928 Cost: 167.525369 Iter: 1215 Weight: -0.242598 Bias: 23.068500 Cost: 167.442662 Iter: 1220 Weight: -0.242440 Bias: 23.062072 Cost: 167.359995 Iter: 1225 Weight: -0.242281 Bias: 23.055647 Cost: 167.277370 Iter: 1230 Weight: -0.242122 Bias: 23.049223 Cost: 167.194785 Iter: 1235 Weight: -0.241964 Bias: 23.042800 Cost: 167.112242 Iter: 1240 Weight: -0.241805 Bias: 23.036380 Cost: 167.029740 Iter: 1245 Weight: -0.241647 Bias: 23.029960 Cost: 166.947278 Iter: 1250 Weight: -0.241488 Bias: 23.023543 Cost: 166.864858 Iter: 1255 Weight: -0.241330 Bias: 23.017127 Cost: 166.782478 Iter: 1260 Weight: -0.241172 Bias: 23.010712 Cost: 166.700140 Iter: 1265 Weight: -0.241013 Bias: 23.004299 Cost: 166.617842 Iter: 1270 Weight: -0.240855 Bias: 22.997888 Cost: 166.535585 Iter: 1275 Weight: -0.240697 Bias: 22.991478 Cost: 166.453369 Iter: 1280 Weight: -0.240539 Bias: 22.985070 Cost: 166.371194 Iter: 1285 Weight: -0.240380 Bias: 22.978664 Cost: 166.289060 Iter: 1290 Weight: -0.240222 Bias: 22.972259 Cost: 166.206966 Iter: 1295 Weight: -0.240064 Bias: 22.965856 Cost: 166.124913 Iter: 1300 Weight: -0.239906 Bias: 22.959454 Cost: 166.042901 Iter: 1305 Weight: -0.239748 Bias: 22.953054 Cost: 165.960930 Iter: 1310 Weight: -0.239590 Bias: 22.946655 Cost: 165.878999 Iter: 1315 Weight: -0.239432 Bias: 22.940258 Cost: 165.797109 Iter: 1320 Weight: -0.239274 Bias: 22.933863 Cost: 165.715260 Iter: 1325 Weight: -0.239117 Bias: 22.927469 Cost: 165.633451 Iter: 1330 Weight: -0.238959 Bias: 22.921077 Cost: 165.551683 Iter: 1335 Weight: -0.238801 Bias: 22.914686 Cost: 165.469955 Iter: 1340 Weight: -0.238643 Bias: 22.908297 Cost: 165.388268 Iter: 1345 Weight: -0.238486 Bias: 22.901910 Cost: 165.306622 Iter: 1350 Weight: -0.238328 Bias: 22.895524 Cost: 165.225016 Iter: 1355 Weight: -0.238170 Bias: 22.889140 Cost: 165.143451 Iter: 1360 Weight: -0.238013 Bias: 22.882757 Cost: 165.061926 Iter: 1365 Weight: -0.237855 Bias: 22.876376 Cost: 164.980442 Iter: 1370 Weight: -0.237698 Bias: 22.869996 Cost: 164.898998 Iter: 1375 Weight: -0.237540 Bias: 22.863618 Cost: 164.817594 Iter: 1380 Weight: -0.237383 Bias: 22.857242 Cost: 164.736231 Iter: 1385 Weight: -0.237226 Bias: 22.850867 Cost: 164.654909 Iter: 1390 Weight: -0.237068 Bias: 22.844494 Cost: 164.573627 Iter: 1395 Weight: -0.236911 Bias: 22.838123 Cost: 164.492385 Iter: 1400 Weight: -0.236754 Bias: 22.831753 Cost: 164.411183 Iter: 1405 Weight: -0.236596 Bias: 22.825384 Cost: 164.330022 Iter: 1410 Weight: -0.236439 Bias: 22.819017 Cost: 164.248901 Iter: 1415 Weight: -0.236282 Bias: 22.812652 Cost: 164.167820 Iter: 1420 Weight: -0.236125 Bias: 22.806288 Cost: 164.086780 Iter: 1425 Weight: -0.235968 Bias: 22.799926 Cost: 164.005780 Iter: 1430 Weight: -0.235811 Bias: 22.793566 Cost: 163.924820 Iter: 1435 Weight: -0.235654 Bias: 22.787207 Cost: 163.843900 Iter: 1440 Weight: -0.235497 Bias: 22.780849 Cost: 163.763020 Iter: 1445 Weight: -0.235340 Bias: 22.774494 Cost: 163.682181 Iter: 1450 Weight: -0.235183 Bias: 22.768139 Cost: 163.601382 Iter: 1455 Weight: -0.235026 Bias: 22.761787 Cost: 163.520623 Iter: 1460 Weight: -0.234870 Bias: 22.755436 Cost: 163.439904 Iter: 1465 Weight: -0.234713 Bias: 22.749086 Cost: 163.359225 Iter: 1470 Weight: -0.234556 Bias: 22.742738 Cost: 163.278586 Iter: 1475 Weight: -0.234400 Bias: 22.736392 Cost: 163.197987 Iter: 1480 Weight: -0.234243 Bias: 22.730047 Cost: 163.117428 Iter: 1485 Weight: -0.234086 Bias: 22.723704 Cost: 163.036909 Iter: 1490 Weight: -0.233930 Bias: 22.717362 Cost: 162.956430 Iter: 1495 Weight: -0.233773 Bias: 22.711022 Cost: 162.875991 Iter: 1500 Weight: -0.233617 Bias: 22.704684 Cost: 162.795592 Iter: 1505 Weight: -0.233460 Bias: 22.698347 Cost: 162.715233 Iter: 1510 Weight: -0.233304 Bias: 22.692012 Cost: 162.634914 Iter: 1515 Weight: -0.233148 Bias: 22.685678 Cost: 162.554635 Iter: 1520 Weight: -0.232991 Bias: 22.679346 Cost: 162.474395 Iter: 1525 Weight: -0.232835 Bias: 22.673015 Cost: 162.394196 Iter: 1530 Weight: -0.232679 Bias: 22.666686 Cost: 162.314036 Iter: 1535 Weight: -0.232523 Bias: 22.660359 Cost: 162.233916 Iter: 1540 Weight: -0.232366 Bias: 22.654033 Cost: 162.153836 Iter: 1545 Weight: -0.232210 Bias: 22.647709 Cost: 162.073795 Iter: 1550 Weight: -0.232054 Bias: 22.641386 Cost: 161.993795 Iter: 1555 Weight: -0.231898 Bias: 22.635065 Cost: 161.913834 Iter: 1560 Weight: -0.231742 Bias: 22.628745 Cost: 161.833913 Iter: 1565 Weight: -0.231586 Bias: 22.622427 Cost: 161.754031 Iter: 1570 Weight: -0.231430 Bias: 22.616111 Cost: 161.674189 Iter: 1575 Weight: -0.231274 Bias: 22.609796 Cost: 161.594387 Iter: 1580 Weight: -0.231118 Bias: 22.603482 Cost: 161.514624 Iter: 1585 Weight: -0.230963 Bias: 22.597171 Cost: 161.434901 Iter: 1590 Weight: -0.230807 Bias: 22.590860 Cost: 161.355218 Iter: 1595 Weight: -0.230651 Bias: 22.584552 Cost: 161.275574 Iter: 1600 Weight: -0.230495 Bias: 22.578245 Cost: 161.195969 Iter: 1605 Weight: -0.230340 Bias: 22.571939 Cost: 161.116405 Iter: 1610 Weight: -0.230184 Bias: 22.565635 Cost: 161.036879 Iter: 1615 Weight: -0.230028 Bias: 22.559333 Cost: 160.957393 Iter: 1620 Weight: -0.229873 Bias: 22.553032 Cost: 160.877947 Iter: 1625 Weight: -0.229717 Bias: 22.546733 Cost: 160.798540 Iter: 1630 Weight: -0.229562 Bias: 22.540435 Cost: 160.719173 Iter: 1635 Weight: -0.229407 Bias: 22.534139 Cost: 160.639845 Iter: 1640 Weight: -0.229251 Bias: 22.527845 Cost: 160.560556 Iter: 1645 Weight: -0.229096 Bias: 22.521552 Cost: 160.481307 Iter: 1650 Weight: -0.228940 Bias: 22.515260 Cost: 160.402097 Iter: 1655 Weight: -0.228785 Bias: 22.508970 Cost: 160.322926 Iter: 1660 Weight: -0.228630 Bias: 22.502682 Cost: 160.243795 Iter: 1665 Weight: -0.228475 Bias: 22.496395 Cost: 160.164702 Iter: 1670 Weight: -0.228320 Bias: 22.490110 Cost: 160.085650 Iter: 1675 Weight: -0.228164 Bias: 22.483827 Cost: 160.006636 Iter: 1680 Weight: -0.228009 Bias: 22.477545 Cost: 159.927662 Iter: 1685 Weight: -0.227854 Bias: 22.471264 Cost: 159.848727 Iter: 1690 Weight: -0.227699 Bias: 22.464985 Cost: 159.769831 Iter: 1695 Weight: -0.227544 Bias: 22.458708 Cost: 159.690974 Iter: 1700 Weight: -0.227389 Bias: 22.452432 Cost: 159.612157 Iter: 1705 Weight: -0.227235 Bias: 22.446158 Cost: 159.533378 Iter: 1710 Weight: -0.227080 Bias: 22.439885 Cost: 159.454639 Iter: 1715 Weight: -0.226925 Bias: 22.433614 Cost: 159.375939 Iter: 1720 Weight: -0.226770 Bias: 22.427344 Cost: 159.297277 Iter: 1725 Weight: -0.226615 Bias: 22.421076 Cost: 159.218655 Iter: 1730 Weight: -0.226461 Bias: 22.414810 Cost: 159.140072 Iter: 1735 Weight: -0.226306 Bias: 22.408545 Cost: 159.061528 Iter: 1740 Weight: -0.226151 Bias: 22.402282 Cost: 158.983023 Iter: 1745 Weight: -0.225997 Bias: 22.396020 Cost: 158.904557 Iter: 1750 Weight: -0.225842 Bias: 22.389760 Cost: 158.826130 Iter: 1755 Weight: -0.225688 Bias: 22.383501 Cost: 158.747742 Iter: 1760 Weight: -0.225533 Bias: 22.377244 Cost: 158.669393 Iter: 1765 Weight: -0.225379 Bias: 22.370988 Cost: 158.591082 Iter: 1770 Weight: -0.225224 Bias: 22.364734 Cost: 158.512811 Iter: 1775 Weight: -0.225070 Bias: 22.358482 Cost: 158.434578 Iter: 1780 Weight: -0.224916 Bias: 22.352231 Cost: 158.356384 Iter: 1785 Weight: -0.224761 Bias: 22.345981 Cost: 158.278229 Iter: 1790 Weight: -0.224607 Bias: 22.339734 Cost: 158.200113 Iter: 1795 Weight: -0.224453 Bias: 22.333487 Cost: 158.122036 Iter: 1800 Weight: -0.224299 Bias: 22.327243 Cost: 158.043997 Iter: 1805 Weight: -0.224145 Bias: 22.320999 Cost: 157.965998 Iter: 1810 Weight: -0.223991 Bias: 22.314758 Cost: 157.888036 Iter: 1815 Weight: -0.223837 Bias: 22.308518 Cost: 157.810114 Iter: 1820 Weight: -0.223683 Bias: 22.302279 Cost: 157.732230 Iter: 1825 Weight: -0.223529 Bias: 22.296042 Cost: 157.654385 Iter: 1830 Weight: -0.223375 Bias: 22.289807 Cost: 157.576579 Iter: 1835 Weight: -0.223221 Bias: 22.283573 Cost: 157.498811 Iter: 1840 Weight: -0.223067 Bias: 22.277341 Cost: 157.421082 Iter: 1845 Weight: -0.222913 Bias: 22.271110 Cost: 157.343392 Iter: 1850 Weight: -0.222759 Bias: 22.264881 Cost: 157.265740 Iter: 1855 Weight: -0.222606 Bias: 22.258653 Cost: 157.188126 Iter: 1860 Weight: -0.222452 Bias: 22.252427 Cost: 157.110551 Iter: 1865 Weight: -0.222298 Bias: 22.246202 Cost: 157.033015 Iter: 1870 Weight: -0.222145 Bias: 22.239979 Cost: 156.955517 Iter: 1875 Weight: -0.221991 Bias: 22.233758 Cost: 156.878058 Iter: 1880 Weight: -0.221837 Bias: 22.227538 Cost: 156.800637 Iter: 1885 Weight: -0.221684 Bias: 22.221319 Cost: 156.723254 Iter: 1890 Weight: -0.221530 Bias: 22.215102 Cost: 156.645910 Iter: 1895 Weight: -0.221377 Bias: 22.208887 Cost: 156.568605 Iter: 1900 Weight: -0.221224 Bias: 22.202673 Cost: 156.491337 Iter: 1905 Weight: -0.221070 Bias: 22.196461 Cost: 156.414108 Iter: 1910 Weight: -0.220917 Bias: 22.190250 Cost: 156.336918 Iter: 1915 Weight: -0.220764 Bias: 22.184041 Cost: 156.259766 Iter: 1920 Weight: -0.220610 Bias: 22.177834 Cost: 156.182652 Iter: 1925 Weight: -0.220457 Bias: 22.171628 Cost: 156.105576 Iter: 1930 Weight: -0.220304 Bias: 22.165423 Cost: 156.028539 Iter: 1935 Weight: -0.220151 Bias: 22.159220 Cost: 155.951540 Iter: 1940 Weight: -0.219998 Bias: 22.153019 Cost: 155.874579 Iter: 1945 Weight: -0.219845 Bias: 22.146819 Cost: 155.797656 Iter: 1950 Weight: -0.219692 Bias: 22.140620 Cost: 155.720771 Iter: 1955 Weight: -0.219539 Bias: 22.134424 Cost: 155.643925 Iter: 1960 Weight: -0.219386 Bias: 22.128228 Cost: 155.567117 Iter: 1965 Weight: -0.219233 Bias: 22.122034 Cost: 155.490347 Iter: 1970 Weight: -0.219080 Bias: 22.115842 Cost: 155.413615 Iter: 1975 Weight: -0.218927 Bias: 22.109652 Cost: 155.336921 Iter: 1980 Weight: -0.218774 Bias: 22.103462 Cost: 155.260266 Iter: 1985 Weight: -0.218622 Bias: 22.097275 Cost: 155.183648 Iter: 1990 Weight: -0.218469 Bias: 22.091089 Cost: 155.107068 Iter: 1995 Weight: -0.218316 Bias: 22.084904 Cost: 155.030527 iters = np.arange(2000) # because the derivation of cost is too large # for visulazation, we need to shrink cost cost_history = np.log(np.array(cost_history)) plt.plot(iters, cost_history) [<matplotlib.lines.Line2D at 0x7fb27c9b99e8>] sales_predict = weight_final*radio_sample + bias_final plt.scatter(radio_sample, sales_sample, alpha=0.5) plt.plot(radio_sample, sales_predict, c="r") plt.xlabel("radio") plt.ylabel("sales") Text(0, 0.5, 'sales')The setting of hyperparameters is very import for gradient descent method. There must be lots of tricks which is another topic.
Task 2
Single feature —> multiple features
scalar —> matrix(vector)
Generate data
sample_size = 200 feature_size = 3 np.random.seed(5) feature_sample = 50*np.random.rand(feature_size)*np.random.rand(sample_size, feature_size)np.random.seed(123) weight_sample = np.random.rand(feature_size).reshape(feature_size, 1)np.random.seed(888) noise = np.random.normal(0, 0.01, sample_size).reshape(sample_size ,1)result_sample = np.dot(feature, weight) + noiseNormalization
We want to shrink the data to reduce the time to change weight. There must be lots of tricks which is another topic.
def normalize(feature):#feature: sample size * feature sizefeature = feature.astype("float64")sample_size, feature_size = feature.shapefor i in range(feature_size):fmean = np.mean(feature[:, i])frange = np.amax(feature[:, i]) - np.amin(feature[:, i])feature[:, i] = (feature[:, i] - fmean) / frangereturn featurePrediction format
For simplifing the problem, here we just think bias is zero.
Ysample?1=Fsample?feature?Wfeature?1Y_{sample*1}=F_{sample*feature} \cdot W_{feature*1}Ysample?1?=Fsample?feature??Wfeature?1?
def predict(features, weight):weight = weight.reshape(len(weight), 1)prediction = np.dot(features, weight)return predictionCost function
MSE=12N∥Ytarget?Ypredict∥22MSE=\frac{1}{2 N} \left\|Y_{target}-Y_{predict}\right\|_2^2MSE=2N1?∥Ytarget??Ypredict?∥22?
def cost_function(features, targets, weight):#here weights should be (feature, 1)weight = weight.reshape(len(weight), 1)targets = targets.reshape(len(targets), 1)error = ((targets - predict(features, weight))**2).sum()return error/(2.0*len(targets))Gradient descent
Just the equation derived in the watermelon book(3.11)
def update_weight(features, targets, weight, learning_rate):weight = weight.reshape(len(weight), 1)targets = targets.reshape(len(targets), 1)gradient = np.dot(features.T, (predict(features, weight) - targets))gradient = gradient / len(targets)weight = weight - gradient*learning_ratereturn weightTraining
def train(features, targets, weight, learning_rate, iters):cost_history = []for i in range(iters):weight = update_weight(features, targets, weight, learning_rate)cost = cost_function(features, targets, weight)cost_history.append(cost)if i % 5 == 0:print("Iter: %d \t \t Cost: %4f" %(i, cost))return weight, cost_historyModel evaluation
weight_initial = np.array([0, 0, 0]) weight, cost_history = train(feature_sample, result_sample, weight_initial, 0.0001, 1000) Iter: 0 Cost: 64.953149 Iter: 5 Cost: 33.915984 Iter: 10 Cost: 18.615390 Iter: 15 Cost: 11.053208 Iter: 20 Cost: 7.296763 Iter: 25 Cost: 5.412361 Iter: 30 Cost: 4.449195 Iter: 35 Cost: 3.939745 Iter: 40 Cost: 3.654115 Iter: 45 Cost: 3.479253 Iter: 50 Cost: 3.359556 Iter: 55 Cost: 3.267672 Iter: 60 Cost: 3.190135 Iter: 65 Cost: 3.120312 Iter: 70 Cost: 3.054928 Iter: 75 Cost: 2.992358 Iter: 80 Cost: 2.931792 Iter: 85 Cost: 2.872817 Iter: 90 Cost: 2.815220 Iter: 95 Cost: 2.758881 Iter: 100 Cost: 2.703732 Iter: 105 Cost: 2.649725 Iter: 110 Cost: 2.596827 Iter: 115 Cost: 2.545010 Iter: 120 Cost: 2.494250 Iter: 125 Cost: 2.444522 Iter: 130 Cost: 2.395806 Iter: 135 Cost: 2.348080 Iter: 140 Cost: 2.301324 Iter: 145 Cost: 2.255518 Iter: 150 Cost: 2.210643 Iter: 155 Cost: 2.166678 Iter: 160 Cost: 2.123606 Iter: 165 Cost: 2.081409 Iter: 170 Cost: 2.040067 Iter: 175 Cost: 1.999565 Iter: 180 Cost: 1.959884 Iter: 185 Cost: 1.921007 Iter: 190 Cost: 1.882919 Iter: 195 Cost: 1.845603 Iter: 200 Cost: 1.809044 Iter: 205 Cost: 1.773225 Iter: 210 Cost: 1.738131 Iter: 215 Cost: 1.703749 Iter: 220 Cost: 1.670062 Iter: 225 Cost: 1.637057 Iter: 230 Cost: 1.604721 Iter: 235 Cost: 1.573038 Iter: 240 Cost: 1.541997 Iter: 245 Cost: 1.511583 Iter: 250 Cost: 1.481784 Iter: 255 Cost: 1.452588 Iter: 260 Cost: 1.423981 Iter: 265 Cost: 1.395952 Iter: 270 Cost: 1.368490 Iter: 275 Cost: 1.341582 Iter: 280 Cost: 1.315217 Iter: 285 Cost: 1.289384 Iter: 290 Cost: 1.264073 Iter: 295 Cost: 1.239272 Iter: 300 Cost: 1.214971 Iter: 305 Cost: 1.191160 Iter: 310 Cost: 1.167828 Iter: 315 Cost: 1.144967 Iter: 320 Cost: 1.122566 Iter: 325 Cost: 1.100617 Iter: 330 Cost: 1.079109 Iter: 335 Cost: 1.058034 Iter: 340 Cost: 1.037383 Iter: 345 Cost: 1.017147 Iter: 350 Cost: 0.997318 Iter: 355 Cost: 0.977888 Iter: 360 Cost: 0.958848 Iter: 365 Cost: 0.940190 Iter: 370 Cost: 0.921908 Iter: 375 Cost: 0.903992 Iter: 380 Cost: 0.886435 Iter: 385 Cost: 0.869231 Iter: 390 Cost: 0.852372 Iter: 395 Cost: 0.835851 Iter: 400 Cost: 0.819660 Iter: 405 Cost: 0.803795 Iter: 410 Cost: 0.788246 Iter: 415 Cost: 0.773010 Iter: 420 Cost: 0.758078 Iter: 425 Cost: 0.743444 Iter: 430 Cost: 0.729104 Iter: 435 Cost: 0.715050 Iter: 440 Cost: 0.701277 Iter: 445 Cost: 0.687779 Iter: 450 Cost: 0.674550 Iter: 455 Cost: 0.661586 Iter: 460 Cost: 0.648880 Iter: 465 Cost: 0.636427 Iter: 470 Cost: 0.624223 Iter: 475 Cost: 0.612262 Iter: 480 Cost: 0.600540 Iter: 485 Cost: 0.589050 Iter: 490 Cost: 0.577790 Iter: 495 Cost: 0.566753 Iter: 500 Cost: 0.555936 Iter: 505 Cost: 0.545334 Iter: 510 Cost: 0.534943 Iter: 515 Cost: 0.524758 Iter: 520 Cost: 0.514775 Iter: 525 Cost: 0.504991 Iter: 530 Cost: 0.495400 Iter: 535 Cost: 0.486000 Iter: 540 Cost: 0.476786 Iter: 545 Cost: 0.467754 Iter: 550 Cost: 0.458901 Iter: 555 Cost: 0.450224 Iter: 560 Cost: 0.441718 Iter: 565 Cost: 0.433380 Iter: 570 Cost: 0.425207 Iter: 575 Cost: 0.417196 Iter: 580 Cost: 0.409342 Iter: 585 Cost: 0.401644 Iter: 590 Cost: 0.394097 Iter: 595 Cost: 0.386700 Iter: 600 Cost: 0.379448 Iter: 605 Cost: 0.372339 Iter: 610 Cost: 0.365369 Iter: 615 Cost: 0.358537 Iter: 620 Cost: 0.351839 Iter: 625 Cost: 0.345273 Iter: 630 Cost: 0.338836 Iter: 635 Cost: 0.332525 Iter: 640 Cost: 0.326338 Iter: 645 Cost: 0.320272 Iter: 650 Cost: 0.314326 Iter: 655 Cost: 0.308495 Iter: 660 Cost: 0.302779 Iter: 665 Cost: 0.297175 Iter: 670 Cost: 0.291680 Iter: 675 Cost: 0.286292 Iter: 680 Cost: 0.281010 Iter: 685 Cost: 0.275831 Iter: 690 Cost: 0.270753 Iter: 695 Cost: 0.265774 Iter: 700 Cost: 0.260892 Iter: 705 Cost: 0.256105 Iter: 710 Cost: 0.251411 Iter: 715 Cost: 0.246808 Iter: 720 Cost: 0.242295 Iter: 725 Cost: 0.237870 Iter: 730 Cost: 0.233530 Iter: 735 Cost: 0.229275 Iter: 740 Cost: 0.225102 Iter: 745 Cost: 0.221010 Iter: 750 Cost: 0.216997 Iter: 755 Cost: 0.213062 Iter: 760 Cost: 0.209202 Iter: 765 Cost: 0.205418 Iter: 770 Cost: 0.201706 Iter: 775 Cost: 0.198066 Iter: 780 Cost: 0.194496 Iter: 785 Cost: 0.190995 Iter: 790 Cost: 0.187561 Iter: 795 Cost: 0.184194 Iter: 800 Cost: 0.180891 Iter: 805 Cost: 0.177651 Iter: 810 Cost: 0.174474 Iter: 815 Cost: 0.171358 Iter: 820 Cost: 0.168301 Iter: 825 Cost: 0.165303 Iter: 830 Cost: 0.162362 Iter: 835 Cost: 0.159477 Iter: 840 Cost: 0.156648 Iter: 845 Cost: 0.153873 Iter: 850 Cost: 0.151150 Iter: 855 Cost: 0.148479 Iter: 860 Cost: 0.145860 Iter: 865 Cost: 0.143290 Iter: 870 Cost: 0.140768 Iter: 875 Cost: 0.138295 Iter: 880 Cost: 0.135869 Iter: 885 Cost: 0.133489 Iter: 890 Cost: 0.131153 Iter: 895 Cost: 0.128862 Iter: 900 Cost: 0.126615 Iter: 905 Cost: 0.124410 Iter: 910 Cost: 0.122246 Iter: 915 Cost: 0.120123 Iter: 920 Cost: 0.118040 Iter: 925 Cost: 0.115997 Iter: 930 Cost: 0.113992 Iter: 935 Cost: 0.112024 Iter: 940 Cost: 0.110094 Iter: 945 Cost: 0.108199 Iter: 950 Cost: 0.106340 Iter: 955 Cost: 0.104516 Iter: 960 Cost: 0.102726 Iter: 965 Cost: 0.100970 Iter: 970 Cost: 0.099246 Iter: 975 Cost: 0.097555 Iter: 980 Cost: 0.095895 Iter: 985 Cost: 0.094265 Iter: 990 Cost: 0.092667 Iter: 995 Cost: 0.091097 iters = np.arange(1000) plt.plot(iters, cost_history) [<matplotlib.lines.Line2D at 0x7fb27b4052b0>]由回歸到分類
In this task, I just use the watermelon dataset which is shown below.
Logistic Regression ( Binary Classification）的實際編碼
Dataset
import numpy as np import seaborn as sns import pandas as pd import matplotlib.pyplot as plt import math def createDataSet():"""創(chuàng)建測試的數(shù)據(jù)集，里面的數(shù)值中具有連續(xù)值:return:"""dataSet = [# 1['青綠', '蜷縮', '濁響', '清晰', '凹陷', '硬滑', 0.697, 0.460, '好瓜'],# 2['烏黑', '蜷縮', '沉悶', '清晰', '凹陷', '硬滑', 0.774, 0.376, '好瓜'],# 3['烏黑', '蜷縮', '濁響', '清晰', '凹陷', '硬滑', 0.634, 0.264, '好瓜'],# 4['青綠', '蜷縮', '沉悶', '清晰', '凹陷', '硬滑', 0.608, 0.318, '好瓜'],# 5['淺白', '蜷縮', '濁響', '清晰', '凹陷', '硬滑', 0.556, 0.215, '好瓜'],# 6['青綠', '稍蜷', '濁響', '清晰', '稍凹', '軟粘', 0.403, 0.237, '好瓜'],# 7['烏黑', '稍蜷', '濁響', '稍糊', '稍凹', '軟粘', 0.481, 0.149, '好瓜'],# 8['烏黑', '稍蜷', '濁響', '清晰', '稍凹', '硬滑', 0.437, 0.211, '好瓜'],# ----------------------------------------------------# 9['烏黑', '稍蜷', '沉悶', '稍糊', '稍凹', '硬滑', 0.666, 0.091, '壞瓜'],# 10['青綠', '硬挺', '清脆', '清晰', '平坦', '軟粘', 0.243, 0.267, '壞瓜'],# 11['淺白', '硬挺', '清脆', '模糊', '平坦', '硬滑', 0.245, 0.057, '壞瓜'],# 12['淺白', '蜷縮', '濁響', '模糊', '平坦', '軟粘', 0.343, 0.099, '壞瓜'],# 13['青綠', '稍蜷', '濁響', '稍糊', '凹陷', '硬滑', 0.639, 0.161, '壞瓜'],# 14['淺白', '稍蜷', '沉悶', '稍糊', '凹陷', '硬滑', 0.657, 0.198, '壞瓜'],# 15['烏黑', '稍蜷', '濁響', '清晰', '稍凹', '軟粘', 0.360, 0.370, '壞瓜'],# 16['淺白', '蜷縮', '濁響', '模糊', '平坦', '硬滑', 0.593, 0.042, '壞瓜'],# 17['青綠', '蜷縮', '沉悶', '稍糊', '稍凹', '硬滑', 0.719, 0.103, '壞瓜']]return dataSet dataSet = createDataSet() dataSet = np.array(dataSet)[:, 6:] dataSet[dataSet == '好瓜'] = 1 dataSet[dataSet == '壞瓜'] = 0 dataSet = dataSet.astype('float64') dataSet array([[0.697, 0.46 , 1. ],[0.774, 0.376, 1. ],[0.634, 0.264, 1. ],[0.608, 0.318, 1. ],[0.556, 0.215, 1. ],[0.403, 0.237, 1. ],[0.481, 0.149, 1. ],[0.437, 0.211, 1. ],[0.666, 0.091, 0. ],[0.243, 0.267, 0. ],[0.245, 0.057, 0. ],[0.343, 0.099, 0. ],[0.639, 0.161, 0. ],[0.657, 0.198, 0. ],[0.36 , 0.37 , 0. ],[0.593, 0.042, 0. ],[0.719, 0.103, 0. ]])Visulization
data_in_frame = pd.DataFrame(data=dataSet, columns=["density", "sugar_ratio","label"]) data_in_frame| 0.697 | 0.460 | 1.0 |
| 0.774 | 0.376 | 1.0 |
| 0.634 | 0.264 | 1.0 |
| 0.608 | 0.318 | 1.0 |
| 0.556 | 0.215 | 1.0 |
| 0.403 | 0.237 | 1.0 |
| 0.481 | 0.149 | 1.0 |
| 0.437 | 0.211 | 1.0 |
| 0.666 | 0.091 | 0.0 |
| 0.243 | 0.267 | 0.0 |
| 0.245 | 0.057 | 0.0 |
| 0.343 | 0.099 | 0.0 |
| 0.639 | 0.161 | 0.0 |
| 0.657 | 0.198 | 0.0 |
| 0.360 | 0.370 | 0.0 |
| 0.593 | 0.042 | 0.0 |
| 0.719 | 0.103 | 0.0 |
Prediction Format
\begin{split}p \geq 0.5, class=1 \
p < 0.5, class=0\end{split}
\begin{split}P(class=1) = \frac{1} {1 + e^{-z}}\end{split}
def sigmoid(x, derivative=False):sigm = 1. / (1. + np.exp(-x))if derivative:return sigm * (1. - sigm)return sigm def predict(features, weights):"""features: sample size * feature sizeweights: feature size * 1"""weights = weights.reshape(len(weights),1)z = np.dot(features, weights)return sigmoid(z)How could we do here? From Watermelon_book, we know that we can use MLE to estimate the parameters.
Meanwhile we can also repeat what we did in last experiment where we just included a cost function and optimized the patameters by decrease the cost function.
Cost function
We can still use MSE as a cost function. But here what we use is called cross-entroy function actually. The reason why we abandon the previous one is another topic. Basicly, it’s because of non-linear tranformation.
\begin{split}-{(y\log§ + (1 - y)\log(1 - p))}\end{split}
def cost_function(features, weights, labels):weights = weights.reshape(len(weights),1)labels = labels.reshape(len(labels),1)y = predict(features, weights)class1_cost = -labels*np.log(y)class2_cost = -labels*np.log(y)cost = (class1_cost + class2_cost).sum()/len(labels)return costGradient descent
\begin{align}
s’(z) & = s(z)(1 - s(z))
\end{align}
\begin{split}C’ = x(s(z) - y)\end{split}
def update_weights(features, weights, labels, learning_rate):weights = weights.reshape(len(weights),1)labels = labels.reshape(len(labels),1)p = predict(features, weights)gradient = np.dot(features.T, p - labels) / len(labels)weights = weights - gradient*learning_ratereturn weightsTraining
def training(features, weights, labels, learning_rate, iters):cost_history = []for i in range(iters):weights = update_weights(features, weights, labels, learning_rate)cost = cost_function(features, weights, labels)cost_history.append(cost)if i%1000 == 0:print("iters is %d \t \t cost is %f"%(i, cost))return weights, cost_history weights_initial = np.array([0, 0]) weights, cost_history = training(dataSet[:, :2], weights_initial, dataSet[:, -1], 0.1, 20000) iters is 0 cost is 0.651950 iters is 1000 cost is 0.572514 iters is 2000 cost is 0.546478 iters is 3000 cost is 0.528388 iters is 4000 cost is 0.515437 iters is 5000 cost is 0.505936 iters is 6000 cost is 0.498829 iters is 7000 cost is 0.493431 iters is 8000 cost is 0.489282 iters is 9000 cost is 0.486062 iters is 10000 cost is 0.483545 iters is 11000 cost is 0.481564 iters is 12000 cost is 0.479999 iters is 13000 cost is 0.478757 iters is 14000 cost is 0.477769 iters is 15000 cost is 0.476981 iters is 16000 cost is 0.476351 iters is 17000 cost is 0.475847 iters is 18000 cost is 0.475443 iters is 19000 cost is 0.475119Model evaluation
Visulization
iters = np.arange(20000) plt.plot(iters, cost_history) [<matplotlib.lines.Line2D at 0x7fc819b8eda0>]From the fig abobe, we can think the cost function may still decrease when we update the weights furthur.
Mapping probabilities to classes
def classify(predictions):predictions[predictions >= 0.5 ] = 1predictions[predictions < 0.5 ] = 0return predictions def accuracy(predictions, labels):predictions = predictions.astype('int').reshape(len(predictions,))labels = labels.astype('int')diff = np.abs(predictions - labels)same = len(labels) - diff.sum()return same/len(labels) predicted_label = classify(predict(dataSet[:, :2], weights)) accuracy(predicted_label, dataSet[:, -1]) 0.8235294117647058總結(jié)
以上是生活随笔為你收集整理的[Watermelon_book] Chapter 3 Linear Model的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 当 IDENTITY_INSERT 设置
- 下一篇： Linux软件包管理— rpm软件包查询