21.01.08. 머신러닝 - 선형모델학습

어제 수업한 파일에 이어서 수업 시작.

 

---

예측값

def h(w,x,b):
    return w * x + b

비용함수

• 평균제곱오차(MSE)

def cost(data, target, weight, intercept):              # MSE 구하기
#         시간, 점수, 가중치, 절편
    y_pre = h(weight,data,intercept)                    # 예측값
#   y_pre = weight * data + intercept
    return ((y_pre - target)**2).mean()                # 평균제곱오차(MSE)
#             (예측 - 실제) ← 오차
#     **는 제곱(^2)을 뜻함. mean()은 평균.

# 기울기 10, 절편 0
cost(data['시간'],data['성적'],10,0)

out : 0.0

# 기울기 12, 절편 5
cost(data['시간'],data['성적'],12,5)

out : 305.0

 

---

 

경사하강법 (Gradient Descent Algorithm)

• 비용함수의 기울기(경사)를 구하여 기울기가 낮은 쪽으로 계속 이동하여 값을 최적화 시키는 방법
• 어느 한 점에서 미분을 통해 기울기를 구하며 기울기가 0이 될 때까지 이동.

 

Learning rate(학습률)가 너무 커버리면 오히려 위쪽으로 벗어나고,

learning rate가 작으면 시간이 오래 걸리거나 국소지역점을 가장 작은 값으로 착각해서 최종결론을 내려버릴 수 있다.

 

---

 

import matplotlib.pyplot as plt

그래프를 그리기 위해 import

 

weight_arr = range(-10,31) # -10 ~ 30 ← w(가중치)로 활용
cost_list = []             # cost(평균제곱오차)값 담기
for w in weight_arr:
    c = cost(data['시간'],data['성적'],w,0)
    cost_list.append(c)
    
plt.plot(weight_arr, cost_list) # .plot 그래프 생성
plt.show() # .show 그래프를 보여줘라

out :

 

지금 우리가 입력한 코드대로면 국소지역점이 따로 발생하지 않는데요?

- 보기 편하게 정사각형으로 만들었다.

- 절편을 사용하지 않았다.

- 비용함수가 2차원에서 멈췄다.

 

차원을 n차원으로 만들면 만들수록 매끄러운 그래프가 나오지 않고 굴곡이 많은 그래프로 생성될 것이다.

그렇게 했을 때 learning rate를 작게 하면 제일 낮은 지점을 찾아가기도 전에 오해를 해서 전혀 다른 곳에서 결론을 내려버리게 되므로 learning rate에 적당한 값을 주는 것이 중요하다.

 

 

---

 

SGDRegressor

 

import 먼저

from sklearn.linear_model import SGDRegressor

sgd_model = SGDRegressor(max_iter=500,  # 가중치 업데이트 횟수
                         eta0=0.05,     # 학습률(learning rate)
                         verbose=1      # 학습과정을 확인
                        )

sgd_model.fit(data[['시간']],data['성적'])

out ↓ (접은글은 되도록이면 펼치지 말자..)

-- Epoch 1 Norm: 2.79, NNZs: 1, Bias: 1.642179, T: 4, Avg. loss: 467.968924 Total training time: 0.00 seconds. -- Epoch 2 Norm: 3.58, NNZs: 1, Bias: 3.296865, T: 8, Avg. loss: 1406.056237 Total training time: 0.00 seconds. -- Epoch 3 Norm: 6.67, NNZs: 1, Bias: 4.156475, T: 12, Avg. loss: 199.401762 Total training time: 0.00 seconds. -- Epoch 4 Norm: 7.90, NNZs: 1, Bias: 4.313131, T: 16, Avg. loss: 181.143138 Total training time: 0.00 seconds. -- Epoch 5 Norm: 8.78, NNZs: 1, Bias: 4.438388, T: 20, Avg. loss: 29.350812 Total training time: 0.00 seconds. -- Epoch 6 Norm: 9.03, NNZs: 1, Bias: 4.415704, T: 24, Avg. loss: 5.552248 Total training time: 0.00 seconds. -- Epoch 7 Norm: 9.37, NNZs: 1, Bias: 4.416447, T: 28, Avg. loss: 3.753420 Total training time: 0.00 seconds. -- Epoch 8 Norm: 9.10, NNZs: 1, Bias: 4.314839, T: 32, Avg. loss: 1.902873 Total training time: 0.00 seconds. -- Epoch 9 Norm: 9.32, NNZs: 1, Bias: 4.287998, T: 36, Avg. loss: 5.435799 Total training time: 0.00 seconds. -- Epoch 10 Norm: 9.25, NNZs: 1, Bias: 4.210074, T: 40, Avg. loss: 2.006776 Total training time: 0.00 seconds. -- Epoch 11 Norm: 9.50, NNZs: 1, Bias: 4.193991, T: 44, Avg. loss: 2.449498 Total training time: 0.00 seconds. -- Epoch 12 Norm: 9.20, NNZs: 1, Bias: 4.096993, T: 48, Avg. loss: 1.517316 Total training time: 0.00 seconds. -- Epoch 13 Norm: 9.42, NNZs: 1, Bias: 4.075368, T: 52, Avg. loss: 3.948540 Total training time: 0.00 seconds. -- Epoch 14 Norm: 9.32, NNZs: 1, Bias: 3.999678, T: 56, Avg. loss: 1.773840 Total training time: 0.00 seconds. -- Epoch 15 Norm: 9.55, NNZs: 1, Bias: 3.985185, T: 60, Avg. loss: 2.045520 Total training time: 0.00 seconds. -- Epoch 16 Norm: 9.26, NNZs: 1, Bias: 3.896145, T: 64, Avg. loss: 1.382948 Total training time: 0.00 seconds. -- Epoch 17 Norm: 9.48, NNZs: 1, Bias: 3.878481, T: 68, Avg. loss: 3.167908 Total training time: 0.00 seconds. -- Epoch 18 Norm: 9.36, NNZs: 1, Bias: 3.807480, T: 72, Avg. loss: 1.600908 Total training time: 0.00 seconds. -- Epoch 19 Norm: 9.59, NNZs: 1, Bias: 3.794541, T: 76, Avg. loss: 1.784457 Total training time: 0.00 seconds. -- Epoch 20 Norm: 9.31, NNZs: 1, Bias: 3.712734, T: 80, Avg. loss: 1.271109 Total training time: 0.00 seconds. -- Epoch 21 Norm: 9.52, NNZs: 1, Bias: 3.697622, T: 84, Avg. loss: 2.652506 Total training time: 0.00 seconds. -- Epoch 22 Norm: 9.40, NNZs: 1, Bias: 3.631448, T: 88, Avg. loss: 1.452629 Total training time: 0.00 seconds. -- Epoch 23 Norm: 9.61, NNZs: 1, Bias: 3.619685, T: 92, Avg. loss: 1.587304 Total training time: 0.00 seconds. -- Epoch 24 Norm: 9.35, NNZs: 1, Bias: 3.544224, T: 96, Avg. loss: 1.170425 Total training time: 0.00 seconds. -- Epoch 25 Norm: 9.55, NNZs: 1, Bias: 3.530838, T: 100, Avg. loss: 2.276718 Total training time: 0.00 seconds. -- Epoch 26 Norm: 9.43, NNZs: 1, Bias: 3.469178, T: 104, Avg. loss: 1.322738 Total training time: 0.00 seconds. -- Epoch 27 Norm: 9.63, NNZs: 1, Bias: 3.458302, T: 108, Avg. loss: 1.427488 Total training time: 0.00 seconds. -- Epoch 28 Norm: 9.39, NNZs: 1, Bias: 3.388409, T: 112, Avg. loss: 1.078999 Total training time: 0.00 seconds. -- Epoch 29 Norm: 9.58, NNZs: 1, Bias: 3.376259, T: 116, Avg. loss: 1.986322 Total training time: 0.00 seconds. -- Epoch 30 Norm: 9.46, NNZs: 1, Bias: 3.318716, T: 120, Avg. loss: 1.208004 Total training time: 0.00 seconds. -- Epoch 31 Norm: 9.64, NNZs: 1, Bias: 3.308532, T: 124, Avg. loss: 1.292883 Total training time: 0.00 seconds. -- Epoch 32 Norm: 9.42, NNZs: 1, Bias: 3.243554, T: 128, Avg. loss: 0.995860 Total training time: 0.00 seconds. -- Epoch 33 Norm: 9.60, NNZs: 1, Bias: 3.232333, T: 132, Avg. loss: 1.753177 Total training time: 0.00 seconds. -- Epoch 34 Norm: 9.49, NNZs: 1, Bias: 3.178522, T: 136, Avg. loss: 1.106045 Total training time: 0.00 seconds. -- Epoch 35 Norm: 9.66, NNZs: 1, Bias: 3.168898, T: 140, Avg. loss: 1.176855 Total training time: 0.00 seconds. -- Epoch 36 Norm: 9.45, NNZs: 1, Bias: 3.108292, T: 144, Avg. loss: 0.920176 Total training time: 0.00 seconds. -- Epoch 37 Norm: 9.62, NNZs: 1, Bias: 3.097802, T: 148, Avg. loss: 1.560897 Total training time: 0.01 seconds. -- Epoch 38 Norm: 9.51, NNZs: 1, Bias: 3.047374, T: 152, Avg. loss: 1.014970 Total training time: 0.01 seconds. -- Epoch 39 Norm: 9.67, NNZs: 1, Bias: 3.038220, T: 156, Avg. loss: 1.075305 Total training time: 0.01 seconds. -- Epoch 40 Norm: 9.48, NNZs: 1, Bias: 2.981528, T: 160, Avg. loss: 0.851187 Total training time: 0.01 seconds. -- Epoch 41 Norm: 9.64, NNZs: 1, Bias: 2.971635, T: 164, Avg. loss: 1.399137 Total training time: 0.01 seconds. -- Epoch 42 Norm: 9.53, NNZs: 1, Bias: 2.924279, T: 168, Avg. loss: 0.933251 Total training time: 0.01 seconds. -- Epoch 43 Norm: 9.68, NNZs: 1, Bias: 2.915531, T: 172, Avg. loss: 0.985472 Total training time: 0.01 seconds. -- Epoch 44 Norm: 9.50, NNZs: 1, Bias: 2.862367, T: 176, Avg. loss: 0.788205 Total training time: 0.01 seconds. -- Epoch 45 Norm: 9.65, NNZs: 1, Bias: 2.852978, T: 180, Avg. loss: 1.260970 Total training time: 0.01 seconds. -- Epoch 46 Norm: 9.55, NNZs: 1, Bias: 2.808421, T: 184, Avg. loss: 0.859637 Total training time: 0.01 seconds. -- Epoch 47 Norm: 9.70, NNZs: 1, Bias: 2.800032, T: 188, Avg. loss: 0.905374 Total training time: 0.01 seconds. -- Epoch 48 Norm: 9.53, NNZs: 1, Bias: 2.750064, T: 192, Avg. loss: 0.730618 Total training time: 0.01 seconds. -- Epoch 49 Norm: 9.67, NNZs: 1, Bias: 2.741110, T: 196, Avg. loss: 1.141532 Total training time: 0.01 seconds. -- Epoch 50 Norm: 9.57, NNZs: 1, Bias: 2.699113, T: 200, Avg. loss: 0.793093 Total training time: 0.01 seconds. -- Epoch 51 Norm: 9.71, NNZs: 1, Bias: 2.691047, T: 204, Avg. loss: 0.833522 Total training time: 0.01 seconds. -- Epoch 52 Norm: 9.55, NNZs: 1, Bias: 2.643990, T: 208, Avg. loss: 0.677883 Total training time: 0.01 seconds. -- Epoch 53 Norm: 9.68, NNZs: 1, Bias: 2.635421, T: 212, Avg. loss: 1.037286 Total training time: 0.01 seconds. -- Epoch 54 Norm: 9.59, NNZs: 1, Bias: 2.595772, T: 216, Avg. loss: 0.732758 Total training time: 0.01 seconds. -- Epoch 55 Norm: 9.72, NNZs: 1, Bias: 2.588003, T: 220, Avg. loss: 0.768752 Total training time: 0.01 seconds. -- Epoch 56 Norm: 9.57, NNZs: 1, Bias: 2.543607, T: 224, Avg. loss: 0.629521 Total training time: 0.01 seconds. -- Epoch 57 Norm: 9.69, NNZs: 1, Bias: 2.535385, T: 228, Avg. loss: 0.945573 Total training time: 0.01 seconds. -- Epoch 58 Norm: 9.61, NNZs: 1, Bias: 2.497896, T: 232, Avg. loss: 0.677903 Total training time: 0.01 seconds. -- Epoch 59 Norm: 9.73, NNZs: 1, Bias: 2.490402, T: 236, Avg. loss: 0.710137 Total training time: 0.01 seconds. -- Epoch 60 Norm: 9.59, NNZs: 1, Bias: 2.448448, T: 240, Avg. loss: 0.585106 Total training time: 0.01 seconds. -- Epoch 61 Norm: 9.70, NNZs: 1, Bias: 2.440543, T: 244, Avg. loss: 0.864354 Total training time: 0.01 seconds. -- Epoch 62 Norm: 9.62, NNZs: 1, Bias: 2.405047, T: 248, Avg. loss: 0.627910 Total training time: 0.01 seconds. -- Epoch 63 Norm: 9.74, NNZs: 1, Bias: 2.397811, T: 252, Avg. loss: 0.656914 Total training time: 0.01 seconds. -- Epoch 64 Norm: 9.60, NNZs: 1, Bias: 2.358107, T: 256, Avg. loss: 0.544263 Total training time: 0.01 seconds. -- Epoch 65 Norm: 9.71, NNZs: 1, Bias: 2.350494, T: 260, Avg. loss: 0.792022 Total training time: 0.01 seconds. -- Epoch 66 Norm: 9.64, NNZs: 1, Bias: 2.316842, T: 264, Avg. loss: 0.582250 Total training time: 0.01 seconds. -- Epoch 67 Norm: 9.74, NNZs: 1, Bias: 2.309849, T: 268, Avg. loss: 0.608451 Total training time: 0.01 seconds. -- Epoch 68 Norm: 9.62, NNZs: 1, Bias: 2.272224, T: 272, Avg. loss: 0.506657 Total training time: 0.01 seconds. -- Epoch 69 Norm: 9.72, NNZs: 1, Bias: 2.264882, T: 276, Avg. loss: 0.727295 Total training time: 0.01 seconds. -- Epoch 70 Norm: 9.65, NNZs: 1, Bias: 2.232941, T: 280, Avg. loss: 0.540464 Total training time: 0.01 seconds. -- Epoch 71 Norm: 9.75, NNZs: 1, Bias: 2.226178, T: 284, Avg. loss: 0.564213 Total training time: 0.01 seconds. -- Epoch 72 Norm: 9.64, NNZs: 1, Bias: 2.190479, T: 288, Avg. loss: 0.471991 Total training time: 0.01 seconds. -- Epoch 73 Norm: 9.73, NNZs: 1, Bias: 2.183393, T: 292, Avg. loss: 0.669133 Total training time: 0.01 seconds. -- Epoch 74 Norm: 9.67, NNZs: 1, Bias: 2.153042, T: 296, Avg. loss: 0.502157 Total training time: 0.01 seconds. -- Epoch 75 Norm: 9.76, NNZs: 1, Bias: 2.146498, T: 300, Avg. loss: 0.523744 Total training time: 0.01 seconds. -- Epoch 76 Norm: 9.65, NNZs: 1, Bias: 2.112588, T: 304, Avg. loss: 0.439999 Total training time: 0.01 seconds. -- Epoch 77 Norm: 9.74, NNZs: 1, Bias: 2.105742, T: 308, Avg. loss: 0.616678 Total training time: 0.01 seconds. -- Epoch 78 Norm: 9.68, NNZs: 1, Bias: 2.076874, T: 312, Avg. loss: 0.466981 Total training time: 0.01 seconds. -- Epoch 79 Norm: 9.77, NNZs: 1, Bias: 2.070539, T: 316, Avg. loss: 0.486653 Total training time: 0.01 seconds. -- Epoch 80 Norm: 9.66, NNZs: 1, Bias: 2.038294, T: 320, Avg. loss: 0.410445 Total training time: 0.01 seconds. -- Epoch 81 Norm: 9.75, NNZs: 1, Bias: 2.031676, T: 324, Avg. loss: 0.569218 Total training time: 0.01 seconds. -- Epoch 82 Norm: 9.69, NNZs: 1, Bias: 2.004191, T: 328, Avg. loss: 0.434632 Total training time: 0.01 seconds. -- Epoch 83 Norm: 9.78, NNZs: 1, Bias: 1.998057, T: 332, Avg. loss: 0.452597 Total training time: 0.01 seconds. -- Epoch 84 Norm: 9.68, NNZs: 1, Bias: 1.967366, T: 336, Avg. loss: 0.383116 Total training time: 0.01 seconds. -- Epoch 85 Norm: 9.76, NNZs: 1, Bias: 1.960965, T: 340, Avg. loss: 0.526155 Total training time: 0.01 seconds. -- Epoch 86 Norm: 9.70, NNZs: 1, Bias: 1.934773, T: 344, Avg. loss: 0.404842 Total training time: 0.01 seconds.

-- Epoch 87 Norm: 9.78, NNZs: 1, Bias: 1.928832, T: 348, Avg. loss: 0.421280 Total training time: 0.01 seconds. -- Epoch 88 Norm: 9.69, NNZs: 1, Bias: 1.899593, T: 352, Avg. loss: 0.357821 Total training time: 0.01 seconds. -- Epoch 89 Norm: 9.77, NNZs: 1, Bias: 1.893399, T: 356, Avg. loss: 0.486981 Total training time: 0.01 seconds. -- Epoch 90 Norm: 9.71, NNZs: 1, Bias: 1.868419, T: 360, Avg. loss: 0.377374 Total training time: 0.01 seconds. -- Epoch 91 Norm: 9.79, NNZs: 1, Bias: 1.862663, T: 364, Avg. loss: 0.392441 Total training time: 0.01 seconds. -- Epoch 92 Norm: 9.70, NNZs: 1, Bias: 1.834784, T: 368, Avg. loss: 0.334387 Total training time: 0.01 seconds. -- Epoch 93 Norm: 9.78, NNZs: 1, Bias: 1.828787, T: 372, Avg. loss: 0.451262 Total training time: 0.01 seconds. -- Epoch 94 Norm: 9.72, NNZs: 1, Bias: 1.804945, T: 376, Avg. loss: 0.352016 Total training time: 0.01 seconds. -- Epoch 95 Norm: 9.80, NNZs: 1, Bias: 1.799366, T: 380, Avg. loss: 0.365848 Total training time: 0.01 seconds. -- Epoch 96 Norm: 9.71, NNZs: 1, Bias: 1.772763, T: 384, Avg. loss: 0.312660 Total training time: 0.01 seconds. -- Epoch 97 Norm: 9.78, NNZs: 1, Bias: 1.766956, T: 388, Avg. loss: 0.418626 Total training time: 0.01 seconds. -- Epoch 98 Norm: 9.73, NNZs: 1, Bias: 1.744182, T: 392, Avg. loss: 0.328580 Total training time: 0.01 seconds. -- Epoch 99 Norm: 9.80, NNZs: 1, Bias: 1.738775, T: 396, Avg. loss: 0.341297 Total training time: 0.01 seconds. -- Epoch 100 Norm: 9.72, NNZs: 1, Bias: 1.713370, T: 400, Avg. loss: 0.292499 Total training time: 0.01 seconds. -- Epoch 101 Norm: 9.79, NNZs: 1, Bias: 1.707744, T: 404, Avg. loss: 0.388749 Total training time: 0.01 seconds. -- Epoch 102 Norm: 9.74, NNZs: 1, Bias: 1.685977, T: 408, Avg. loss: 0.306899 Total training time: 0.02 seconds. -- Epoch 103 Norm: 9.81, NNZs: 1, Bias: 1.680735, T: 412, Avg. loss: 0.318605 Total training time: 0.02 seconds. -- Epoch 104 Norm: 9.73, NNZs: 1, Bias: 1.656457, T: 416, Avg. loss: 0.273777 Total training time: 0.02 seconds. -- Epoch 105 Norm: 9.80, NNZs: 1, Bias: 1.651006, T: 420, Avg. loss: 0.361351 Total training time: 0.02 seconds. -- Epoch 106 Norm: 9.75, NNZs: 1, Bias: 1.630186, T: 424, Avg. loss: 0.286821 Total training time: 0.02 seconds. -- Epoch 107 Norm: 9.81, NNZs: 1, Bias: 1.625104, T: 428, Avg. loss: 0.297610 Total training time: 0.02 seconds. -- Epoch 108 Norm: 9.74, NNZs: 1, Bias: 1.601887, T: 432, Avg. loss: 0.256378 Total training time: 0.02 seconds. -- Epoch 109 Norm: 9.80, NNZs: 1, Bias: 1.596604, T: 436, Avg. loss: 0.336185 Total training time: 0.02 seconds. -- Epoch 110 Norm: 9.76, NNZs: 1, Bias: 1.576679, T: 440, Avg. loss: 0.268211 Total training time: 0.02 seconds. -- Epoch 111 Norm: 9.82, NNZs: 1, Bias: 1.571750, T: 444, Avg. loss: 0.278165 Total training time: 0.02 seconds. -- Epoch 112 Norm: 9.75, NNZs: 1, Bias: 1.549535, T: 448, Avg. loss: 0.240198 Total training time: 0.02 seconds. -- Epoch 113 Norm: 9.81, NNZs: 1, Bias: 1.544414, T: 452, Avg. loss: 0.313035 Total training time: 0.02 seconds. -- Epoch 114 Norm: 9.77, NNZs: 1, Bias: 1.525333, T: 456, Avg. loss: 0.250946 Total training time: 0.02 seconds. -- Epoch 115 Norm: 9.83, NNZs: 1, Bias: 1.520552, T: 460, Avg. loss: 0.260139 Total training time: 0.02 seconds. -- Epoch 116 Norm: 9.76, NNZs: 1, Bias: 1.499283, T: 464, Avg. loss: 0.225141 Total training time: 0.02 seconds. -- Epoch 117 Norm: 9.82, NNZs: 1, Bias: 1.494318, T: 468, Avg. loss: 0.291710 Total training time: 0.02 seconds. -- Epoch 118 Norm: 9.78, NNZs: 1, Bias: 1.476035, T: 472, Avg. loss: 0.234915 Total training time: 0.02 seconds. -- Epoch 119 Norm: 9.83, NNZs: 1, Bias: 1.471398, T: 476, Avg. loss: 0.243415 Total training time: 0.02 seconds. -- Epoch 120 Norm: 9.77, NNZs: 1, Bias: 1.451022, T: 480, Avg. loss: 0.211120 Total training time: 0.02 seconds. -- Epoch 121 Norm: 9.82, NNZs: 1, Bias: 1.446207, T: 484, Avg. loss: 0.272041 Total training time: 0.02 seconds. -- Epoch 122 Norm: 9.78, NNZs: 1, Bias: 1.428681, T: 488, Avg. loss: 0.220020 Total training time: 0.02 seconds. -- Epoch 123 Norm: 9.84, NNZs: 1, Bias: 1.424182, T: 492, Avg. loss: 0.227886 Total training time: 0.02 seconds. -- Epoch 124 Norm: 9.78, NNZs: 1, Bias: 1.404652, T: 496, Avg. loss: 0.198056 Total training time: 0.02 seconds. -- Epoch 125 Norm: 9.83, NNZs: 1, Bias: 1.399982, T: 500, Avg. loss: 0.253878 Total training time: 0.02 seconds. -- Epoch 126 Norm: 9.79, NNZs: 1, Bias: 1.383171, T: 504, Avg. loss: 0.206170 Total training time: 0.03 seconds. -- Epoch 127 Norm: 9.84, NNZs: 1, Bias: 1.378806, T: 508, Avg. loss: 0.213454 Total training time: 0.03 seconds. -- Epoch 128 Norm: 9.78, NNZs: 1, Bias: 1.360078, T: 512, Avg. loss: 0.185877 Total training time: 0.03 seconds. -- Epoch 129 Norm: 9.83, NNZs: 1, Bias: 1.355547, T: 516, Avg. loss: 0.237087 Total training time: 0.03 seconds. -- Epoch 130 Norm: 9.80, NNZs: 1, Bias: 1.339416, T: 520, Avg. loss: 0.193281 Total training time: 0.03 seconds. -- Epoch 131 Norm: 9.85, NNZs: 1, Bias: 1.335180, T: 524, Avg. loss: 0.200033 Total training time: 0.03 seconds. -- Epoch 132 Norm: 9.79, NNZs: 1, Bias: 1.317211, T: 528, Avg. loss: 0.174517 Total training time: 0.03 seconds. -- Epoch 133 Norm: 9.84, NNZs: 1, Bias: 1.312816, T: 532, Avg. loss: 0.221548 Total training time: 0.03 seconds. -- Epoch 134 Norm: 9.80, NNZs: 1, Bias: 1.297328, T: 536, Avg. loss: 0.181280 Total training time: 0.03 seconds. -- Epoch 135 Norm: 9.85, NNZs: 1, Bias: 1.293218, T: 540, Avg. loss: 0.187543 Total training time: 0.03 seconds. -- Epoch 136 Norm: 9.80, NNZs: 1, Bias: 1.275969, T: 544, Avg. loss: 0.163913 Total training time: 0.03 seconds. -- Epoch 137 Norm: 9.84, NNZs: 1, Bias: 1.271704, T: 548, Avg. loss: 0.207154 Total training time: 0.03 seconds. -- Epoch 138 Norm: 9.81, NNZs: 1, Bias: 1.256829, T: 552, Avg. loss: 0.170097 Total training time: 0.03 seconds. -- Epoch 139 Norm: 9.85, NNZs: 1, Bias: 1.252840, T: 556, Avg. loss: 0.175911 Total training time: 0.03 seconds. -- Epoch 140 Norm: 9.80, NNZs: 1, Bias: 1.236276, T: 560, Avg. loss: 0.154012 Total training time: 0.03 seconds. -- Epoch 141 Norm: 9.85, NNZs: 1, Bias: 1.232137, T: 564, Avg. loss: 0.193807 Total training time: 0.03 seconds. -- Epoch 142 Norm: 9.82, NNZs: 1, Bias: 1.217843, T: 568, Avg. loss: 0.159671 Total training time: 0.03 seconds. -- Epoch 143 Norm: 9.86, NNZs: 1, Bias: 1.213971, T: 572, Avg. loss: 0.165072 Total training time: 0.03 seconds. -- Epoch 144 Norm: 9.81, NNZs: 1, Bias: 1.198057, T: 576, Avg. loss: 0.144760 Total training time: 0.03 seconds. -- Epoch 145 Norm: 9.85, NNZs: 1, Bias: 1.194040, T: 580, Avg. loss: 0.181420 Total training time: 0.03 seconds. -- Epoch 146 Norm: 9.82, NNZs: 1, Bias: 1.180300, T: 584, Avg. loss: 0.149945 Total training time: 0.03 seconds. -- Epoch 147 Norm: 9.86, NNZs: 1, Bias: 1.176541, T: 588, Avg. loss: 0.154966 Total training time: 0.03 seconds. -- Epoch 148 Norm: 9.82, NNZs: 1, Bias: 1.161246, T: 592, Avg. loss: 0.136113 Total training time: 0.03 seconds. -- Epoch 149 Norm: 9.86, NNZs: 1, Bias: 1.157346, T: 596, Avg. loss: 0.169916 Total training time: 0.03 seconds. -- Epoch 150 Norm: 9.83, NNZs: 1, Bias: 1.144133, T: 600, Avg. loss: 0.140867 Total training time: 0.03 seconds. -- Epoch 151 Norm: 9.87, NNZs: 1, Bias: 1.140483, T: 604, Avg. loss: 0.145536 Total training time: 0.03 seconds. -- Epoch 152 Norm: 9.82, NNZs: 1, Bias: 1.125777, T: 608, Avg. loss: 0.128026 Total training time: 0.03 seconds. -- Epoch 153 Norm: 9.86, NNZs: 1, Bias: 1.121992, T: 612, Avg. loss: 0.159222 Total training time: 0.03 seconds. -- Epoch 154 Norm: 9.83, NNZs: 1, Bias: 1.109281, T: 616, Avg. loss: 0.132388 Total training time: 0.03 seconds. -- Epoch 155 Norm: 9.87, NNZs: 1, Bias: 1.105737, T: 620, Avg. loss: 0.136734 Total training time: 0.03 seconds. -- Epoch 156 Norm: 9.83, NNZs: 1, Bias: 1.091591, T: 624, Avg. loss: 0.120459 Total training time: 0.03 seconds. -- Epoch 157 Norm: 9.86, NNZs: 1, Bias: 1.087916, T: 628, Avg. loss: 0.149274 Total training time: 0.03 seconds. -- Epoch 158 Norm: 9.84, NNZs: 1, Bias: 1.075683, T: 632, Avg. loss: 0.124465 Total training time: 0.03 seconds. -- Epoch 159 Norm: 9.87, NNZs: 1, Bias: 1.072242, T: 636, Avg. loss: 0.128512 Total training time: 0.03 seconds. -- Epoch 160 Norm: 9.83, NNZs: 1, Bias: 1.058631, T: 640, Avg. loss: 0.113377 Total training time: 0.03 seconds. -- Epoch 161 Norm: 9.87, NNZs: 1, Bias: 1.055062, T: 644, Avg. loss: 0.140013 Total training time: 0.03 seconds. -- Epoch 162 Norm: 9.84, NNZs: 1, Bias: 1.043286, T: 648, Avg. loss: 0.117058 Total training time: 0.03 seconds. -- Epoch 163 Norm: 9.88, NNZs: 1, Bias: 1.039944, T: 652, Avg. loss: 0.120829 Total training time: 0.03 seconds. -- Epoch 164 Norm: 9.84, NNZs: 1, Bias: 1.026843, T: 656, Avg. loss: 0.106744 Total training time: 0.03 seconds. -- Epoch 165 Norm: 9.87, NNZs: 1, Bias: 1.023377, T: 660, Avg. loss: 0.131386 Total training time: 0.03 seconds. -- Epoch 166 Norm: 9.85, NNZs: 1, Bias: 1.012036, T: 664, Avg. loss: 0.110130 Total training time: 0.03 seconds. -- Epoch 167 Norm: 9.88, NNZs: 1, Bias: 1.008790, T: 668, Avg. loss: 0.113646 Total training time: 0.03 seconds. -- Epoch 168 Norm: 9.84, NNZs: 1, Bias: 0.996175, T: 672, Avg. loss: 0.100530 Total training time: 0.03 seconds. -- Epoch 169 Norm: 9.88, NNZs: 1, Bias: 0.992809, T: 676, Avg. loss: 0.123344 Total training time: 0.03 seconds. -- Epoch 170 Norm: 9.85, NNZs: 1, Bias: 0.981884, T: 680, Avg. loss: 0.103646 Total training time: 0.04 seconds. -- Epoch 171 Norm: 9.88, NNZs: 1, Bias: 0.978731, T: 684, Avg. loss: 0.106926 Total training time: 0.04 seconds. -- Epoch 172 Norm: 9.85, NNZs: 1, Bias: 0.966581, T: 688, Avg. loss: 0.094707 Total training time: 0.04 seconds. -- Epoch 173 Norm: 9.88, NNZs: 1, Bias: 0.963311, T: 692, Avg. loss: 0.115844 Total training time: 0.04 seconds. -- Epoch 174 Norm: 9.86, NNZs: 1, Bias: 0.952784, T: 696, Avg. loss: 0.097577 Total training time: 0.04 seconds. -- Epoch 175 Norm: 9.89, NNZs: 1, Bias: 0.949721, T: 700, Avg. loss: 0.100638 Total training time: 0.04 seconds. -- Epoch 176 Norm: 9.85, NNZs: 1, Bias: 0.938014, T: 704, Avg. loss: 0.089247 Total training time: 0.04 seconds. -- Epoch 177 Norm: 9.88, NNZs: 1, Bias: 0.934838, T: 708, Avg. loss: 0.108843 Total training time: 0.04 seconds. -- Epoch 178 Norm: 9.86, NNZs: 1, Bias: 0.924690, T: 712, Avg. loss: 0.091892 Total training time: 0.04 seconds. -- Epoch 179 Norm: 9.89, NNZs: 1, Bias: 0.921715, T: 716, Avg. loss: 0.094750 Total training time: 0.04 seconds. -- Epoch 180 Norm: 9.86, NNZs: 1, Bias: 0.910432, T: 720, Avg. loss: 0.084126 Total training time: 0.04 seconds. -- Epoch 181 Norm: 9.89, NNZs: 1, Bias: 0.907346, T: 724, Avg. loss: 0.102306 Total training time: 0.04 seconds. -- Epoch 182 Norm: 9.87, NNZs: 1, Bias: 0.897562, T: 728, Avg. loss: 0.086564 Total training time: 0.04 seconds. -- Epoch 183 Norm: 9.89, NNZs: 1, Bias: 0.894671, T: 732, Avg. loss: 0.089234 Total training time: 0.04 seconds. -- Epoch 184 Norm: 9.86, NNZs: 1, Bias: 0.883794, T: 736, Avg. loss: 0.079320 Total training time: 0.04 seconds. -- Epoch 185 Norm: 9.89, NNZs: 1, Bias: 0.880796, T: 740, Avg. loss: 0.096198 Total training time: 0.04 seconds. -- Epoch 186 Norm: 9.87, NNZs: 1, Bias: 0.871359, T: 744, Avg. loss: 0.081571 Total training time: 0.04 seconds. -- Epoch 187 Norm: 9.90, NNZs: 1, Bias: 0.868550, T: 748, Avg. loss: 0.084066 Total training time: 0.04 seconds. -- Epoch 188 Norm: 9.87, NNZs: 1, Bias: 0.858060, T: 752, Avg. loss: 0.074810 Total training time: 0.04 seconds. -- Epoch 189 Norm: 9.89, NNZs: 1, Bias: 0.855147, T: 756, Avg. loss: 0.090488 Total training time: 0.04 seconds. -- Epoch 190 Norm: 9.87, NNZs: 1, Bias: 0.846043, T: 760, Avg. loss: 0.076888 Total training time: 0.04 seconds. -- Epoch 191 Norm: 9.90, NNZs: 1, Bias: 0.843313, T: 764, Avg. loss: 0.079221 Total training time: 0.04 seconds. -- Epoch 192 Norm: 9.87, NNZs: 1, Bias: 0.833195, T: 768, Avg. loss: 0.070575 Total training time: 0.04 seconds. -- Epoch 193 Norm: 9.90, NNZs: 1, Bias: 0.830364, T: 772, Avg. loss: 0.085147 Total training time: 0.04 seconds. -- Epoch 194 Norm: 9.88, NNZs: 1, Bias: 0.821578, T: 776, Avg. loss: 0.072494 Total training time: 0.04 seconds. -- Epoch 195 Norm: 9.90, NNZs: 1, Bias: 0.818926, T: 780, Avg. loss: 0.074676 Total training time: 0.04 seconds. -- Epoch 196 Norm: 9.87, NNZs: 1, Bias: 0.809163, T: 784, Avg. loss: 0.066596 Total training time: 0.04 seconds. -- Epoch 197 Norm: 9.90, NNZs: 1, Bias: 0.806412, T: 788, Avg. loss: 0.080149 Total training time: 0.04 seconds. -- Epoch 198 Norm: 9.88, NNZs: 1, Bias: 0.797931, T: 792, Avg. loss: 0.068371 Total training time: 0.04 seconds. -- Epoch 199 Norm: 9.91, NNZs: 1, Bias: 0.795353, T: 796, Avg. loss: 0.070413 Total training time: 0.04 seconds. -- Epoch 200 Norm: 9.88, NNZs: 1, Bias: 0.785931, T: 800, Avg. loss: 0.062859 Total training time: 0.04 seconds. -- Epoch 201 Norm: 9.90, NNZs: 1, Bias: 0.783257, T: 804, Avg. loss: 0.075470 Total training time: 0.04 seconds. -- Epoch 202 Norm: 9.88, NNZs: 1, Bias: 0.775069, T: 808, Avg. loss: 0.064499 Total training time: 0.04 seconds. -- Epoch 203 Norm: 9.91, NNZs: 1, Bias: 0.772563, T: 812, Avg. loss: 0.066412 Total training time: 0.04 seconds. -- Epoch 204 Norm: 9.88, NNZs: 1, Bias: 0.763468, T: 816, Avg. loss: 0.059345 Total training time: 0.04 seconds. -- Epoch 205 Norm: 9.90, NNZs: 1, Bias: 0.760868, T: 820, Avg. loss: 0.071087 Total training time: 0.04 seconds. -- Epoch 206 Norm: 9.89, NNZs: 1, Bias: 0.752961, T: 824, Avg. loss: 0.060864 Total training time: 0.04 seconds. -- Epoch 207 Norm: 9.91, NNZs: 1, Bias: 0.750525, T: 828, Avg. loss: 0.062654 Total training time: 0.04 seconds. -- Epoch 208 Norm: 9.89, NNZs: 1, Bias: 0.741743, T: 832, Avg. loss: 0.056042 Total training time: 0.04 seconds. -- Epoch 209 Norm: 9.91, NNZs: 1, Bias: 0.739215, T: 836, Avg. loss: 0.066980 Total training time: 0.05 seconds. -- Epoch 210 Norm: 9.89, NNZs: 1, Bias: 0.731578, T: 840, Avg. loss: 0.057448 Total training time: 0.05 seconds. -- Epoch 211 Norm: 9.91, NNZs: 1, Bias: 0.729210, T: 844, Avg. loss: 0.059126 Total training time: 0.05 seconds. -- Epoch 212 Norm: 9.89, NNZs: 1, Bias: 0.720727, T: 848, Avg. loss: 0.052935 Total training time: 0.05 seconds. -- Epoch 213 Norm: 9.91, NNZs: 1, Bias: 0.718270, T: 852, Avg. loss: 0.063129 Total training time: 0.05 seconds. -- Epoch 214 Norm: 9.89, NNZs: 1, Bias: 0.710891, T: 856, Avg. loss: 0.054237 Total training time: 0.05 seconds. -- Epoch 215 Norm: 9.92, NNZs: 1, Bias: 0.708589, T: 860, Avg. loss: 0.055810 Total training time: 0.05 seconds. -- Epoch 216 Norm: 9.89, NNZs: 1, Bias: 0.700394, T: 864, Avg. loss: 0.050012 Total training time: 0.05 seconds. -- Epoch 217 Norm: 9.91, NNZs: 1, Bias: 0.698006, T: 868, Avg. loss: 0.059518 Total training time: 0.05 seconds. -- Epoch 218 Norm: 9.90, NNZs: 1, Bias: 0.690874, T: 872, Avg. loss: 0.051219 Total training time: 0.05 seconds. -- Epoch 219 Norm: 9.92, NNZs: 1, Bias: 0.688635, T: 876, Avg. loss: 0.052694 Total training time: 0.05 seconds. -- Epoch 220 Norm: 9.89, NNZs: 1, Bias: 0.680718, T: 880, Avg. loss: 0.047262 Total training time: 0.05 seconds. -- Epoch 221 Norm: 9.91, NNZs: 1, Bias: 0.678395, T: 884, Avg. loss: 0.056130 Total training time: 0.05 seconds. -- Epoch 222 Norm: 9.90, NNZs: 1, Bias: 0.671502, T: 888, Avg. loss: 0.048380 Total training time: 0.05 seconds. -- Epoch 223 Norm: 9.92, NNZs: 1, Bias: 0.669325, T: 892, Avg. loss: 0.049764 Total training time: 0.05 seconds. -- Epoch 224 Norm: 9.90, NNZs: 1, Bias: 0.661673, T: 896, Avg. loss: 0.044673 Total training time: 0.05 seconds. -- Epoch 225 Norm: 9.92, NNZs: 1, Bias: 0.659414, T: 900, Avg. loss: 0.052950 Total training time: 0.05 seconds. -- Epoch 226 Norm: 9.90, NNZs: 1, Bias: 0.652750, T: 904, Avg. loss: 0.045710 Total training time: 0.05 seconds. -- Epoch 227 Norm: 9.92, NNZs: 1, Bias: 0.650633, T: 908, Avg. loss: 0.047008 Total training time: 0.05 seconds. -- Epoch 228 Norm: 9.90, NNZs: 1, Bias: 0.643236, T: 912, Avg. loss: 0.042234 Total training time: 0.05 seconds. -- Epoch 229 Norm: 9.92, NNZs: 1, Bias: 0.641039, T: 916, Avg. loss: 0.049964 Total training time: 0.05 seconds. -- Epoch 230 Norm: 9.91, NNZs: 1, Bias: 0.634595, T: 920, Avg. loss: 0.043197 Total training time: 0.05 seconds. -- Epoch 231 Norm: 9.92, NNZs: 1, Bias: 0.632535, T: 924, Avg. loss: 0.044416 Total training time: 0.05 seconds. -- Epoch 232 Norm: 9.90, NNZs: 1, Bias: 0.625384, T: 928, Avg. loss: 0.039938 Total training time: 0.05 seconds. -- Epoch 233 Norm: 9.92, NNZs: 1, Bias: 0.623247, T: 932, Avg. loss: 0.047159 Total training time: 0.05 seconds. -- Epoch 234 Norm: 9.91, NNZs: 1, Bias: 0.617014, T: 936, Avg. loss: 0.040832 Total training time: 0.05 seconds. -- Epoch 235 Norm: 9.93, NNZs: 1, Bias: 0.615011, T: 940, Avg. loss: 0.041976 Total training time: 0.05 seconds. -- Epoch 236 Norm: 9.91, NNZs: 1, Bias: 0.608095, T: 944, Avg. loss: 0.037775 Total training time: 0.05 seconds. -- Epoch 237 Norm: 9.92, NNZs: 1, Bias: 0.606017, T: 948, Avg. loss: 0.044523 Total training time: 0.05 seconds. -- Epoch 238 Norm: 9.91, NNZs: 1, Bias: 0.599987, T: 952, Avg. loss: 0.038605 Total training time: 0.05 seconds. -- Epoch 239 Norm: 9.93, NNZs: 1, Bias: 0.598039, T: 956, Avg. loss: 0.039679 Total training time: 0.05 seconds. -- Epoch 240 Norm: 9.91, NNZs: 1, Bias: 0.591349, T: 960, Avg. loss: 0.035736 Total training time: 0.05 seconds. -- Epoch 241 Norm: 9.93, NNZs: 1, Bias: 0.589328, T: 964, Avg. loss: 0.042046 Total training time: 0.05 seconds. -- Epoch 242 Norm: 9.91, NNZs: 1, Bias: 0.583494, T: 968, Avg. loss: 0.036507 Total training time: 0.05 seconds. -- Epoch 243 Norm: 9.93, NNZs: 1, Bias: 0.581599, T: 972, Avg. loss: 0.037517 Total training time: 0.05 seconds. -- Epoch 244 Norm: 9.91, NNZs: 1, Bias: 0.575127, T: 976, Avg. loss: 0.033814 Total training time: 0.05 seconds. -- Epoch 245 Norm: 9.93, NNZs: 1, Bias: 0.573160, T: 980, Avg. loss: 0.039716 Total training time: 0.05 seconds. -- Epoch 246 Norm: 9.92, NNZs: 1, Bias: 0.567514, T: 984, Avg. loss: 0.034531 Total training time: 0.05 seconds. -- Epoch 247 Norm: 9.93, NNZs: 1, Bias: 0.565671, T: 988, Avg. loss: 0.035480 Total training time: 0.05 seconds. -- Epoch 248 Norm: 9.91, NNZs: 1, Bias: 0.559408, T: 992, Avg. loss: 0.032002 Total training time: 0.05 seconds. -- Epoch 249 Norm: 9.93, NNZs: 1, Bias: 0.557495, T: 996, Avg. loss: 0.037525 Total training time: 0.05 seconds. -- Epoch 250 Norm: 9.92, NNZs: 1, Bias: 0.552030, T: 1000, Avg. loss: 0.032668 Total training time: 0.05 seconds. -- Epoch 251 Norm: 9.93, NNZs: 1, Bias: 0.550236, T: 1004, Avg. loss: 0.033560 Total training time: 0.05 seconds. -- Epoch 252 Norm: 9.92, NNZs: 1, Bias: 0.544176, T: 1008, Avg. loss: 0.030293 Total training time: 0.05 seconds. -- Epoch 253 Norm: 9.93, NNZs: 1, Bias: 0.542314, T: 1012, Avg. loss: 0.035463 Total training time: 0.05 seconds. -- Epoch 254 Norm: 9.92, NNZs: 1, Bias: 0.537024, T: 1016, Avg. loss: 0.030913 Total training time: 0.05 seconds. -- Epoch 255 Norm: 9.94, NNZs: 1, Bias: 0.535278, T: 1020, Avg. loss: 0.031752 Total training time: 0.05 seconds. -- Epoch 256 Norm: 9.92, NNZs: 1, Bias: 0.529412, T: 1024, Avg. loss: 0.028680 Total training time: 0.06 seconds. -- Epoch 257 Norm: 9.93, NNZs: 1, Bias: 0.527600, T: 1028, Avg. loss: 0.033522 Total training time: 0.06 seconds. -- Epoch 258 Norm: 9.92, NNZs: 1, Bias: 0.522478, T: 1032, Avg. loss: 0.029257 Total training time: 0.06 seconds. -- Epoch 259 Norm: 9.94, NNZs: 1, Bias: 0.520779, T: 1036, Avg. loss: 0.030047 Total training time: 0.06 seconds. -- Epoch 260 Norm: 9.92, NNZs: 1, Bias: 0.515100, T: 1040, Avg. loss: 0.027159 Total training time: 0.06 seconds. -- Epoch 261 Norm: 9.93, NNZs: 1, Bias: 0.513337, T: 1044, Avg. loss: 0.031695 Total training time: 0.06 seconds. -- Epoch 262 Norm: 9.93, NNZs: 1, Bias: 0.508376, T: 1048, Avg. loss: 0.027696 Total training time: 0.06 seconds. -- Epoch 263 Norm: 9.94, NNZs: 1, Bias: 0.506723, T: 1052, Avg. loss: 0.028439 Total training time: 0.06 seconds. -- Epoch 264 Norm: 9.92, NNZs: 1, Bias: 0.501224, T: 1056, Avg. loss: 0.025724 Total training time: 0.06 seconds. -- Epoch 265 Norm: 9.94, NNZs: 1, Bias: 0.499508, T: 1060, Avg. loss: 0.029975 Total training time: 0.06 seconds. -- Epoch 266 Norm: 9.93, NNZs: 1, Bias: 0.494704, T: 1064, Avg. loss: 0.026224 Total training time: 0.06 seconds. -- Epoch 267 Norm: 9.94, NNZs: 1, Bias: 0.493095, T: 1068, Avg. loss: 0.026923 Total training time: 0.06 seconds. -- Epoch 268 Norm: 9.93, NNZs: 1, Bias: 0.487769, T: 1072, Avg. loss: 0.024368 Total training time: 0.06 seconds. -- Epoch 269 Norm: 9.94, NNZs: 1, Bias: 0.486099, T: 1076, Avg. loss: 0.028354 Total training time: 0.06 seconds. -- Epoch 270 Norm: 9.93, NNZs: 1, Bias: 0.481445, T: 1080, Avg. loss: 0.024834 Total training time: 0.06 seconds. -- Epoch 271 Norm: 9.94, NNZs: 1, Bias: 0.479879, T: 1084, Avg. loss: 0.025493 Total training time: 0.06 seconds. -- Epoch 272 Norm: 9.93, NNZs: 1, Bias: 0.474720, T: 1088, Avg. loss: 0.023088 Total training time: 0.06 seconds. -- Epoch 273 Norm: 9.94, NNZs: 1, Bias: 0.473095, T: 1092, Avg. loss: 0.026826 Total training time: 0.06 seconds. -- Epoch 274 Norm: 9.93, NNZs: 1, Bias: 0.468585, T: 1096, Avg. loss: 0.023523 Total training time: 0.06 seconds. -- Epoch 275 Norm: 9.94, NNZs: 1, Bias: 0.467061, T: 1100, Avg. loss: 0.024143 Total training time: 0.06 seconds. -- Epoch 276 Norm: 9.93, NNZs: 1, Bias: 0.462063, T: 1104, Avg. loss: 0.021880 Total training time: 0.06 seconds. -- Epoch 277 Norm: 9.94, NNZs: 1, Bias: 0.460481, T: 1108, Avg. loss: 0.025387 Total training time: 0.06 seconds. -- Epoch 278 Norm: 9.93, NNZs: 1, Bias: 0.456111, T: 1112, Avg. loss: 0.022285 Total training time: 0.06 seconds. -- Epoch 279 Norm: 9.94, NNZs: 1, Bias: 0.454628, T: 1116, Avg. loss: 0.022869 Total training time: 0.06 seconds. -- Epoch 280 Norm: 9.93, NNZs: 1, Bias: 0.449785, T: 1120, Avg. loss: 0.020738 Total training time: 0.06 seconds. -- Epoch 281 Norm: 9.94, NNZs: 1, Bias: 0.448245, T: 1124, Avg. loss: 0.024029 Total training time: 0.06 seconds. -- Epoch 282 Norm: 9.93, NNZs: 1, Bias: 0.444010, T: 1128, Avg. loss: 0.021115 Total training time: 0.06 seconds. -- Epoch 283 Norm: 9.95, NNZs: 1, Bias: 0.442566, T: 1132, Avg. loss: 0.021666 Total training time: 0.06 seconds. -- Epoch 284 Norm: 9.93, NNZs: 1, Bias: 0.437873, T: 1136, Avg. loss: 0.019659 Total training time: 0.06 seconds. -- Epoch 285 Norm: 9.94, NNZs: 1, Bias: 0.436373, T: 1140, Avg. loss: 0.022749 Total training time: 0.06 seconds. -- Epoch 286 Norm: 9.94, NNZs: 1, Bias: 0.432268, T: 1144, Avg. loss: 0.020011 Total training time: 0.06 seconds. -- Epoch 287 Norm: 9.95, NNZs: 1, Bias: 0.430862, T: 1148, Avg. loss: 0.020531 Total training time: 0.06 seconds. -- Epoch 288 Norm: 9.94, NNZs: 1, Bias: 0.426314, T: 1152, Avg. loss: 0.018640 Total training time: 0.06 seconds. -- Epoch 289 Norm: 9.95, NNZs: 1, Bias: 0.424854, T: 1156, Avg. loss: 0.021541 Total training time: 0.06 seconds. -- Epoch 290 Norm: 9.94, NNZs: 1, Bias: 0.420874, T: 1160, Avg. loss: 0.018968 Total training time: 0.06 seconds. -- Epoch 291 Norm: 9.95, NNZs: 1, Bias: 0.419505, T: 1164, Avg. loss: 0.019458 Total training time: 0.06 seconds. -- Epoch 292 Norm: 9.94, NNZs: 1, Bias: 0.415096, T: 1168, Avg. loss: 0.017676 Total training time: 0.06 seconds. -- Epoch 293 Norm: 9.95, NNZs: 1, Bias: 0.413675, T: 1172, Avg. loss: 0.020402 Total training time: 0.06 seconds. Convergence after 293 epochs took 0.06 seconds

Out[16]:

SGDRegressor(eta0=0.05, max_iter=500, verbose=1)

print('가중치 : ',sgd_model.coef_)
print('절편 : ',sgd_model.intercept_)
# 값은 실행할 때마다 조금씩 바뀌게 된다.

out :

가중치 : [9.94722341] 절편 : [0.41367463]

# 예측
# 시간값을 7시간으로 해서 예측
sgd_model.predict([[7]])

out : array([70.0442385])

 

학습률을 낮춰보자.

sgd_model = SGDRegressor(max_iter=500,
                         eta0=0.001,
                         verbose=1
                        )

sgd_model.fit(data[['시간']],data['성적'])

out : 너무 긴 부분은 과감히 안 가져오지 않았다.(?)

     SGDRegressor(eta0=0.001, max_iter=500, verbose=1)

print('가중치 : ',sgd_model.coef_)
print('절편 : ',sgd_model.intercept_)

out :

가중치 : [9.79233997] 절편 : [1.33376825]

sgd_model.predict([[7]])

out :

array([69.88014803])

 

 

보통 어느정도가 적당한가?

• 가중치를 작게, 학습률을 조금씩 키워가면서 적당한 값을 찾고
• 적당한 값을 찾게 되면 학습률을 둔 상태로 가중치를 높여보면서 찾는다.
• 데이터마다 기준이 다르기 때문에 정답은 없다.

데이터가 복잡해졌을 때 결과값이 국소점에 빠졌는지 제대로 값을 찾은건지 어떻게 확인하나요?

• verbose를 통해 확인 가능
• 실행 후 결과에서 loss값을 확인하면 찾을 수 있다.
• lose는 평균제곱오차. 우리는 평균제곱오차가 최소가 될 때 w와 b를 선형함수에 적용한다.
• 국소지역점에 빠지게 되면 최저값이 상당히 크게 나올 것이다.
• loss값이 크게 나왔다면 국소지역점에 빠진 것이 아닐까 의심하면서 학습률을 키운다.
• 학습률을 키웠을 때 최저점이 더 낮아지게 된다면 국소지역점에 빠진 상태였으며 잘 빠져나온 것을 알 수 있다.

 

max_iter는 작은값을 두지 않고 주로 100 ~ 5000 이렇게 크게 두며,

목표값을 찾더라도 멈추지 않고 그 주변에서 계속 배회하며 움직인다.

학습률이 너무 작으면 목표값을 제대로 찾지 못하고 자신이 찾은 국소점에서 주변을 배회한다.

그러므로 최적값을 찾기 위해 학습률을 높이면 목표를 잘 찾을 수 있다.

 

Linear Model

• 장점
  • 결과예측(추론) 속도가 빠르다.
  • 대용량 데이터에도 충분히 활용 가능하다.
  • 특성이 많은 데이터 세트라면 훌륭한 성능을 낼 수 있다.
• 단점
  • 특성이 적은 저차원 데이터에서는 다른 모델의 일반화 성능이 더 좋을 수 있다. → 특성확장을 하기도 한다.
  • LinearRegression Model은 복잡도를 제어할 방법이 없어 과대적합되기 쉽다. → 파라미터 제어가 어렵다.
  • └→모델 정규화(Regularization)을 통해 과대적합을 제어한다.

 

모델정규화

• 가중치(w)의 값을 조정하여 제약을 주는 것.
• L1 규제 : Lasso
  • w의 모든 원소에 똑같은 힘(같은 값만큼)으로 규제를 적용하는 방법.
  • 특정 계수들은 0이 됨.
  • 특성선택(Feature Selection)이 자동으로 이루어진다.
• L2 규제 : Ridge
  • w의 모든 원소에 골고루 규제(같은 퍼센트만큼)를 적용하여 0에 가깝게 만든다. (0이 되지는 않는다.)
    

 

 

---

 

선형 모델을 활용한 보스턴 주택값 예측 실습

 

import numpy as np
import pandas as pd
from sklearn.datasets import load_boston # 보스톤 주택값 데이터

이런 오류가 뜰 것이다.

버전이 바뀌면서 값이 이렇게 바뀔 수 있으니 이렇게 사용해라! 라고 알려주는 오류.

무시해도 상관 없다고 한다.

 

boston = load_boston()

불러온 보스턴을 boston이라는 변수에 담아준다.

이후 각각 담아진 값 확인. (이 부분은 캡쳐로 대신)

print(boston.filename)

out : 해당 파일이 있는 경로를 출력

# boston에 담긴 데이터를 DataFrame로 만들기
# boston의 feature_names를 column으로 사용
# X 데이터 = 문제 데이터
df_boston = pd.DataFrame(boston.data, columns=boston.feature_names)
df_boston.head()

# boston의 target값 사용
# y 데이터 = 정답데이터
house_price = boston.target

# train과 test로 분리
# X_train, X_test, y_train, y_test
# train_test_split을 사용
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(df_boston, house_price)

# 특성 확장
extend_X_train = X_train.copy() # copy로 값만 복사해서 extend_X_train변수에 담기
for col1 in X_train.columns: # 1~13
    for col2 in X_train.columns: # 1~13
        extend_X_train[col1+'X'+col2] = X_train[col1]*X_train[col2]

extend_X_train.columns

 

방금 코드에 대한 선생님의 설명↓

# 특성 확장 : X_test도 동일하게.
extend_X_test = X_test.copy() # copy로 값만 복사해서 extend_X_test변수에 담기
for col1 in X_test.columns: # 1~13
    for col2 in X_test.columns: # 1~13
        extend_X_test[col1+'X'+col2] = X_test[col1]*X_test[col2]

extend_X_test.columns

Lasso

# Lasso에는 기본값으로 alpha=1을 가지고 있다. 
# 하지만 지금은 배우는 중이니 보기 편하게 적어주자.
lasso = Lasso(alpha = 1)
lasso.fit(extend_X_train, y_train)
print('train score : ',lasso.score(extend_X_train,y_train))
print('test score : ',lasso.score(extend_X_test,y_test))

# 사용한 특성의 개수
print(np.sum(lasso.coef_!=0))

방금 위 코드에서 일부만 설명 ↓

# 가중치가 0인 것만 빼고 확인
lasso.coef_!=0

# np.sum을 해주면 true인 값만 세서 출력
np.sum(lasso.coef_!=0)

out : 91

 

alpha값을 바꿔보자.

# alpha = 100
lasso = Lasso(alpha = 100)
lasso.fit(extend_X_train, y_train)
print('train score : ',lasso.score(extend_X_train,y_train))
print('test score : ',lasso.score(extend_X_test,y_test))

# 사용한 특성의 개수
print('사용한 특성의 개수 : ',np.sum(lasso.coef_!=0))

out :

train score : 0.7821091502420976

test score : 0.715837958206043

사용한 특성의 개수 : 34

# alpha = 0.001
lasso = Lasso(alpha = 0.001)
lasso.fit(extend_X_train, y_train)
print('train score : ',lasso.score(extend_X_train,y_train))
print('test score : ',lasso.score(extend_X_test,y_test))

# 사용한 특성의 개수
print('사용한 특성의 개수 : ',np.sum(lasso.coef_!=0))

out :

train score : 0.9227384806395925

test score : 0.8230963662601883

사용한 특성의 개수 : 177

 

lasso모델은 가중치에서 일정한 값을 뺀다.

ridge모델은 일정 퍼센트를 뺀다.

alpha : w(가중치)를 얼마나 사용할 것인가. ← alpha는 lidge에도 적용된다.

과대적합이 걸리면 alpha값을 높이고 규제를 강화해서 일반화, 과소적합이 걸리면 alpha값을 낮춰 규제를 약화시키고 특성을 많이 사용하도록 하여 일반화 영역에 도달할 수 있도록 해야한다.

 

ridge모델은 여러분이 직접 해보도록 합시다!