Role of validation for neural network (deep learning) - validation 역할

Why do we need validation (validation이 필요한 이유)?

• In order to estimate how well your model has been trained (that is dependent upon the size of your data, the value you would like to predict, input etc) and to estimate model properties (mean error for numeric predictors, classification errors for classifiers, recall and precision for IR-models etc.)
  - 모델이 학습이 잘 되었는지 측정하기 위해
  
  
• A set of examples used to tune the parameters of a classifier In the MLP case, we would use the validation set to find the “optimal” number of hidden units or determine a stopping point for the back-propagation algorithm
  - 모델의 파리미터를 조정하기 위해
  - 히든 유닛의 최적 개수를 찾기 위해
  - 역전파 알고리즘의 종료 시점을 결정 하기 위해
  
  
• This data set is used to compare the performances of the prediction algorithms that were created based on the training set. We choose the algorithm that has the best performance
  - 예측 알고리즘(모델)의 성능을 비교하기 위해
  
  
• You now have a collection of algorithms. You must pick one algorithm. That’s why you have a test set. Most people pick the algorithm that performs best on the validation set (and that's ok). But, if you do not measure your top-performing algorithm’s error rate on the test set, and just go with its error rate on the validation set, then you have blindly mistaken the “best possible scenario” for the “most likely scenario.” That's a recipe for disaster
  - 최고 성능의 알고리즘(모델)을 확인하기 위해 (특정 epoch에서 최고 성능을 보여준다면, 그 epoch를 저장)
  
  
• Example of Validation (Record best recorded epoch)
   - 예제 코드

# validate

if validate and i % iters_per_epoch == 0:

x, y = data_dict['validate'].next()

feed_dict = {self.x: x, self.y: y,

self.dropout: 1.0} # keep prob 1

accuracy = sesh.run(self.accuracy, feed_dict)

cross_vals.append(accuracy)

if accuracy >= best_acc:

best_acc = accuracy

if save:

epoch = (i // iters_per_epoch)

print 'Record validation accuracy (epoch {}): '.format(

epoch) + '{}. Freezing!'.format(best_acc)

self._freeze(epoch = (i // iters_per_epoch))

References

• https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa
  
  
• https://machinelearningmastery.com/difference-test-validation-datasets/
  
  
• https://github.com/meereeum/vANNilla-tf