2.1 What to Do if My Network Fails to Train - Joooy | Zeng

1 Model Bias

Solution: Redesign model to make it more flexible

2 Optimization Issue

Model Bias v.s. Optimization Issue

• Start from shallower network (or other models), which are easier to optimize.
• If deeper networks do not obtain smaller loss on training data, then there is optimization issue.

3 Overfitting

A more complex model yields lower error on training data.

But large loss on testing data.

Solution

1. More training data

2. Data Augmentation

• Fully-Connected is more flexible,

  CNN is relatively less flexible, and the constrain is relatively large.

Too much constrain back to model bias

4 Bias-Complexity Trade-off

Solution: Cross Validation

N-fold Cross Validation

5 Mismatch

Your training and testing data have different distributions.

Solution: To understand how data is generated.