Experiment Proposal

hypothesis 1: neural net learns low level features first

How to set up:

train neural network to fit polynomial

using random sampling and expectation to calculate each coefficient.

calculate the MSE of coefficient

• if the MSE of low level features are lower, hypothesis is true.

hypothesis 2: the more data used, the higher accuracy of a coefficient will be achieved.

How to set up:

using different data scale to train a network

calculate the MSE of each coefficient

draw the MSE-coefficient curve for different data scale.

draw the MSE of an coefficient v.s. different data scale.

• whether a critical behavior appears
• See whether the MSE of a coefficient is incrasing by data scale

hypothesis 4: With the incrase of network parameters, there will be a critical behavior.

• How to set up:

fixed data scale

draw the MSE-coefficient curve for different parameter scale.

draw the MSE of an coefficient v.s. different parameter scale.

• whether a critical behavior appears
• See whether the MSE of a coefficient is incrasing by parameter scale

hypothesis 3: If only degree one appears, i.e. \(f(x)=a_1x_1+a_2x_2\), the coefficient with large scale will be learnt first.

How to set up:

using fixed data scale and parameters.

Set \(a_1\) as larger value.

calculate the MSE of each coefficient

• If the MSE of \(a_1\) is larger, the hypothesis is proved