Purdue Proposal

Experiment Proposal

hypothesis 1: neural net learns low level features first

How to set up:

train neural network to fit polynomial

using random sampling and expectation to calculate each coefficient.

calculate the MSE of coefficient

  • if the MSE of low level features are lower, hypothesis is true.

hypothesis 2: the more data used, the higher accuracy of a coefficient will be achieved.

How to set up:

using different data scale to train a network

calculate the MSE of each coefficient

draw the MSE-coefficient curve for different data scale.

draw the MSE of an coefficient v.s. different data scale.

  • whether a critical behavior appears
  • See whether the MSE of a coefficient is incrasing by data scale

hypothesis 4: With the incrase of network parameters, there will be a critical behavior.

  • How to set up:

fixed data scale

draw the MSE-coefficient curve for different parameter scale.

draw the MSE of an coefficient v.s. different parameter scale.

  • whether a critical behavior appears
  • See whether the MSE of a coefficient is incrasing by parameter scale

hypothesis 3: If only degree one appears, i.e. $f(x)=a_1x_1+a_2x_2$, the coefficient with large scale will be learnt first.

How to set up:

using fixed data scale and parameters.

Set $a_1$ as larger value.

calculate the MSE of each coefficient

  • If the MSE of $a_1$ is larger, the hypothesis is proved