Norm Constraints¶
Norm constraints is a more “direct” way of restricting the model complexity by explicitly shrinking the parameters every n iterations if the norm of the parameters exceeds a given threshold.
-
class
NoCons
¶ No constraint is applied.
-
class
L2Cons
¶ Constrain the Euclidean norm of parameters. Note that the threshold and shrinking are applied to each parameter. Specifically, for the filters parameter of a convolution layer, the threshold is applied to each filter. Similarly, for the weights parameter of an inner product layer, the threshold is applied to the weights corresponding to each single output dimension of the inner product layer. When the norm of the parameter exceed the threshold, it is scaled down to have exactly the norm specified in threshold.
See the MNIST with dropout code in the
examples
directory for an example of howL2Cons
is used.-
threshold
¶ The norm threshold.
-
every_n_iter
¶ Defautl 1. Indicates the frequency of norm constraint application.
-