The NNET Procedure
OPTIMIZATION Statement
OPTIMIZATION <options>;
The OPTIMIZATION statement specifies options for the optimization method that is used to train your model. When you are training your model, the objective function to be minimized is
where is the loss associated with observation i having data and correct classification , and is a regularization term defined by
You can specify the following options:
-
ALGORITHM=ADAM <sgd-options> |LBFGS <lbfgs-options> |SGD <sgd-options> |HF
ALG=ADAM <sgd-options> |LBFGS <lbfgs-options> |SGD <sgd-options> |HF specifies the optimization algorithm to use during training. You can specify the following optimization algorithms:
- ADAM <sgd-options>
specifies the adaptive moments (ADAM) algorithm, which is one of the variations of the stochastic gradient descent algorithm. It keeps track of the decaying averages of the past gradients and past squared gradients.
You can specify these additional sgd-options:
- ANNEALINGRATE=number
specifies the annealing parameter, . Annealing is a way to automatically reduce the learning rate as the algorithm progresses, causing smaller steps as the algorithm approaches a solution. Effectively, it replaces the learning rate parameter, , with
where t is the number of iterations that the algorithm has performed.
By default, ANNEALINGRATE=1.0E–6. The number must be a nonnegative double.
- COMMFREQ=number
specifies the number of minibatches that each computational thread processes before weights are synchronized across all threads and nodes.
- LEARNINGRATE=number
specifies the learning rate parameter, , for the algorithm. New iterates for the algorithm are found by using
where is the current weight vector, is the new weight vector, is the minibatch used during iteration k, and is the loss associated with the ith observation.
If you see a huge objective value from the algorithm, especially for a small data set, it is likely that the learning rate is set too high.
By default, LEARNINGRATE=0.001. The number must be a nonnegative double.
- MINIBATCHSIZE=number
specifies the size of the minibatches to use in the algorithm.
By default, MINIBATCHSIZE=10.
- MOMENTUM=number
specifies the value for momentum. The number must be greater than or equal to 0 and less than or equal to 1. By default, MOMENTUM=0.
- SEED=number
specifies the seed for random access of observations on each thread for the algorithm. If number is less than or equal to 0 or not specified, a random seed is generated by reading the time of day from the computer’s clock.
- USELOCKING
specifies that computational threads share a common weight vector and update weight vector without race conditions. If you do not specify this option, computational threads update a single weight vector simultaneously. This causes intentional race conditions and nondeterministic behavior but increases performance significantly.
- HF (Experimental)
specifies the Hessian-free algorithm, a nonlinear optimization algorithm that uses the Hessian vector product to build the second-order information. This algorithm is based on a modified conjugate gradient method. Because it does not use the Hessian directly, it can work for large-scale problems.
- LBFGS <lbfgs-options>
specifies the limited-memory Broyden-Fletcher-Goldfarb-Shanno algorithm.
You can specify the following lbfgs-option:
- SGD <sgd-options>
specifies the plain stochastic gradient descent (SGD) algorithm.
You can specify the following sgd-options:
- ANNEALINGRATE=number
specifies the annealing parameter, . Annealing is a way to automatically reduce the learning rate as the algorithm progresses, causing smaller steps as the algorithm approaches a solution. Effectively, it replaces the learning rate parameter, , with
where t is the number of iterations that the algorithm has performed.
By default, ANNEALINGRATE=1.0E–6. The number must be a nonnegative double.
- COMMFREQ=number
specifies the number of minibatches that each computational thread processes before weights are synchronized across all threads and nodes.
- LEARNINGRATE=number
specifies the learning rate parameter, , for the algorithm. New iterates for the algorithm are found by using
where is the current weight vector, is the new weight vector, is the minibatch used during iteration k, and is the loss associated with the ith observation.
If you see a huge objective value from the algorithm, especially for a small data set, it is likely that the learning rate is set too high.
By default, LEARNINGRATE=0.001. The number must be a nonnegative double.
- MINIBATCHSIZE=number
specifies the size of the minibatches to use in the algorithm.
By default, MINIBATCHSIZE=10.
- MOMENTUM=number
specifies the value for momentum. The number must be greater than or equal to 0 and less than or equal to 1. By default, MOMENTUM=0.
- SEED=number
specifies the seed for random access of observations on each thread for the algorithm. If number is less than or equal to 0 or not specified, a random seed is generated by reading the time of day from the computer’s clock.
- USELOCKING
specifies that computational threads share a common weight vector and update weight vector without race conditions. If you do not specify this option, computational threads update a single weight vector simultaneously. This causes intentional race conditions and nondeterministic behavior but increases performance significantly.
By default, ALGORITHM=LBFGS.
- MAXITER=number
specifies the iteration budget for training. When ALGORITHM=LBFGS, the algorithm stops after MAXITER= iterations if convergence has not been achieved. When ALGORITHM=SGD or ADAM, number specifies the desired number of training epochs.
By default, MAXITER=250.
- MAXTIME=number
specifies the maximum time (in seconds) allowed for optimization, where number must be greater than or equal to 1. When this value is reached, the optimization terminates the search and returns results. When MAXTIME=0, no maximum time is set.
By default, MAXTIME=0.
- REGL1=number
specifies the L1 regularization parameter for the model loss function. The number must be nonnegative. Note that this value is autotuned when you specify the AUTOTUNE statement.
By default, REGL1=0.
- REGL2=number
specifies the L2 regularization parameter . The number must be nonnegative. Note that this value is autotuned when you specify the AUTOTUNE statement.
By default, REGL2=0.