GPREG Procedure
OPTIMIZATION Statement
OPTIMIZATION <optimization-algorithm><options>;
The optimization statement specifies options for the optimization method that you use to train your model. When you are training your model, the objective function to be minimized is
where 
is the loss associated with observation i having data 
and correct classification 
, and 
is a regularization term defined by
You can specify the following options:
-
ALGORITHM=ADAM <sgd-options> SGD <sgd-options>
ALG=ADAM <sgd-options> SGD <sgd-options> -
specifies the optimization-algorithm to use during training. You can specify one of the four following algorithms:
- ADAM <sgd-options>
-
specifies the adaptive moments (Adam) algorithm, which is one of the variations of the stochastic gradient descent algorithm. It keeps track of the decaying averages of the past gradients and past squared gradients.
You can specify these additional sgd-options:
- ANNEALINGRATE=number
-
specifies the annealing parameter,
. Annealing is a way to automatically reduce the learning rate as the algorithm progresses, producing smaller steps as the algorithm approaches a solution. Effectively, it replaces the learning rate parameter, 
, with 
where t is the number of iterations that the algorithm has performed.
By default, ANNEALINGRATE=1.0E–6. The number must be a nonnegative double.
- COMMFREQ=number
specifies the number of minibatches that each computational thread processes before weights are synchronized across all threads and nodes.
- LEARNINGRATE=number
-
specifies the learning rate parameter,
, for the algorithm. New iterates for the algorithm are found by using 
where
is the current weight vector, 
is the new weight vector, 
is the minibatch used during iteration k, and 
is the loss associated with the ith observation. If you see a huge objective value from the algorithm, especially for a small data set, it is likely that the learning rate is set too high.
By default, LEARNINGRATE=0.001. The number must be a nonnegative double.
- MINIBATCHSIZE=number
-
specifies the size of the minibatches to use in the algorithm.
By default, MINIBATCHSIZE=10.
- MOMENTUM=number
-
specifies the value for momentum. The number must be greater than or equal to 0 and less than or equal to 1.
By default, MOMENTUM=0.
- SEED=number
specifies the seed for random access of observations on each thread for the algorithm. If number is less than or equal to 0, a random seed is generated by reading the time of day from the computer’s clock.
- USELOCKING
specifies that computational threads share a common weight vector and update weight vector without race conditions. If you omit this option, computational threads update a single weight vector simultaneously. This causes intentional race conditions and nondeterministic behavior but increases performance significantly.
- SGD <sgd-options>
-
specifies the plain stochastic gradient descent (SGD) algorithm.
You can specify the following sgd-options:
- ANNEALINGRATE=number
-
specifies the annealing parameter,
. Annealing is a way to automatically reduce the learning rate as the algorithm progresses, producing smaller steps as the algorithm approaches a solution. Effectively, it replaces the learning rate parameter, , with where t is the number of iterations that the algorithm has performed.
By default, ANNEALINGRATE=1.0E–6. The number must be a nonnegative double.
- COMMFREQ=number
specifies the number of minibatches that each computational thread processes before weights are synchronized across all threads and nodes.
- LEARNINGRATE=number
-
specifies the learning rate parameter,
, for the algorithm. New iterates for the algorithm are found by using where
is the current weight vector, is the new weight vector, is the minibatch used during iteration k, and is the loss associated with the ith observation. If you see a huge objective value from the algorithm, especially for a small data set, it is likely that the learning rate is set too high.
By default, LEARNINGRATE=0.001. The number must be a nonnegative double.
- MINIBATCHSIZE=number
-
specifies the size of the minibatches to use in the algorithm.
By default, MINIBATCHSIZE=10.
- MOMENTUM=number
-
specifies the value for momentum. The number must be greater than or equal to 0 and less than or equal to 1.
By default, MOMENTUM=0.
- SEED=number
specifies the seed for random access of observations on each thread for the algorithm. If number is less than or equal to 0 or not specified, a random seed is generated by reading the time of day from the computer’s clock.
- USELOCKING
specifies that computational threads share a common weight vector and update the weight vector without race conditions. If you omit this option, computational threads update a single weight vector simultaneously. This causes intentional race conditions and nondeterministic behavior but increases performance significantly.
By default, ALGORITHM=ADAM.
- MAXITER=number
-
specifies the iteration budget for training. When ALGORITHM=SGD or ADAM, number specifies the desired number of training epochs.
By default, MAXITER=250.
- MAXTIME=number
-
specifies the maximum time (in seconds) allowed for optimization, where number is greater than or equal to 1. When this value is reached, the optimization terminates the search and returns results. When MAXTIME=0, no maximum time is set.
By default, MAXTIME=0.
- REGL1=number
-
specifies the L1 regularization parameter
for the model loss function. The number must be nonnegative. Note that this value is autotuned when you specify the AUTOTUNE statement. By default, REGL1=0.
- REGL2=number
-
specifies the L2 regularization parameter
. The number must be nonnegative. Note that this value is autotuned when you specify the AUTOTUNE statement. By default, REGL2=0.