A LearningRateSchedule that uses a cosine decay schedule with restarts
learning_rate_schedule_cosine_decay_restarts(
initial_learning_rate,
first_decay_steps,
t_mul = 2,
m_mul = 1,
alpha = 0,
...,
name = NULL
)
A scalar float32
or float64
Tensor or an R
number. The initial learning rate.
A scalar int32
or int64
Tensor
or an R
number. Number of steps to decay over.
A scalar float32
or float64
Tensor
or an R number. Used
to derive the number of iterations in the i-th period.
A scalar float32
or float64
Tensor
or an R number. Used
to derive the initial learning rate of the i-th period.
A scalar float32
or float64
Tensor or an R number. Minimum
learning rate value as a fraction of the initial_learning_rate.
For backwards and forwards compatibility
String. Optional name of the operation. Defaults to 'SGDRDecay'.
See Loshchilov & Hutter, ICLR2016, SGDR: Stochastic Gradient Descent with Warm Restarts.
When training a model, it is often useful to lower the learning rate as
the training progresses. This schedule applies a cosine decay function with
restarts to an optimizer step, given a provided initial learning rate.
It requires a step
value to compute the decayed learning rate. You can
just pass a TensorFlow variable that you increment at each training step.
The schedule is a 1-arg callable that produces a decayed learning rate when passed the current optimizer step. This can be useful for changing the learning rate value across different invocations of optimizer functions.
The learning rate multiplier first decays
from 1 to alpha
for first_decay_steps
steps. Then, a warm
restart is performed. Each new warm restart runs for t_mul
times more
steps and with m_mul
times initial learning rate as the new learning rate.
You can pass this schedule directly into a keras Optimizer
as the learning_rate
.