lr_shedule

cosine_decay

Learning rate cosine decay function (using half of cosine curve).

This method is useful for scheduling learning rates which oscillate over time:

s = fe.schedule.LRScheduler(model=model, lr_fn=lambda step: cosine_decay(step, cycle_length=3750, init_lr=1e-3))
fe.Estimator(..., traces=[s])

Parameters:

Name	Type	Description	Default
time	int	The current step or epoch during training starting from 1.	required
cycle_length	int	The decay cycle length.	required
init_lr	float	Initial learning rate to decay from.	required
min_lr	float	Minimum learning rate.	1e-06
start	int	The step or epoch to start the decay schedule.	1

Returns:

Name	Type	Description
lr		learning rate given current step or epoch.

Source code in fastestimator\fastestimator\schedule\lr_shedule.py

def cosine_decay(time: int, cycle_length: int, init_lr: float, min_lr: float = 1e-6, start: int = 1):
    """Learning rate cosine decay function (using half of cosine curve).

    This method is useful for scheduling learning rates which oscillate over time:
    ```python
    s = fe.schedule.LRScheduler(model=model, lr_fn=lambda step: cosine_decay(step, cycle_length=3750, init_lr=1e-3))
    fe.Estimator(..., traces=[s])
    ```

    Args:
        time: The current step or epoch during training starting from 1.
        cycle_length: The decay cycle length.
        init_lr: Initial learning rate to decay from.
        min_lr: Minimum learning rate.
        start: The step or epoch to start the decay schedule.

    Returns:
        lr: learning rate given current step or epoch.
    """
    if time < start:
        lr = init_lr
    else:
        step_in_cycle = (time - start) % cycle_length / cycle_length
        lr = (init_lr - min_lr) / 2 * math.cos(step_in_cycle * math.pi) + (init_lr + min_lr) / 2
    return lr