There's more...

It is often recommended that you start with a high value of learning rate and reduce it as the learning progresses. This helps in fine-tuning the training. We can use the TensorFlow tf.train.exponential_decay method to achieve this. According to the TensorFlow docs:

When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies an exponential decay function to a provided initial learning rate. It requires a global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.The function returns the decayed learning rate.
Args:
learning_rate: A scalar float32 or float64 Tensor or a Python number. The initial learning rate.
global_step: A scalar int32 or int64 Tensor or a Python number. Global step to use for the decay computation. Must not be negative.
decay_steps: A scalar int32 or int64 Tensor or a Python number. Must be positive. See the decay computation described earlier.
decay_rate: A scalar float32 or float64 Tensor or a Python number. The decay rate.
staircase: Boolean. If True decay the learning rate at discrete intervals
name: String. Optional name of the operation. Defaults to 'ExponentialDecay'.
Returns:
A scalar Tensor of the same type as learning_rate. The decayed learning rate.

To implement an exponentially decaying learning rate, consider the following code example:

global_step = tf.Variable(0, trainable = false)
initial_learning_rate = 0.2
learning_rate = tf.train.exponential_decay(initial_learning_rate, global_step, decay_steps=100000, decay_rate=0.95, staircase=True)
# Pass this learning rate to optimizer as before.