Resource exhausted:understanding OOM when allocating tensor errors

2019-09-09 08:03:13.567901: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ****************************************************************************************************
2019-09-09 08:03:13.568544: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at fused_batch_norm_op.cc:487 : Resource exhausted: OOM when allocating tensor with shape[32,512,20,20] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "train.py", line 195, in <module>
    _main()
  File "train.py", line 89, in _main
    callbacks=[logging, checkpoint, reduce_lr, early_stopping])
  File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py", line 217, in fit_generator
    class_weight=class_weight)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1217, in train_on_batch
    outputs = self.train_function(ins)
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File"/usr/local/lib/python 3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2675, in_call
    fetched=self._callable_fn(*array_vals)
  File"/usr/local/lib/python 3.6/dist-packages/tensorflow/python/client/session.py", line 1458, in__call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.ResourceExhaustedError:2rooterror(s)found.
  (0) Resource exhausted: OOM when allocating sensor with shape [32,512,20,20] and type float on/job: localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{nodebatch_normalization_41/FusedBatchNorm}]]
Hint: If you want to see a list of allocated sensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[loss_1/add_74/_5299]]
Hint: If you want to see a list of allocated sensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating sensor with shape [32,512,20,20] and type float on/job: localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{nodebatch_normalization_41/FusedBatchNorm}]]
Hint: If you want to see a list of allocated sensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.