in deep learning, a method of making learning progress even when layers are deep

Currently, we use a model that combines CNN and RNN to categorize videos.
Since the model is underfitting, I would like to increase the complexity of the model.
However, CNN currently has about four layers, but if we try to increase the number of layers further, learning will stagnate at the initial stage and training errors will not decrease at all.
What can be considered as a way to proceed with effective learning in such cases?
I've heard that batch normalization is enabled, but due to GPU memory, batch size is limited to 1, and batch normalization is probably not available.
I would appreciate it if you could tell me how to proceed with learning in a stable manner other than that.