Memory Error in jupyter notebook

I take in a lot of image data and analyze it on jupyter notebook, but I delete the variables with the del command and gc.collect() in some parts of the code, but I run out of memory in the middle and stop.

I've shown the memory occupied by the variables and deleted all the memory-intensive variables, but I'm still using about half the memory.

This won't happen if I run it as a .py file, but is there something cachey in the jupyter notebook?

Please teach me.

jupyter-notebook memory-management ipython

2022-09-29 22:02

1 Answers

In addition to variables, there are other things that are cached in the Jupiter Notebook.Output (Out) is definitely cached.If you output a variable, you are likely to cache it, and if you output a larger one, it will remain in the cache, so you are likely to run out of memory.

For example, on the Jupiter Notebook, you can use the package memory_profiler to measure memory consumption. If memory_profiler has not been installed, install it first.

 pip install memory_profiler

Memory can be recovered by executing code similar to the following:

%load_ext memory_profiler
import numpy as np
import gc

a=np.random.rand(10000,10000)
%memit
dela
gc.collect()
%memit

peak memory: 839.47MiB, increment: 0.22MiB
peak memory: 76.54MiB, increment: 0.02MiB

However, if you output it to Out: as follows,

 a=np.random.rand(10000,10000)
a

array([0.23041043, 0.88022318, 0.61961303, ..., 0.23188055, 0.03481917,
    0.92450332],
   [0.8104011 , 0.52135031, 0.25772234, ..., 0.90955947, 0.64602805,
    0.10762479],
   [0.55358733, 0.50758164, 0.68215301, ..., 0.45746926, 0.43422664,
    0.24862533],
   ...,
   [0.11373284, 0.10500561, 0.1978364 , ..., 0.75755749, 0.18117871,
    0.3339833 ],
   [0.87190469, 0.54811619, 0.74330171, ..., 0.96712544, 0.30823596,
    0.13202881],
   [0.29832023, 0.6195654 , 0.34837866, ..., 0.51810623, 0.98901862,
    0.99977871]])

Deleting variables in the same way does not reduce memory consumption.

%memit
dela
gc.collect()
%memit

peak memory: 839.91 MiB, increment: 0.02 MiB
peak memory: 839.91 MiB, increment: 0.00 MiB

In the case of the Jupiter Notebook, compared to running the .py file, there are many cases where it is cached in memory, and I think you can guess where the problem is by using the memory_profiler to track memory consumption.

2022-09-29 22:02

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656