I use Spyder in Python 3.
Is there a way to save and invoke variables of huge size that are obtained by executing code somewhere?
The background is a loop statement that returns millions of lines of results, and of course the variable disappears every time I change a file, and I run this loop statement again when I start over.This loop statement that has already been executed and the results are known has been taken up for an hour or two every time.
Python has a module that can store objects called pickles (pickles translated into Japanese) at high speed.
data
allows you to easily save and read the following code:
· Official documentation pickle
import pickle
with open('data.pickle', 'wb') as f:
pickle.dump(data,f,pickle.HIGHEST_PROTOCOL)
with open('data.pickle','rb') asf:
data=pickle.load(f)
If the variable of a large size is Pandas' DataFrame, it is easier to save and load in a pickle.
df.to_pickle('data.pickle')
df = pd.read_pickle('data.pickle')
In the case of numpy ndarray, it is convenient to use it because it can be saved in binary format.If you try using the code below, you can save it in seconds and load it in less than a second.If you use a pickle, the processing time is about the same.
import numpy as np
na=np.random.rand (10000000,10)
np.save ('data.npy', na)
# If you want to compress it
# np.savez ('data.npz', na)
na=np.load('data.npy')
© 2024 OneMinuteCode. All rights reserved.