Python Memory Leaks

Asked 2 years ago, Updated 2 years ago, 16 views

I wrote a program for data aggregation in Python.
I'd like to put in about 5000 input images.
Memory usage keeps increasing as you run it.
As memory leaks are suspected, I tried garbage collection but could not resolve them.
I would like to ask for your advice on how to solve this problem.
I look forward to your kind cooperation.

import tensorflow as tf
import numpy as np
from PIL import Image
import glob
importos

import configparser

infile=configparser.ConfigParser()
infile.read ('./config.ini', 'UTF-8-SIG')
in_image=glob.glob(inifile.get('directory','in_dir')+'/*')
in_fileName = os.listdir(inifile.get('directory','in_dir')

#<<debug>>
# print(in_image)
# print(in_fileName)
print(len(in_image))

for num in range (len(in_image)) :
    image=tf.read_file(str(in_image[num]))
    direct, filename = os.path.split(str(in_image[num]))
    name,ext=os.path.splitext(filename)

    print('direct:{}, file:{}.format(direct, filename))

    if image is None:
        print("Not Open")
        continue

    with tf.Session() asess:
        code=tf.train.Coordinator()
        threads=tf.train.start_queue_runners(code=code)
        for i in range (5):

            if int(inifile.get('format','JPG'))==1:
                tmp_img=tf.cast(tf.image.decode_jpeg(image,channels=3), tf.float32)
            elifint(inifile.get('format', 'BMP'))) == 1:
                tmp_img=tf.cast(tf.image.decode_image(image,channels=3), tf.float32)

            if int(inifile.get('data_arg_flip','flag_flip')))==1:
                tmp_img = tf.image.random_flip_left_right(tmp_img)

            if int(inifile.get('data_arg_brightnes', 'flag_brightness')))==1:
                tmp_img=tf.image.random_brightness(tmp_img,max_delta=int(inifile.get('data_arg_brightnes','max_delta'))))

            if int(inifile.get('data_arg_contrast','flag_contrast')))==1:
                tmp_img=tf.image.random_contrast(tmp_img, lower=float(inifile.get('data_arg_contrast','lower')), upper=float(inifile.get('data_arg_contrast','upper')        

            if int(inifile.get('data_arg_rot','flag_rot')))==1:
                tmp_img=tf.image.rot90(tmp_img,k=i)

            out_img = sess.run(tmp_img)
#            print(img.shape)
#            print(img)

            if int(inifile.get('format','JPG'))==1:
                Image.fromarray(np.uint8(out_img)) .save(inifile.get('directory','out_dir')+'/'+name+'_{0:03d}.jpg'.format(i))
            elifint(inifile.get('format', 'BMP'))) == 1:
                Image.fromarray(np.uint8(out_img)) .save(inifile.get('directory','out_dir')+'/'+name+'_{0:03d}.bmp'.format(i))

ini file

#file directory
directory
in_dir=./original_image
out_dir=./out_image

# data format
[format]
JPG = 0
BMP = 1

# arg_settings
[data_arg_set]
arg_num = 4

# function settings
[data_arg_flip]
flag_flip = 0

[data_arg_brightnes]
flag_brightness=1
max_delta = 1
# max —63

[data_arg_contrast]
flag_contrast=1
lower = 0.8
upper = 1.0
# min —0.1
# max: 2.0

[data_arg_rot]
flag_rot = 0

python

2022-09-30 16:00

1 Answers

I have never used a library of codes like tf.train.Coordinator(), but
Judging from the code rules, is it some kind of class?
This code is sequenced in for num in range(len(in_image)): and has been called many times.python does not overwrite new instances in the loop and creates many new instances in memory until the sequence ends.
I have experienced the same experience in the while loop of pygame, so I think this is the reason.Therefore, if you want to create an instance once, you'd better run it out of the loop.
Especially in this case, it would be better to do so.


2022-09-30 16:00

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.