if __name__ == '__main__':
pool = Pool(processes=8)
pool.map(loop1, range(3,10003))
pool.close()
pool.join()
Is it right that the eight processes do the same thing over and over again? For example, is it right to calculate the same by substituting all processes 1 to 8 from 3 to 10002?
I'm asking you this question because I've tried it. If it is correct, is there a way not to use arguments that have already been used between processes?
python pool multiprocessing map
No, divide the numbers from 3 to 10002 into the worker process and run them.
Let's look at the example below.
from multiprocessing import Pool
from collections import Counter
import multiprocessing as mp
import time
def f(x):
current = mp.current_process()
time.sleep(0.0001)
return (current.name, x * x)
if __name__ == "__main__":
with Pool(processes=8) as pool:
result = pool.map(f, range(100))
print(result)
print(Counter([e[0] for e in result]))
I simply changed the phrase with to just make a pool and close it. (This is a slightly modified code using only the map part in the Python documentation.) To run 0-99, divide it into 8 processes and see if it really does, multiprocessing.current_process() is used to find out the worker process and return its name.
In the last print statement, count the number of worker process names among the results by Counter.
The results were as follows:
[('SpawnPoolWorker-3', 0), ('SpawnPoolWorker-3', 1), ('SpawnPoolWorker-3', 4), ('SpawnPoolWorker-3', 9), ('SpawnPoolWorker-1', 16), ('SpawnPoolWorker-1', 25), ('SpawnPoolWorker-1', 36), ('SpawnPoolWorker-1', 49), ('SpawnPoolWorker-1', 64), ('SpawnPoolWorker-1', 81), ('SpawnPoolWorker-1', 100), ('SpawnPoolWorker-1', 121), ('SpawnPoolWorker-3', 144), ('SpawnPoolWorker-3', 169), ('SpawnPoolWorker-3', 196), ('SpawnPoolWorker-3', 225), ('SpawnPoolWorker-8', 256), ('SpawnPoolWorker-8', 289), ('SpawnPoolWorker-8', 324), ('SpawnPoolWorker-8', 361), ('SpawnPoolWorker-1', 400), ('SpawnPoolWorker-1', 441), ('SpawnPoolWorker-1', 484), ('SpawnPoolWorker-1', 529), ('SpawnPoolWorker-3', 576), ('SpawnPoolWorker-3', 625), ('SpawnPoolWorker-3', 676), ('SpawnPoolWorker-3', 729), ('SpawnPoolWorker-8', 784), ('SpawnPoolWorker-8', 841), ('SpawnPoolWorker-8', 900), ('SpawnPoolWorker-8', 961), ('SpawnPoolWorker-1', 1024), ('SpawnPoolWorker-1', 1089), ('SpawnPoolWorker-1', 1156), ('SpawnPoolWorker-1', 1225), ('SpawnPoolWorker-3', 1296), ('SpawnPoolWorker-3', 1369), ('SpawnPoolWorker-3', 1444), ('SpawnPoolWorker-3', 1521), ('SpawnPoolWorker-8', 1600), ('SpawnPoolWorker-8', 1681), ('SpawnPoolWorker-8', 1764), ('SpawnPoolWorker-8', 1849), ('SpawnPoolWorker-3', 1936), ('SpawnPoolWorker-3', 2025), ('SpawnPoolWorker-3', 2116), ('SpawnPoolWorker-3', 2209), ('SpawnPoolWorker-1', 2304), ('SpawnPoolWorker-1', 2401), ('SpawnPoolWorker-1', 2500), ('SpawnPoolWorker-1', 2601),
('SpawnPoolWorker-8', 2704), ('SpawnPoolWorker-8', 2809), ('SpawnPoolWorker-8', 2916), ('SpawnPoolWorker-8', 3025), ('SpawnPoolWorker-5', 3136), ('SpawnPoolWorker-5', 3249), ('SpawnPoolWorker-5', 3364), ('SpawnPoolWorker-5', 3481), ('SpawnPoolWorker-1', 3600), ('SpawnPoolWorker-1', 3721), ('SpawnPoolWorker-1', 3844), ('SpawnPoolWorker-1', 3969), ('SpawnPoolWorker-3', 4096), ('SpawnPoolWorker-3', 4225), ('SpawnPoolWorker-3', 4356), ('SpawnPoolWorker-3', 4489), ('SpawnPoolWorker-8', 4624), ('SpawnPoolWorker-8', 4761), ('SpawnPoolWorker-8', 4900), ('SpawnPoolWorker-8', 5041), ('SpawnPoolWorker-1', 5184), ('SpawnPoolWorker-1', 5329), ('SpawnPoolWorker-1', 5476), ('SpawnPoolWorker-1', 5625), ('SpawnPoolWorker-3', 5776), ('SpawnPoolWorker-3', 5929), ('SpawnPoolWorker-3', 6084), ('SpawnPoolWorker-3', 6241), ('SpawnPoolWorker-5', 6400), ('SpawnPoolWorker-5', 6561), ('SpawnPoolWorker-5', 6724), ('SpawnPoolWorker-5', 6889), ('SpawnPoolWorker-8', 7056), ('SpawnPoolWorker-8', 7225), ('SpawnPoolWorker-8', 7396), ('SpawnPoolWorker-8', 7569), ('SpawnPoolWorker-1', 7744), ('SpawnPoolWorker-1', 7921), ('SpawnPoolWorker-1', 8100), ('SpawnPoolWorker-1', 8281), ('SpawnPoolWorker-5', 8464), ('SpawnPoolWorker-5', 8649), ('SpawnPoolWorker-5', 8836), ('SpawnPoolWorker-5', 9025), ('SpawnPoolWorker-3', 9216), ('SpawnPoolWorker-3', 9409), ('SpawnPoolWorker-3', 9604), ('SpawnPoolWorker-3', 9801)]
Counter({'SpawnPoolWorker-3': 32, 'SpawnPoolWorker-1': 32, 'SpawnPoolWorker-8': 24, 'SpawnPoolWorker-5': 12})
SpawnPoolWorker-3, 1, 8, 5 4 walkers divided 100 into 32, 32, 24, and 12 respectively. It was coded to be performed with 8 worker processes, but the function did not take long, so it seems that only 4 workers were properly created and performed.
In fact, the time of the f function.If you don't give me sleep, I can see the result of making only one worker and performing everything. If you do time.sleep a little longer, or if you increase the range a little more, you can also see the results of all eight workers.
© 2024 OneMinuteCode. All rights reserved.