Question about Python multiprocessing.

Asked 2 years ago, Updated 2 years ago, 137 views

a=[1,2,3,4,5]
b=[]

for c in a:
    b.append(c)

print(b) # Output is [1,2,3,4,5]

I'm trying to execute the above code quickly using multiprocessing.

from multiprocessing import Pool

a=[1,2,3,4,5]
def abc(a):
    b=[]
    b.append(a)
    print(b)

if __name__=='__main__':
    pool = Pool(processes=6) 
    pool.map(abc,a) 

If you change it like this, the output value doesn't go [1,2,3,4,5] [1] [2] [3] [4] [5] will be.

When using the for statement, you could take out the list of b from the repeat door and add it to the existing list of b whenever it was repeated

To use multiprocessing, I'm worried that I can't subtract the conjugate of b from the function abc(a).

Is there a way to make the output come out like the top code while using multiprocessing?

python3 multiprocessing pool

2022-09-22 20:05

1 Answers

Multiprocessing modules are not easy to use.

Multiprocessing is not an easy subject.

It should not be done like normal Python programming.

Since we used the map, we divide the work by making five processes with the elements of a as a factor.

That is, you send 1, add one to the b list, and you repeat 5 times like this, so each list comes with [1] [2] [3] [4] [5].

You must use shared memory for the desired operation.

The problem is that since the data types available in shared memory are limited to cterput, only Value and Array are provided by default.

Of course, if you have knowledge of C language, you can use the ctypes module to expand it in various ways.

Please refer to the example below https://docs.python.org/3.7/library/multiprocessing.html?highlight=process#sharing-state-between-processes

Please be familiar with the documentation in the link above.

from multiprocessing import Pool, Array

a=[1,2,3,4,5]
def abc(a):
    arr[a - 1] = a

if __name__=='__main__':
    arr = Array('i', 5)    # multiprocessing.array that can hold 5 ints of sharedctypes.c_int_Array_5c
    pool = Pool(processes=6) 
    pool.map(abc, a) 
    print(arr[:])

[1, 2, 3, 4, 5]

[1] [2] [3] [4] [5]


2022-09-22 20:05

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.