There is a CPU-bound parallel task, so I would like to synchronize the barrier in the middle, add the aggregation task, and then resume the processing
||
A1 B1 C1
↓ ↓ ↓
+---+---+---+ barrier synchronization
| | | |
A2 B2 C2 M2
↓ ↓ ↓
+---+---+---+ barrier synchronization
| | | |
A3 B3 C3 M3
Exactly
A2
data created in A1
B2
data created in B1
C2
C1 created data
Continue using
I just want to add another aggregation task when A1 B1 C1 is all over.
A2 B2 C2 does not have to wait for other termination
There is no difference in the task granularity of the ABC, so if you can easily realize it, you can wait a little longer
If it's multi-threaded, you can use shared variables, so there are many ways to do it as soon as you use semaphores.
Python threads are not native OS
It's an event loop, and it doesn't seem to use multiple CPUs
If it's a process, it seems to bind to multiple CPUs.
The memory space will be separated, so besides handing it over with the first and last args,
I don't know how to share information in the middle
You can wait for the end of the subprocess + the JOIN of the main process, but
If you exit the process, you will not be able to use A1 data even if you create a new A2 because you do not share memory space with A1
Of course, you should have a counter for external services such as file systems and databases.
Increment the counters in each process
I think I can do it if I keep an eye on the counter in M's task.
Is there a good way to achieve a light synchronization as the system becomes complicated?
I would appreciate it if you could tell me how to use multiple CPUs in Python threads in the first place.
python
There are several multi-processes in Python, and the standard ones seem to be as follows.
In multiprocessing, Queue, Pipe, Lock, Shared memory are available, and the following statements are also available
manager objects.
See Manager for more information.
Threads in Python…CPython are OS threads and can run on multiple CPUs.Because of the GIL, Python processing does not run at the same time, but for example, NumPy calculations and Python itself should run.
(Additional information about Thread)
threading.get_native_id() reads as follows (where you can see that the OS-native thread IDs are different from each other)
Return the native integral Thread ID of the current thread assigned by the kernel.
(By the way, the process ID can be obtained by os.getpid())
If the normal processing is set to Thread, it will be overhead and slightly slow.
When the thread side becomes slightly faster due to NumPy processing (checked by colab)
(primesfrom2to is from SO)
import concurrent.futures
import numpy as np
def primesfrom2to(n):
""" Input n>=6, Returns a array of primes, 2<=p<n""
save=np.ones (n//3+(n%6==2), dtype=bool)
for i in range (1, int(n**0.5)//3+1):
if save [i]:
k = 3*i+1 | 1
save [k*k//3::2*k] = False
save [k*(k-2*(i&1)+4)//3::2*k] = False
return np.r_[2,3,(3*np.nonzero(sieve)[0][1:]+1)|1)]
def primesum (num):
return primesfrom2to(num).sum()
defmain(flag):
nums = [70000000, 80000000, 90000000, 100000000, 110000000, 120000000]
if flag:
print('normal:')
for n in nums:
print(primesum(n))
else:
print('thread:')
with concurrent.futures.ThreadPoolExecutor() as executor:
for number in executor.map (primesum, nums):
print(number)
© 2024 OneMinuteCode. All rights reserved.