When processing through multi-process, I would like to continue processing after catching exceptions.

I would like to use Python's multiprocessing.Pool to process them in parallel as shown in the code below.

import multiprocessing

def foo(i):
    if i%2 == 0:
        raise RuntimeError ("even")
    returni

with multiprocessing.Pool(4)asp:
    result=p.map (foo, [1,3,5])
    print(result)
    # =>[1,3,5]

Pass [1,2,3] to the map function to throw a RuntimeError when i=2 as follows:

 with multiprocessing.Pool(4)asp:
    result=p.map (foo, [1,2,3])

RemoteTraceback: 
"""
Traceback (most recent call last):
  File"/home/vagrant/.pyenv/versions/3.9.7/lib/python 3.9/multiprocessing/pool.py", line 125, inworker
    result=(True, func(*args,**kwds))
  File"/home/vagrant/.pyenv/versions/3.9.7/lib/python 3.9/multiprocessing/pool.py", line48, in mapstar
    return list(map(*args))
  File "<ipython-input-102-bbf59efb0d82>", line 3, info
    raise RuntimeError ("even")
RuntimeError—Even number
"""

I want to catch the exception when i=2, and then get the result when i=1, i=3.What kind of code should I write?

Try/except in the foo function, as shown in the following code, to get results when i=1, i=3.
However, since try/except should originally be done by the caller of the foo function, I would prefer not to adopt this method.


def foo(i):
    try:
        if i%2 == 0:
            raise RuntimeError ("custom")
        returni
    except Exception
        return None

python

2022-09-30 22:02

1 Answers

Why don't you use the imap method to retrieve the iterator and proceed with the iterator while handling the exceptions?

import multiprocessing

def foo(i):
    if i%2 == 0:
        raise RuntimeError(f "even: {i}")
    returni

if__name__=='__main__':
    with multiprocessing.Pool(4)asp:
        result_iterator=p.imap(foo, [1,2,3])
        while True:
            try:
                result=next(result_iterator)
                print(result)
            exceptStopIteration:
                break
            except Exception as:
                print(e)

2022-09-30 22:02

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656