Python 捕获多进程 OR 多线程 error 的最佳实现是什么？

def test():
    a=[2]
    print(a[3])

def run():
    test()

def err(error):
    logger.exception(error)

if __name__ == '__main__':
    pool = Pool(2)
    t1 = pool.apply_async(run,error_callback=err)
    pool.close()
    pool.join()

日志记录 :

list index out of range

显然并没有拿到堆栈信息,在 callback 里再加上 try

def test():
    a=[2]
    print(a[3])

def run():
    test()

def err(error):
    #这边已经拿到 error 了但是还要 try，要不然日志只记录 list index out of range
    try:
        raise error
    except Exception as e:
        logger.exception(e)

if __name__ == '__main__':
    pool = Pool(2)
    t1 = pool.apply_async(run,error_callback=err)
    pool.close()
    pool.join()

日志输出 :

list index out of range
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/work_dir/test/timer.py", line 13, in run
    test()
  File "/home/work_dir/test/timer.py", line 10, in test
    print(a[3])
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/work_dir/test/timer.py", line 17, in err
    raise error
IndexError: list index out of range

难道要在回调函数里再次捕获输出？

Supplement 1 · Jan 7, 2020

用了个简单的办法

from multiprocessing.pool import rebuild_exc
from multiprocessing import pool

class My_Except(pool.ExceptionWithTraceback):
    def __init__(self, exc, tb):
        logger.exception(tb)
        return super().__init__(exc,tb)

pool.ExceptionWithTraceback = My_Except

在包的__init__.py中打一个猴子补丁，这样就可以抓到堆栈，然后写日志，或刷新状态.

然后在代码中导入__init__.py,也不需要回调函数直接就拿到堆栈了

from multiprocessing import Pool
import py_mutilprocessing

def test():
    a=[2]
    print(a[3])

def run():
    test()

if __name__ == '__main__':
    pool = Pool(2)
    t1 = pool.apply_async(run,)
    pool.close()
    pool.join()

日志:

<traceback object at 0x7ff050695988>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/tarena/work_dir/py_mutilprocessing/timer.py", line 9, in run
    test()
  File "/home/tarena/work_dir/py_mutilprocessing/timer.py", line 6, in test
    print(a[3])
IndexError: list index out of range

但是觉得这么写怪怪的~~~~

10 replies • 2020-05-19 16:29:07 +08:00

BingoXuan

Jan 7, 2020

多进程和多线程通讯的问题，内置有 queue 之类的数据类型可以使用。

wuwukai007

Jan 7, 2020

@BingoXuan 不是通信，是抓到子进程子线程错误，输出到日志或者写状态等，但是拿不到堆栈信息。我现在是再次 try 或者是用装饰器来做

zyqzyq08

Jan 7, 2020

可以考虑直接重写 sys.stderr

BingoXuan

Jan 7, 2020

你试图捕抓子进程或子线程信息本质就是通讯。为什么一定要主进程去干着活呢？你完全可以让子进程或子线程运行时候去捕捉啊。装饰器的其中一个广泛用途就是捕获并处理错误信息。如果子进程是一个 worker，那么子进程 worker 接受主进程的请求运行指定函数，记录运行结果或捕获运行错误，并通过 socket 或文件通知或保存信息。

Vegetable

Jan 7, 2020

traceback.format_exc()是你想要的?

oahebky

Jan 7, 2020

了解一下 traceback 库或许你就有答案了。

wuwukai007

Jan 7, 2020

@Vegetable traceback.format_exc() 是主进程中错误会抛出，用的进程池或线程池，除非用装饰器包起来，要不然子进程错误不会抛出来的，比如我现在用的回调，用这个直接输出 NoneType

todd7zhang

Jan 7, 2020

子进程的错误, 就让子进程抛呗

wuwukai007

Jan 7, 2020

@todd7zhang 子进程要捕获，之前用的装饰器把子进程整个包起来可以拿到 error，但是这样感觉很浪费，应为进程池本身就能拿到 error，这样多此一举了

hiyoi

May 19, 2020

最近刚好碰到这个情况，我是这么解决的，把要子线程和日志对象封装到一起，使用 logger.exception()来 traceback 错误

```
from concurrent.futures import ThreadPoolExecutor
import logging

class Main(object):
def __init__(self, logger=None):
self.pool = ThreadPoolExecutor
self.logger = logger or self._get_logger(__name__)

def _get_logger(self, name):
logger = logging.getLogger(name)
handler = logging.StreamHandler()
formatter = logging.Formatter(
"%(asctime)s - %(levelname)s - %(threadName)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
return logger

def do_something(self):
try:
a = 1 / 0
except Exception as e:
self.logger.exception(e)

def async_task(self):
with self.pool() as p:
for i in range(3):
p.submit(self.do_something)

if __name__ == '__main__':
m = Main()
m.async_task()
```

日志：
```
2020-05-19 16:23:38,378 - ERROR - ThreadPoolExecutor-0_0 - division by zero
Traceback (most recent call last):
File "thread.py", line 22, in do_something
a = 1 / 0
ZeroDivisionError: division by zero
2020-05-19 16:23:38,378 - ERROR - ThreadPoolExecutor-0_0 - division by zero
Traceback (most recent call last):
File "thread.py", line 22, in do_something
a = 1 / 0
ZeroDivisionError: division by zero
2020-05-19 16:23:38,378 - ERROR - ThreadPoolExecutor-0_2 - division by zero
Traceback (most recent call last):
File "thread.py", line 22, in do_something
a = 1 / 0
ZeroDivisionError: division by zero
```