问题描述
我正在尝试使用 multiprocessing 的 Pool.map() 函数同时划分工作.当我使用以下代码时,它工作正常:
I'm trying to use multiprocessing's Pool.map() function to divide out work simultaneously. When I use the following code, it works fine:
import multiprocessing def f(x): return x*x def go(): pool = multiprocessing.Pool(processes=4) print pool.map(f, range(10)) if __name__== '__main__' : go()
但是,当我在更面向对象的方法中使用它时,它就不起作用了.它给出的错误信息是:
However, when I use it in a more object-oriented approach, it doesn't work. The error message it gives is:
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
当以下是我的主程序时会发生这种情况:
This occurs when the following is my main program:
import someClass if __name__== '__main__' : sc = someClass.someClass() sc.go()
以下是我的 someClass 类:
import multiprocessing class someClass(object): def __init__(self): pass def f(self, x): return x*x def go(self): pool = multiprocessing.Pool(processes=4) print pool.map(self.f, range(10))
任何人都知道问题可能是什么,或者解决它的简单方法?
Anyone know what the problem could be, or an easy way around it?
推荐答案
问题是多处理必须腌制东西以将它们吊在进程之间,并且绑定的方法是不可腌制的.解决方法(无论您是否认为它容易";-)是将基础结构添加到您的程序中以允许对此类方法进行腌制,并使用 copy_reg 标准库方法.
The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it "easy" or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method.
例如,Steven Bethard 对 此线程(接近线程的末尾)显示了一种完全可行的方法,允许通过 copy_reg 进行方法酸洗/取消酸洗.
For example, Steven Bethard's contribution to this thread (towards the end of the thread) shows one perfectly workable approach to allow method pickling/unpickling via copy_reg.