问题描述
我已经定义了这个函数
def writeonfiles(a,seed): random.seed(seed) f = open(a, "w+") for i in range(0,10): j = random.randint(0,10) #print j f.write(j) f.close()
其中 a 是包含文件路径的字符串,seed 是整数种子.我想以这样一种方式并行化一个简单的程序,即每个内核采用我提供的可用路径之一,为其随机生成器播种并在该文件上写入一些随机数,例如,如果我通过向量
Where a is a string containing the path of the file and seed is an integer seed. I want to parallelize a simple program in such a way that each core takes one of the available paths that I give in, seeds its random generator and write some random numbers on that files, so, for example, if I pass the vector
vector = [Test/file1.txt, Test/file2.txt]
和种子
seeds = (123412, 989898),
它为第一个可用的核心提供功能
it gives to the first available core the function
writeonfiles(Test/file1.txt, 123412)
对于第二个具有不同参数的相同函数:
and to the second one the same function with different arguments:
writeonfiles(Test/file2.txt, 989898)
我在 Stackoverflow 上查看了很多类似的问题,但我无法做出任何解决方案.我尝试的是:
I have looked through a lot of similar questions here on Stackoverflow, but I cannot make any solution work. What I tried is:
def writeonfiles_unpack(args): return writeonfiles(*args) if __name__ == "__main__": folder = ["Test/%d.csv" %i for i in range(0,4)] seed = [234124, 663123, 12345 ,123833] p = multiprocessing.Pool() p.map(writeonfiles, (folder,seed))
并给我 TypeError: writeonfiles() 正好需要 2 个参数(1 个给定).
and gives me TypeError: writeonfiles() takes exactly 2 arguments (1 given).
我也试过了
if __name__ == "__main__": folder = ["Test/%d.csv" %i for i in range(0,4)] seed = [234124, 663123, 12345 ,123833] p = multiprocessing.Process(target=writeonfiles, args= [folder,seed]) p.start()
但它给了我
种子文件/usr/lib/python2.7/random.py",第 120 行超级(随机,自我).seed(一)TypeError: unhashable type: 'list'
But it gives me
File "/usr/lib/python2.7/random.py", line 120, in seed
super(Random, self).seed(a)
TypeError: unhashable type: 'list'
最后,我尝试了上下文管理器
Finally, I tried the contextmanager
@contextmanager def poolcontext(*args, **kwargs): pool = multiprocessing.Pool(*args, **kwargs) yield pool pool.terminate() if __name__ == "__main__": folder = ["Test/%d" %i for i in range(0,4)] seed = [234124, 663123, 12345 ,123833] a = zip(folder, seed) with poolcontext(processes = 3) as pool: results = pool.map(writeonfiles_unpack,a )
它会导致文件/usr/lib/python2.7/multiprocessing/pool.py",第 572 行,在 get提高self._value
and it results in File "/usr/lib/python2.7/multiprocessing/pool.py", line 572, in get raise self._value
TypeError: 'module' 对象不可调用
TypeError: 'module' object is not callable
推荐答案
Python 2.7 缺少 Python 3.3+ 中的 starmap 池方法.您可以通过使用包装器来装饰您的目标函数来克服这个问题,该包装器将参数元组解包并调用目标函数:
Python 2.7 lacks the starmap pool-method from Python 3.3+ . You can overcome this by decorating your target function with a wrapper, which unpacks the argument-tuple and calls the target function:
import os from multiprocessing import Pool import random from functools import wraps def unpack(func): @wraps(func) def wrapper(arg_tuple): return func(*arg_tuple) return wrapper @unpack def write_on_files(a, seed): random.seed(seed) print("%d opening file %s" % (os.getpid(), a)) # simulate for _ in range(10): j = random.randint(0, 10) print("%d writing %d to file %s" % (os.getpid(), j, a)) # simulate if __name__ == '__main__': folder = ["Test/%d.csv" % i for i in range(0, 4)] seed = [234124, 663123, 12345, 123833] arguments = zip(folder, seed) pool = Pool(4) pool.map(write_on_files, iterable=arguments) pool.close() pool.join()