问题描述
我想使用 Pool.map() 将关键字参数传递给我的工作函数.在搜索论坛时,我找不到明确的例子.
I would like to pass keyword arguments to my worker-function with Pool.map(). I can't find a clear example of this when searching forums.
示例代码:
import multiprocessing as mp def worker((x,y), **kwargs): kwarg_test = kwargs.get('kwarg_test', False) print("kwarg_test = {}".format(kwarg_test)) if kwarg_test: print("Success") return x*y def wrapper_process(**kwargs): jobs = [] pool=mp.Pool(4) for i, n in enumerate(range(4)): jobs.append((n,i)) pool.map(worker, jobs) #works pool.map(worker, jobs, kwargs) #how to do this? def main(**kwargs): worker((1,2),kwarg_test=True) #accepts kwargs wrapper_process(kwarg_test=True) if __name__ == "__main__": main()
输出:
kwarg_test = True Success kwarg_test = False kwarg_test = False kwarg_test = False kwarg_test = False TypeError: unsupported operand type(s) for //: 'int' and 'dict'
类型错误与解析 multiprocessing.Pool 或 Queue 中的参数有关,我尝试了其他几种语法,例如制作 kwargs 列表;[kwargs, kwargs, kwargs, kwargs],以及多次尝试将 kwarg 包含在工作列表中,但没有运气.我从 map 到 map_async 跟踪 multiprocessing.pool 中的代码,并达到了task_batches = Pool._get_tasks(func, iterable, chunksize)当我遇到生成器结构时,在 pool.py 中.我很高兴将来能进一步了解这方面的信息,但现在我只是想了解一下:
The type error has to do with parsing arguments inside of multiprocessing.Pool or Queue, and I have tried several other syntaxes, like making a list of the kwargs; [kwargs, kwargs, kwargs, kwargs], as well as several attempts to include the kwarg in the jobs list but no luck. I traced the code in multiprocessing.pool from map to map_async and got as far as task_batches = Pool._get_tasks(func, iterable, chunksize) in pool.py when I encountered the generator structure. I'm happy to learn more about this in future but for now I am just trying to find out:
是否有允许通过 pool.map 传递 kwargs 的简单语法?
Is there a simple syntax for allowing the passing of kwargs with pool.map?
推荐答案
如果您想遍历其他参数,请使用@ArcturusB 的答案.
If you want to iterate over the other arguments, use @ArcturusB's answer.
如果您只想传递它们,并且每次迭代都具有相同的值,那么您可以这样做:
If you just want to pass them, having the same value for each iteration, then you can do this:
from functools import partial pool.map(partial(worker, **kwargs), jobs)
Partial 将参数绑定"到函数.但是,旧版本的 Python 不能序列化部分对象.
Partial 'binds' arguments to a function. Old versions of Python cannot serialize partial objects though.