问题描述
我正在尝试运行一个简单的命令,该命令使用多处理按名称猜测性别.这段代码在以前的机器上工作过,所以也许我的设置与它有关.
I am trying to run a simple command that guesses gender by name using multiprocessing. This code worked on a previous machine so perhaps my setup had something to do with it.
下面是我的多处理代码:
Below is my multiprocessing code:
import sys import gender_guesser.detector as gender import multiprocessing import time d = gender.Detector() def guess_gender (name): n = name.title() # make first letter upper case and the rest lower case g = d.get_gender(n) # guess gender return g ls = ['john','joe','amamda','derick','peter','ashley','john','joe','amamda','derick','peter','ashley'] t=time.time() results=[] def callBack(x): results.append(x) pool = multiprocessing.Pool(processes=multiprocessing.cpu_count()-1, maxtasksperchild=1) for n in ls: print (n) pool.apply_async(guess_gender,args=[n],callback=callBack) pool.close() pool.join() results = pd.concat(results) print(time.time()-t)
它只是运行,不做任何事情.在我的 cmd 窗口中,我在错误消息的末尾看到以下内容:
It simply runs and doesn't do anything. In my cmd window, I see the following at the end of an error message:
AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)>
我在 Anaconda 上运行 python 版本 3.6.1:
Am running python version 3.6.1 on Anaconda:
import sys print(sys.version) 3.6.1 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)]
更新:仍然无法让它工作.下面是我运行提供的代码时的整个 cmd 日志.感谢您的任何想法!
Update: Still cannot get it to work. Below is the entire cmd log when I ran the code provided. I appreciate any thoughts you may have!
C:UsersywuGoogle Drive>jupyter notebook [I 10:13:43.954 NotebookApp] Serving notebooks from local directory: C:UsersywuGoogle Drive [I 10:13:43.954 NotebookApp] 0 active kernels [I 10:13:43.955 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=255a5c0c9af337a1c2187feb63f1c426fb903e5929a0b2f0 [I 10:13:43.956 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 10:13:43.959 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://localhost:8888/?token=255a5c0c9af337a1c2187feb63f1c426fb903e5929a0b2f0 [I 10:13:44.264 NotebookApp] Accepting one-time-token-authenticated connection from ::1 [W 10:13:44.319 NotebookApp] 404 GET /api/kernels/aceb78ee-73e4-4481-9993-63e5ee8f72cb/channels?session_id=AEA3C6B2B0A440FC84FF3BAF5F5CB615 (127.0.0.1): Kernel does not exist: aceb78ee-73e4-4481-9993-63e5ee8f72cb [W 10:13:44.328 NotebookApp] 404 GET /api/kernels/aceb78ee-73e4-4481-9993-63e5ee8f72cb/channels?session_id=AEA3C6B2B0A440FC84FF3BAF5F5CB615 (127.0.0.1) 20.07ms referer=None [I 10:13:54.740 NotebookApp] Creating new notebook in /code/python [I 10:13:55.241 NotebookApp] Kernel started: 45ab2da6-7466-408c-aa5a-98f7db54e711 [W 10:14:00.341 NotebookApp] Replacing stale connection: aceb78ee-73e4-4481-9993-63e5ee8f72cb:AEA3C6B2B0A440FC84FF3BAF5F5CB615 Process SpawnPoolWorker-2: Traceback (most recent call last): File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) Process SpawnPoolWorker-1: File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) Process SpawnPoolWorker-4: AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> Traceback (most recent call last): File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() Traceback (most recent call last): File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() Process SpawnPoolWorker-3: AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) Process SpawnPoolWorker-5: File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() Process SpawnPoolWorker-6: File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) Traceback (most recent call last): AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() Traceback (most recent call last): File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) Traceback (most recent call last): File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> Process SpawnPoolWorker-7: AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> Traceback (most recent call last): File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> Process SpawnPoolWorker-8: Process SpawnPoolWorker-9: Process SpawnPoolWorker-10: Traceback (most recent call last): Traceback (most recent call last): File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() Traceback (most recent call last): File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() Process SpawnPoolWorker-11: File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> Process SpawnPoolWorker-14: File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) Traceback (most recent call last): AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) Traceback (most recent call last): AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 249, in _bootstrap self.run() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingprocess.py", line 93, in run self._target(*self._args, **self._kwargs) File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingpool.py", line 108, in worker task = get() File "C:UsersywuAppDataLocalContinuumAnaconda3libmultiprocessingqueues.py", line 345, in get return _ForkingPickler.loads(res) AttributeError: Can't get attribute 'guess_gender' on <module '__main__' (built-in)> [W 10:14:15.043 NotebookApp] 404 GET /api/kernels/c1224db6-69c6-470e-b74c-4c7b94fb48fe/channels?session_id=D8DC8A440B044EED8EBCA374EBEAF7C6 (127.0.0.1): Kernel does not exist: c1224db6-69c6-470e-b74c-4c7b94fb48fe [W 10:14:15.046 NotebookApp] 404 GET /api/kernels/c1224db6-69c6-470e-b74c-4c7b94fb48fe/channels?session_id=D8DC8A440B044EED8EBCA374EBEAF7C6 (127.0.0.1) 7.48ms referer=None
推荐答案
通过将我的函数保存在单独的 .py 文件中并将该文件包含在我的笔记本中,我可以在 Windows 上的 Jupyter 笔记本中进行多处理.
I got multiprocessing to work from within a Jupyter notebook on Windows by saving my function in a separate .py file and including that file in my notebook.
例子:
f.py:
def f(name, output): output.put('hello {0}'.format(name)) return
Jupyter 笔记本中的代码:
from multiprocessing import Process, Queue #Having the function definition here results in #AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)> #The solution seems to be importing the function from a separate file. import f #Also, the original version of f only had a print statement in it. #That doesn't work with Process - in the sense that it prints to the console #instead of the notebook. #The trick is to let f write the string to print into an output-queue. #When Process is done, the result is retrieved from the queue and printed. if __name__ == '__main__': # Define an output queue output=Queue() # Setup a list of processes that we want to run p = Process(target=f.f, args=('Bob',output)) # Run process p.start() # Exit the completed process p.join() # Get process results from the output queue result = output.get(p) print(result)
我是 Python 新手,我可能错过了各种细节,但这对我有用.
I'm a Python newby and I may have missed all sorts of details, but this works for me.