다중 처리 스트림 및 검색 API 결과 PickleError : NoneType

Question

트위터 검색 및 스트림 API를 다중 처리하려고하는데 프로세스를 시작할 수 없습니다. 나는 더 간단한 함수로 시도하고 start() 메소드가 작동한다.

이 문제를 해결할 수있는 방법을 알려 주시면 정말 고맙겠습니다.

감사합니다.

Traceback (most recent call last): 
    File "C:\Python27\Code\tweets\Stream.py", line 620, in <module> 
    tweetParser.start() 
    File "C:\Python27\lib\multiprocessing\process.py", line 130, in start 
    self._popen = Popen(self) 
    File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__ 
    dump(process_obj, to_child, HIGHEST_PROTOCOL) 
    File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump 
    ForkingPickler(file, protocol).dump(obj) 
    File "C:\Python27\lib\pickle.py", line 224, in dump 
    self.save(obj) 
    File "C:\Python27\lib\pickle.py", line 331, in save 
    self.save_reduce(obj=obj, *rv) 
    File "C:\Python27\lib\pickle.py", line 425, in save_reduce 
    save(state) 
    File "C:\Python27\lib\pickle.py", line 286, in save 
    f(self, obj) # Call unbound method with explicit self 
    File "C:\Python27\lib\pickle.py", line 655, in save_dict 
    self._batch_setitems(obj.iteritems()) 
    File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems 
    save(v) 
    File "C:\Python27\lib\pickle.py", line 286, in save 
    f(self, obj) # Call unbound method with explicit self 
    File "C:\Python27\lib\pickle.py", line 568, in save_tuple 
    save(element) 
    File "C:\Python27\lib\pickle.py", line 331, in save 
    self.save_reduce(obj=obj, *rv) 
    File "C:\Python27\lib\pickle.py", line 425, in save_reduce 
    save(state) 
    File "C:\Python27\lib\pickle.py", line 286, in save 
    f(self, obj) # Call unbound method with explicit self 
    File "C:\Python27\lib\pickle.py", line 655, in save_dict 
    self._batch_setitems(obj.iteritems()) 
    File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems 
    save(v) 
    File "C:\Python27\lib\pickle.py", line 286, in save 
    f(self, obj) # Call unbound method with explicit self 
    File "C:\Python27\lib\pickle.py", line 655, in save_dict 
    self._batch_setitems(obj.iteritems()) 
    File "C:\Python27\lib\pickle.py", line 686, in _batch_setitems 
    save(k) 
    File "C:\Python27\lib\pickle.py", line 286, in save 
    f(self, obj) # Call unbound method with explicit self 
    File "C:\Python27\lib\pickle.py", line 754, in save_global 
    (obj, module, name)) 
pickle.PicklingError: Can't pickle <type 'NoneType'>: it's not found as __builtin__.NoneType 
[INFO/MainProcess] process shutting down 
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0 
[DEBUG/MainProcess] running the remaining "atexit" finalizers 
Traceback (most recent call last): 
    File "<string>", line 1, in <module> 
    File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main 
    self = load(from_parent) 
    File "C:\Python27\lib\pickle.py", line 1384, in load 
    return Unpickler(file).load() 
    File "C:\Python27\lib\pickle.py", line 864, in load 
    dispatch[key](self) 
    File "C:\Python27\lib\pickle.py", line 886, in load_eof 
    raise EOFError 
EOFError 
[INFO/Process-1] process shutting down 
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0 
[DEBUG/Process-1] running the remaining "atexit" finalizers 

Answer 1

당신은 가지고있다 : 여기

queue = Queue() 
    exitEvent = Event() 
    multiprocessing.log_to_stderr(logging.DEBUG) 

    print "-- starting tweetParser! --" 
    tweetParser = Process(target = parseTweets, args = (conn, cursor, queue, exitEvent, config['debug'],)) 
    tweetParser.daemon = True 
    tweetParser.start() 

    print "-- starting twitterStream! --" 
    twitterStream = Process(target = StreamingTwitter, args = (streamAPI, queue, exitEvent, config['keywords'], config['time'], config['debug'],)) 
    twitterStream.daemon = True 
    twitterStream.start() 

    print "-- starting twitterSearch! --" 
    twitterSearch = Process(target = SearchingTwitter, args = (searchAPI, queue, exitEvent, config['keywords'], config['time'], config['debug'],)) 
    twitterSearch.start() 

    tweetParser.join() 

내 오류의 전체 역 추적입니다 : 감사합니다

, 테렌스 여기

프로세스를 시작하는 내 코드의 일부이다 직렬화 문제. multiprocessing으로 None을 피클링 할 수 없습니다. 그러나 multiprocess으로 할 수 있습니다. 더 나은 직렬화를 사용하는 포크입니다.

>>> import multiprocess 
>>> multiprocess.Pool().map(lambda x: x, [None, None, None]) 
[None, None, None] 
>>> 

그게 전부 일 수도 있습니다.