51Testing软件测试论坛

标题: python 将进程池放在装饰器里为什么不生效也没报错 [打印本页]

作者: 草帽路飞UU    时间: 2017-6-26 11:33
标题: python 将进程池放在装饰器里为什么不生效也没报错

我想把进程池封装在装饰器里,但是它既没生效也没报错

  1. def handle_request(response):
  2.     print str(response)

  3. def run_in_process(process_num):
  4.     def _run_in_process(f):
  5.         def __run_in_process(*args, **kwargs):
  6.             pool = multiprocessing.Pool(processes=process_num)
  7.             for i in range(process_num):
  8.                 pool.apply_async(f, args=args, kwds=kwargs, callback=kwargs.get("callback"))
  9.             pool.close()
  10.             pool.join()

  11.         return __run_in_process

  12.     return _run_in_process


  13. @run_in_process(process_num)
  14. def main():
  15.     http_client = AsyncHTTPClient()
  16.     http_client.fetch(url, callback=handle_request)
  17.     global loop
  18.     loop = tornado.ioloop.IOLoop.instance()
  19.     if loop._running is False:
  20.         loop.start()
复制代码

结果如下:
  1. /usr/bin/python2.7 /home/workspace/py_project/crawler/center/sample.py

  2. Process finished with exit code 0
复制代码

但是奇怪的是,我用多进程的方式重写一次,发现是可以生效的
  1. def handle_request(response):
  2.     print str(response)

  3. def run_in_process(process_num):
  4.     def _run_in_process(f):
  5.         def __run_in_process(*args, **kwargs):
  6.             _processes = []
  7.             for i in xrange(process_num):
  8.                 p = multiprocessing.Process(target=f, args=args, kwargs=kwargs)
  9.                 p.start()
  10.                 _processes.append(p)

  11.             for p in _processes:
  12.                 p.join()

  13.         return __run_in_process
  14.     return _run_in_process


  15. @run_in_process(process_num)
  16. def main():
  17.     http_client = AsyncHTTPClient()
  18.     http_client.fetch(url, callback=handle_request)
  19.     global loop
  20.     loop = tornado.ioloop.IOLoop.instance()
  21.     if loop._running is False:
  22.         loop.start()


  23. if __name__ == '__main__':
  24.     main()
复制代码


日志如下

  1. /usr/bin/python2.7 /home/workspace/py_project/crawler/center/sample.py
  2. HTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa21ef0>,code=200,effective_url='http://www.baidu.com',error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa425d0>,reason='OK',request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa42250>,request_time=0.014312028884887695,time_info={})
  3. HTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa21ef0>,code=200,effective_url='http://www.baidu.com',error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa43450>,reason='OK',request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa430d0>,request_time=0.02327895164489746,time_info={})
  4. HTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa21ef0>,code=200,effective_url='http://www.baidu.com',error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa43510>,reason='OK',request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa43190>,request_time=0.026951074600219727,time_info={})
  5. HTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa21ef0>,code=200,effective_url='http://www.baidu.com',error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa42690>,reason='OK',request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa42310>,request_time=0.0552978515625,time_info={})
  6. HTTPResponse(_body=None,buffer=<_io.BytesIO object at 0x7f2fdaa24ef0>,code=200,effective_url='http://www.baidu.com',error=None,headers=<tornado.httputil.HTTPHeaders object at 0x7f2fdaa39e10>,reason='OK',request=<tornado.httpclient.HTTPRequest object at 0x7f2fdaa39a90>,request_time=0.05612993240356445,time_info={})
复制代码
同样的情况也会出现在线程池跟协程的使用上,有谁知道这是怎么回事吗?


作者: 八戒你干嘛    时间: 2017-6-26 17:31
同问,等大神来解答




欢迎光临 51Testing软件测试论坛 (http://bbs.51testing.com/) Powered by Discuz! X3.2