scrapy一些内容

姿态

• python 3.6
• scrapy 1.1.1
• twisted 17.1.0
  

运行scrapy shell ，得到TypeError：

1. (scrapy) D:\>scrapy shell "www.python.com"
   
2. 2017-03-08 12:12:05 [scrapy] INFO: Scrapy 1.1.1 started (bot: scrapybot)
   
3. 2017-03-08 12:12:05 [scrapy] INFO: Overridden settings: {'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'LOGSTATS_INTERVAL': 0}
   
4. 2017-03-08 12:12:05 [scrapy] INFO: Enabled extensions:
   
5. ['scrapy.extensions.corestats.CoreStats',
   
6. 'scrapy.extensions.telnet.TelnetConsole']
   
7. 2017-03-08 12:12:05 [scrapy] INFO: Enabled downloader middlewares:
   
8. ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
   
9. 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
   
10. 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
    
11. 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
    
12. 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
    
13. 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
    
14. 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
    
15. 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
    
16. 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
    
17. 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
    
18. 'scrapy.downloadermiddlewares.stats.DownloaderStats']
    
19. 2017-03-08 12:12:05 [scrapy] INFO: Enabled spider middlewares:
    
20. ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
    
21. 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
    
22. 'scrapy.spidermiddlewares.referer.RefererMiddleware',
    
23. 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
    
24. 'scrapy.spidermiddlewares.depth.DepthMiddleware']
    
25. 2017-03-08 12:12:05 [scrapy] INFO: Enabled item pipelines:
    
26. []
    
27. 2017-03-08 12:12:05 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
    
28. 2017-03-08 12:12:05 [scrapy] INFO: Spider opened
    
29. Traceback (most recent call last):
    
30.   File "d:\Anaconda3\envs\scrapy\Scripts\scrapy-script.py", line 5, in <module>
    
31.     sys.exit(scrapy.cmdline.execute())
    
32.   File "d:\Anaconda3\envs\scrapy\lib\site-packages\scrapy\cmdline.py", line 142, in execute
    
33.     _run_print_help(parser, _run_command, cmd, args, opts)
    
34.   File "d:\Anaconda3\envs\scrapy\lib\site-packages\scrapy\cmdline.py", line 88, in _run_print_help
    
35.     func(*a, **kw)
    
36.   File "d:\Anaconda3\envs\scrapy\lib\site-packages\scrapy\cmdline.py", line 149, in _run_command
    
37.     cmd.run(args, opts)
    
38.   File "d:\Anaconda3\envs\scrapy\lib\site-packages\scrapy\commands\shell.py", line 71, in run
    
39.     shell.start(url=url)
    
40.   File "d:\Anaconda3\envs\scrapy\lib\site-packages\scrapy\shell.py", line 47, in start
    
41.     self.fetch(url, spider)
    
42.   File "d:\Anaconda3\envs\scrapy\lib\site-packages\scrapy\shell.py", line 112, in fetch
    
43.     reactor, self._schedule, request, spider)
    
44.   File "d:\Anaconda3\envs\scrapy\lib\site-packages\twisted\internet\threads.py", line 122, in blockingCallFromThread
    
45.     result.raiseException()
    
46.   File "d:\Anaconda3\envs\scrapy\lib\site-packages\twisted\python\failure.py", line 372, in raiseException
    
47.     raise self.value.with_traceback(self.tb)
    
48. TypeError: 'float' object is not iterable
    

复制代码

解决： 我在台式机上的虚拟环境为python3.6，scrapy1.1.1，同样可以运行，于是看了下返回的错误发生在twisted，检查了一下，台式机上是twisted16.6.0，然后应该是最近升级了一次，笔记本自动安装的是17.1.0的版本，然后执行

1. conda install twisted==16.6.0

复制代码

然后DOWNGRADE，yes，再次运行scrapy shell “[url=http://www.python.org]www.python.org”，可以成功进入ipython。

Miss_love

感谢分享