找到問題的那一刻,罵了本身N次的豬腦子!!!python
問題:爬蟲腳本原本一切正常的,臨時有其餘事情耽擱了,而後回頭正式運行的時候發現一隻報錯 [twisted] CRITICAL: Unhandled error in Deferred,錯誤就是下面這樣的(由於都同樣的錯誤,因此直接從網上貼過來的):python2.7
2016-03-13 08:50:50 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState Unhandled error in Deferred: 2016-03-13 08:50:50 [twisted] CRITICAL: Unhandled error in Deferred: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 150, in _run_command cmd.run(args, opts) File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/crawl.py", line 57, in run self.crawler_process.crawl(spname, **opts.spargs) File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 153, in crawl d = crawler.crawl(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1274, in unwindGenerator return _inlineCallbacks(None, gen, Deferred()) --- <exception caught here> --- File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 70, in crawl self.spider = self._create_spider(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 80, in _create_spider return self.spidercls.from_crawler(self, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/scrapy/spiders/crawl.py", line 91, in from_crawler spider = super(CrawlSpider, cls).from_crawler(crawler, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/scrapy/spiders/__init__.py", line 50, in from_crawler spider = cls(*args, **kwargs) exceptions.TypeError: __init__() takes at least 3 arguments (1 given) 2016-03-13 08:50:50 [twisted] CRITICAL:
問題解決思路:scrapy
一、網上有說多是twisted版本太高的,本人scrapy版本爲1.3.3,twisted版本爲13.1.0,已是1.3.3版本要求的最低的twisted版本,依然不行。ide
二、有考慮過pip源的問題,因此換了源,依然不行。(可是本人以前在安裝cryptography庫的時候確實是由於pypi源有問題,更換爲非官方的庫就行了。)post
官方庫下載地址:https://pypi.org/spa
非官方庫下載地址:https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpycode
三、在python安裝目錄script下運行安裝:ip
python C:\Python36\Scripts\pywin32_postinstall.py -installci
依然不行!!!get
問題解決:
要崩潰了,而後看到 Stack Overflow上一我的(https://stackoverflow.com/questions/35970518/scrapy-twisted-critical-unhandled-error-in-deferred)解決這個問題的思路,靈光一閃,對比一下本身代碼及配置文件,而後就發現上次清理文件時手殘的把scrapy.cfg配置文件刪除了!!!
因此出現詭異報錯時仍是要認真檢查配置項呀。