Scrapy - [twisted] CRITICAL: Unhandled error in Deferred-你的配置文件還在嗎?

找到問題的那一刻,罵了本身N次的豬腦子!!!python

問題:爬蟲腳本原本一切正常的,臨時有其餘事情耽擱了,而後回頭正式運行的時候發現一隻報錯 [twisted] CRITICAL: Unhandled error in Deferred,錯誤就是下面這樣的(由於都同樣的錯誤,因此直接從網上貼過來的):python2.7

2016-03-13 08:50:50 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
Unhandled error in Deferred:
2016-03-13 08:50:50 [twisted] CRITICAL: Unhandled error in Deferred:


Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 150, in _run_command
    cmd.run(args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/crawl.py", line 57, in run
    self.crawler_process.crawl(spname, **opts.spargs)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 153, in crawl
    d = crawler.crawl(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 70, in crawl
    self.spider = self._create_spider(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 80, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spiders/crawl.py", line 91, in from_crawler
    spider = super(CrawlSpider, cls).from_crawler(crawler, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spiders/__init__.py", line 50, in from_crawler
    spider = cls(*args, **kwargs)
exceptions.TypeError: __init__() takes at least 3 arguments (1 given)
2016-03-13 08:50:50 [twisted] CRITICAL:

問題解決思路:scrapy

一、網上有說多是twisted版本太高的,本人scrapy版本爲1.3.3,twisted版本爲13.1.0,已是1.3.3版本要求的最低的twisted版本,依然不行。ide

二、有考慮過pip源的問題,因此換了源,依然不行。(可是本人以前在安裝cryptography庫的時候確實是由於pypi源有問題,更換爲非官方的庫就行了。)post

官方庫下載地址:https://pypi.org/spa

非官方庫下載地址:https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpycode

三、在python安裝目錄script下運行安裝:ip

python C:\Python36\Scripts\pywin32_postinstall.py -installci

依然不行!!!get

問題解決:

要崩潰了,而後看到 Stack Overflow上一我的(https://stackoverflow.com/questions/35970518/scrapy-twisted-critical-unhandled-error-in-deferred)解決這個問題的思路,靈光一閃,對比一下本身代碼及配置文件,而後就發現上次清理文件時手殘的把scrapy.cfg配置文件刪除了!!!

因此出現詭異報錯時仍是要認真檢查配置項呀。

相關文章
相關標籤/搜索