一 介紹
Scrapy一個開源和協做的框架，其最初是爲了頁面抓取 (更確切來講, 網絡抓取 )所設計的，使用它能夠以快速、簡單、可擴展的方式從網站中提取所需的數據。但目前Scrapy的用途十分普遍，可用於如數據挖掘、監測和自動化測試等領域，也能夠應用在獲取API所返回的數據(例如 Amazon Associates Web Services ) 或者通用的網絡爬蟲。css
Scrapy 是基於twisted框架開發而來，twisted是一個流行的事件驅動的python網絡框架。所以Scrapy使用了一種非阻塞（又名異步）的代碼來實現併發。總體架構大體以下
html
The data flow in Scrapy is controlled by the execution engine, and goes like this:python
- The Engine gets the initial Requests to crawl from the Spider.
- The Engine schedules the Requests in the Scheduler and asks for the next Requests to crawl.
- The Scheduler returns the next Requests to the Engine.
- The Engine sends the Requests to the Downloader, passing through the Downloader Middlewares (see
process_request()). - Once the page finishes downloading the Downloader generates a Response (with that page) and sends it to the Engine, passing through the Downloader Middlewares (see
process_response()). - The Engine receives the Response from the Downloader and sends it to the Spider for processing, passing through the Spider Middleware (see
process_spider_input()). - The Spider processes the Response and returns scraped items and new Requests (to follow) to the Engine, passing through the Spider Middleware (see
process_spider_output()). - The Engine sends processed items to Item Pipelines, then send processed Requests to the Scheduler and asks for possible next Requests to crawl.
- The process repeats (from step 1) until there are no more requests from the Scheduler.
Components：web
- 引擎(EGINE)
引擎負責控制系統全部組件之間的數據流，並在某些動做發生時觸發事件。有關詳細信息，請參見上面的數據流部分。ajax
- 調度器(SCHEDULER)
用來接受引擎發過來的請求, 壓入隊列中, 並在引擎再次請求的時候返回. 能夠想像成一個URL的優先級隊列, 由它來決定下一個要抓取的網址是什麼, 同時去除重複的網址 - 下載器(DOWLOADER)
用於下載網頁內容, 並將網頁內容返回給EGINE，下載器是創建在twisted這個高效的異步模型上的 - 爬蟲(SPIDERS)
SPIDERS是開發人員自定義的類，用來解析responses，而且提取items，或者發送新的請求 - 項目管道(ITEM PIPLINES)
在items被提取後負責處理它們，主要包括清理、驗證、持久化（好比存到數據庫）等操做 - 下載器中間件(Downloader Middlewares)
位於Scrapy引擎和下載器之間，主要用來處理從EGINE傳到DOWLOADER的請求request，已經從DOWNLOADER傳到EGINE的響應response，你可用該中間件作如下幾件事- process a request just before it is sent to the Downloader (i.e. right before Scrapy sends the request to the website);
- change received response before passing it to a spider;
- send a new Request instead of passing received response to a spider;
- pass response to a spider without fetching a web page;
- silently drop some requests.
- 爬蟲中間件(Spider Middlewares)
位於EGINE和SPIDERS之間，主要工做是處理SPIDERS的輸入（即responses）和輸出（即requests）
官網連接：https://docs.scrapy.org/en/latest/topics/architecture.html正則表達式
二 安裝
#Windows平臺 一、pip3 install wheel #安裝後，便支持經過wheel文件安裝軟件，wheel文件官網：https://www.lfd.uci.edu/~gohlke/pythonlibs 3、pip3 install lxml 4、pip3 install pyopenssl 五、下載並安裝pywin32：https://sourceforge.net/projects/pywin32/files/pywin32/ 六、下載twisted的wheel文件：http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted 七、執行pip3 install 下載目錄\Twisted-17.9.0-cp36-cp36m-win_amd64.whl 8、pip3 install scrapy #Linux平臺 一、pip3 install scrapy
三 命令行工具
#1 查看幫助 scrapy -h scrapy <command> -h #2 有兩種命令：其中Project-only必須切到項目文件夾下才能執行，而Global的命令則不須要 Global commands: startproject #建立項目 genspider #建立爬蟲程序 settings #若是是在項目目錄下，則獲得的是該項目的配置 runspider #運行一個獨立的python文件，沒必要建立項目 shell #scrapy shell url地址 在交互式調試，如選擇器規則正確與否 fetch #獨立於程單純地爬取一個頁面，能夠拿到請求頭 view #下載完畢後直接彈出瀏覽器，以此能夠分辨出哪些數據是ajax請求 version #scrapy version 查看scrapy的版本，scrapy version -v查看scrapy依賴庫的版本 Project-only commands: crawl #運行爬蟲，必須建立項目才行，確保配置文件中ROBOTSTXT_OBEY = False check #檢測項目中有無語法錯誤 list #列出項目中所包含的爬蟲名 edit #編輯器，通常不用 parse #scrapy parse url地址 --callback 回調函數 #以此能夠驗證咱們的回調函數是否正確 bench #scrapy bentch壓力測試 #3 官網連接 https://docs.scrapy.org/en/latest/topics/commands.html
#一、執行全局命令：請確保不在某個項目的目錄下，排除受該項目配置的影響 scrapy startproject MyProject cd MyProject scrapy genspider baidu www.baidu.com scrapy settings --get XXX #若是切換到項目目錄下，看到的則是該項目的配置 scrapy runspider baidu.py scrapy shell https://www.baidu.com response response.status response.body view(response) scrapy view https://www.taobao.com #若是頁面顯示內容不全，不全的內容則是ajax請求實現的，以此快速定位問題 scrapy fetch --nolog --headers https://www.taobao.com scrapy version #scrapy的版本 scrapy version -v #依賴庫的版本 #二、執行項目命令：切到項目目錄下 scrapy crawl baidu scrapy check scrapy list scrapy parse http://quotes.toscrape.com/ --callback parse scrapy bench
四 項目結構以及爬蟲應用簡介
project_name/ scrapy.cfg project_name/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py 爬蟲1.py 爬蟲2.py 爬蟲3.py
文件說明：算法
- scrapy.cfg 項目的主配置信息，用來部署scrapy時使用，爬蟲相關的配置信息在settings.py文件中。
- items.py 設置數據存儲模板，用於結構化數據，如：Django的Model
- pipelines 數據處理行爲，如：通常結構化的數據持久化
- settings.py 配置文件，如：遞歸的層數、併發數，延遲下載等。強調:配置文件的選項必須大寫不然視爲無效，正確寫法USER_AGENT='xxxx'
- spiders 爬蟲目錄，如：建立文件，編寫爬蟲規則
注意：通常建立爬蟲文件時，以網站域名命名mongodb
#在項目目錄下新建：entrypoint.py from scrapy.cmdline import execute execute(['scrapy', 'crawl', 'xiaohua'])
import sys,os sys.stdout=io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030')
五 Spiders
一、介紹shell
#一、Spiders是由一系列類（定義了一個網址或一組網址將被爬取）組成，具體包括如何執行爬取任務而且如何從頁面中提取結構化的數據。 #二、換句話說，Spiders是你爲了一個特定的網址或一組網址自定義爬取和解析頁面行爲的地方
二、Spiders會循環作以下事情數據庫
#一、生成初始的Requests來爬取第一個URLS，而且標識一個回調函數 第一個請求定義在start_requests()方法內默認從start_urls列表中得到url地址來生成Request請求，默認的回調函數是parse方法。回調函數在下載完成返回response時自動觸發 #二、在回調函數中，解析response而且返回值 返回值能夠4種： 包含解析數據的字典 Item對象 新的Request對象（新的Requests也須要指定一個回調函數） 或者是可迭代對象（包含Items或Request） #三、在回調函數中解析頁面內容 一般使用Scrapy自帶的Selectors，但很明顯你也可使用Beutifulsoup，lxml或其餘你愛用啥用啥。 #四、最後，針對返回的Items對象將會被持久化到數據庫 經過Item Pipeline組件存到數據庫：https://docs.scrapy.org/en/latest/topics/item-pipeline.html#topics-item-pipeline） 或者導出到不一樣的文件（經過Feed exports：https://docs.scrapy.org/en/latest/topics/feed-exports.html#topics-feed-exports）
三、Spiders總共提供了五種類：
#一、scrapy.spiders.Spider #scrapy.Spider等同於scrapy.spiders.Spider #二、scrapy.spiders.CrawlSpider #三、scrapy.spiders.XMLFeedSpider #四、scrapy.spiders.CSVFeedSpider #五、scrapy.spiders.SitemapSpider
四、導入使用
# -*- coding: utf-8 -*- import scrapy from scrapy.spiders import Spider,CrawlSpider,XMLFeedSpider,CSVFeedSpider,SitemapSpider class AmazonSpider(scrapy.Spider): #自定義類，繼承Spiders提供的基類 name = 'amazon' allowed_domains = ['www.amazon.cn'] start_urls = ['http://www.amazon.cn/'] def parse(self, response): pass
五、class scrapy.spiders.Spider
這是最簡單的spider類，任何其餘的spider類都須要繼承它（包含你本身定義的）。
該類不提供任何特殊的功能，它僅提供了一個默認的start_requests方法默認從start_urls中讀取url地址發送requests請求，而且默認parse做爲回調函數
class AmazonSpider(scrapy.Spider): name = 'amazon' allowed_domains = ['www.amazon.cn'] start_urls = ['http://www.amazon.cn/'] custom_settings = { 'BOT_NAME' : 'Egon_Spider_Amazon', 'REQUEST_HEADERS' : { 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en', } } def parse(self, response): pass
#一、name = 'amazon' 定義爬蟲名，scrapy會根據該值定位爬蟲程序 因此它必需要有且必須惟一（In Python 2 this must be ASCII only.） #二、allowed_domains = ['www.amazon.cn'] 定義容許爬取的域名，若是OffsiteMiddleware啓動（默認就啓動）， 那麼不屬於該列表的域名及其子域名都不容許爬取 若是爬取的網址爲：https://www.example.com/1.html，那就添加'example.com'到列表. #三、start_urls = ['http://www.amazon.cn/'] 若是沒有指定url，就從該列表中讀取url來生成第一個請求 #四、custom_settings 值爲一個字典，定義一些配置信息，在運行爬蟲程序時，這些配置會覆蓋項目級別的配置 因此custom_settings必須被定義成一個類屬性，因爲settings會在類實例化前被加載 #五、settings 經過self.settings['配置項的名字']能夠訪問settings.py中的配置，若是本身定義了custom_settings仍是以本身的爲準 #六、logger 日誌名默認爲spider的名字 self.logger.debug('=============>%s' %self.settings['BOT_NAME']) #五、crawler：瞭解 該屬性必須被定義到類方法from_crawler中 #六、from_crawler(crawler, *args, **kwargs)：瞭解 You probably won’t need to override this directly because the default implementation acts as a proxy to the __init__() method, calling it with the given arguments args and named arguments kwargs. #七、start_requests() 該方法用來發起第一個Requests請求，且必須返回一個可迭代的對象。它在爬蟲程序打開時就被Scrapy調用，Scrapy只調用它一次。 默認從start_urls裏取出每一個url來生成Request(url, dont_filter=True) #針對參數dont_filter,請看自定義去重規則 若是你想要改變起始爬取的Requests，你就須要覆蓋這個方法，例如你想要起始發送一個POST請求，以下 class MySpider(scrapy.Spider): name = 'myspider' def start_requests(self): return [scrapy.FormRequest("http://www.example.com/login", formdata={'user': 'john', 'pass': 'secret'}, callback=self.logged_in)] def logged_in(self, response): # here you would extract links to follow and return Requests for # each of them, with another callback pass #八、parse(response) 這是默認的回調函數，全部的回調函數必須返回an iterable of Request and/or dicts or Item objects. #九、log(message[, level, component])：瞭解 Wrapper that sends a log message through the Spider’s logger, kept for backwards compatibility. For more information see Logging from Spiders. #十、closed(reason) 爬蟲程序結束時自動觸發
去重規則應該多個爬蟲共享的，但凡一個爬蟲爬取了，其餘都不要爬了，實現方式以下 #方法一： 1、新增類屬性 visited=set() #類屬性 2、回調函數parse方法內： def parse(self, response): if response.url in self.visited: return None ....... self.visited.add(response.url) #方法一改進：針對url可能過長，因此咱們存放url的hash值 def parse(self, response): url=md5(response.request.url) if url in self.visited: return None ....... self.visited.add(url) #方法二：Scrapy自帶去重功能 配置文件： DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter' #默認的去重規則幫咱們去重，去重規則在內存中 DUPEFILTER_DEBUG = False JOBDIR = "保存範文記錄的日誌路徑，如：/root/" # 最終路徑爲 /root/requests.seen，去重規則放文件中 scrapy自帶去重規則默認爲RFPDupeFilter，只須要咱們指定 Request(...,dont_filter=False) ，若是dont_filter=True則告訴Scrapy這個URL不參與去重。 #方法三： 咱們也能夠仿照RFPDupeFilter自定義去重規則， from scrapy.dupefilter import RFPDupeFilter，看源碼，仿照BaseDupeFilter #步驟一：在項目目錄下自定義去重文件dup.py class UrlFilter(object): def __init__(self): self.visited = set() #或者放到數據庫 @classmethod def from_settings(cls, settings): return cls() def request_seen(self, request): if request.url in self.visited: return True self.visited.add(request.url) def open(self): # can return deferred pass def close(self, reason): # can return a deferred pass def log(self, request, spider): # log that a request has been filtered pass #步驟二：配置文件settings.py： DUPEFILTER_CLASS = '項目名.dup.UrlFilter' # 源碼分析： from scrapy.core.scheduler import Scheduler 見Scheduler下的enqueue_request方法：self.df.request_seen(request)
#例一： import scrapy class MySpider(scrapy.Spider): name = 'example.com' allowed_domains = ['example.com'] start_urls = [ 'http://www.example.com/1.html', 'http://www.example.com/2.html', 'http://www.example.com/3.html', ] def parse(self, response): self.logger.info('A response from %s just arrived!', response.url) #例二：一個回調函數返回多個Requests和Items import scrapy class MySpider(scrapy.Spider): name = 'example.com' allowed_domains = ['example.com'] start_urls = [ 'http://www.example.com/1.html', 'http://www.example.com/2.html', 'http://www.example.com/3.html', ] def parse(self, response): for h3 in response.xpath('//h3').extract(): yield {"title": h3} for url in response.xpath('//a/@href').extract(): yield scrapy.Request(url, callback=self.parse) #例三：在start_requests()內直接指定起始爬取的urls，start_urls就沒有用了， import scrapy from myproject.items import MyItem class MySpider(scrapy.Spider): name = 'example.com' allowed_domains = ['example.com'] def start_requests(self): yield scrapy.Request('http://www.example.com/1.html', self.parse) yield scrapy.Request('http://www.example.com/2.html', self.parse) yield scrapy.Request('http://www.example.com/3.html', self.parse) def parse(self, response): for h3 in response.xpath('//h3').extract(): yield MyItem(title=h3) for url in response.xpath('//a/@href').extract(): yield scrapy.Request(url, callback=self.parse)
咱們可能須要在命令行爲爬蟲程序傳遞參數，好比傳遞初始的url，像這樣 #命令行執行 scrapy crawl myspider -a category=electronics #在__init__方法中能夠接收外部傳進來的參數 import scrapy class MySpider(scrapy.Spider): name = 'myspider' def __init__(self, category=None, *args, **kwargs): super(MySpider, self).__init__(*args, **kwargs) self.start_urls = ['http://www.example.com/categories/%s' % category] #... #注意接收的參數全都是字符串，若是想要結構化的數據，你須要用相似json.loads的方法
六、其餘通用Spiders：https://docs.scrapy.org/en/latest/topics/spiders.html#generic-spiders
六 Selectors
#1 //與/ #2 text #三、extract與extract_first:從selector對象中解出內容 #四、屬性：xpath的屬性加前綴@ #四、嵌套查找 #五、設置默認值 #四、按照屬性查找 #五、按照屬性模糊查找 #六、正則表達式 #七、xpath相對路徑 #八、帶變量的xpath
response.selector.css() response.selector.xpath() 可簡寫爲 response.css() response.xpath() #1 //與/ response.xpath('//body/a/')# response.css('div a::text') >>> response.xpath('//body/a') #開頭的//表明從整篇文檔中尋找,body以後的/表明body的兒子 [] >>> response.xpath('//body//a') #開頭的//表明從整篇文檔中尋找,body以後的//表明body的子子孫孫 [<Selector xpath='//body//a' data='<a href="image1.html">Name: My image 1 <'>, <Selector xpath='//body//a' data='<a href="image2.html">Name: My image 2 <'>, <Selector xpath='//body//a' data='<a href=" image3.html">Name: My image 3 <'>, <Selector xpath='//body//a' data='<a href="image4.html">Name: My image 4 <'>, <Selector xpath='//body//a' data='<a href="image5.html">Name: My image 5 <'>] #2 text >>> response.xpath('//body//a/text()') >>> response.css('body a::text') #三、extract與extract_first:從selector對象中解出內容 >>> response.xpath('//div/a/text()').extract() ['Name: My image 1 ', 'Name: My image 2 ', 'Name: My image 3 ', 'Name: My image 4 ', 'Name: My image 5 '] >>> response.css('div a::text').extract() ['Name: My image 1 ', 'Name: My image 2 ', 'Name: My image 3 ', 'Name: My image 4 ', 'Name: My image 5 '] >>> response.xpath('//div/a/text()').extract_first() 'Name: My image 1 ' >>> response.css('div a::text').extract_first() 'Name: My image 1 ' #四、屬性：xpath的屬性加前綴@ >>> response.xpath('//div/a/@href').extract_first() 'image1.html' >>> response.css('div a::attr(href)').extract_first() 'image1.html' #四、嵌套查找 >>> response.xpath('//div').css('a').xpath('@href').extract_first() 'image1.html' #五、設置默認值 >>> response.xpath('//div[@id="xxx"]').extract_first(default="not found") 'not found' #四、按照屬性查找 response.xpath('//div[@id="images"]/a[@href="image3.html"]/text()').extract() response.css('#images a[@href="image3.html"]/text()').extract() #五、按照屬性模糊查找 response.xpath('//a[contains(@href,"image")]/@href').extract() response.css('a[href*="image"]::attr(href)').extract() response.xpath('//a[contains(@href,"image")]/img/@src').extract() response.css('a[href*="imag"] img::attr(src)').extract() response.xpath('//*[@href="image1.html"]') response.css('*[href="image1.html"]') #六、正則表達式 response.xpath('//a/text()').re(r'Name: (.*)') response.xpath('//a/text()').re_first(r'Name: (.*)') #七、xpath相對路徑 >>> res=response.xpath('//a[contains(@href,"3")]')[0] >>> res.xpath('img') [<Selector xpath='img' data='<img src="image3_thumb.jpg">'>] >>> res.xpath('./img') [<Selector xpath='./img' data='<img src="image3_thumb.jpg">'>] >>> res.xpath('.//img') [<Selector xpath='.//img' data='<img src="image3_thumb.jpg">'>] >>> res.xpath('//img') #這就是從頭開始掃描 [<Selector xpath='//img' data='<img src="image1_thumb.jpg">'>, <Selector xpath='//img' data='<img src="image2_thumb.jpg">'>, <Selector xpath='//img' data='<img src="image3_thumb.jpg">'>, <Selector xpa th='//img' data='<img src="image4_thumb.jpg">'>, <Selector xpath='//img' data='<img src="image5_thumb.jpg">'>] #八、帶變量的xpath >>> response.xpath('//div[@id=$xxx]/a/text()',xxx='images').extract_first() 'Name: My image 1 ' >>> response.xpath('//div[count(a)=$yyy]/@id',yyy=5).extract_first() #求有5個a標籤的div的id 'images'
https://docs.scrapy.org/en/latest/topics/selectors.html
七 Items
https://docs.scrapy.org/en/latest/topics/items.html
八 Item Pipeline
#一：能夠寫多個Pipeline類 #一、若是優先級高的Pipeline的process_item返回一個值或者None，會自動傳給下一個pipline的process_item, #二、若是隻想讓第一個Pipeline執行，那得讓第一個pipline的process_item拋出異常raise DropItem() #三、能夠用spider.name == '爬蟲名' 來控制哪些爬蟲用哪些pipeline 二：示範 from scrapy.exceptions import DropItem class CustomPipeline(object): def __init__(self,v): self.value = v @classmethod def from_crawler(cls, crawler): """ Scrapy會先經過getattr判斷咱們是否自定義了from_crawler,有則調它來完 成實例化 """ val = crawler.settings.getint('MMMM') return cls(val) def open_spider(self,spider): """ 爬蟲剛啓動時執行一次 """ print('000000') def close_spider(self,spider): """ 爬蟲關閉時執行一次 """ print('111111') def process_item(self, item, spider): # 操做並進行持久化 # return表示會被後續的pipeline繼續處理 return item # 表示將item丟棄，不會被後續pipeline處理 # raise DropItem()
#一、settings.py HOST="127.0.0.1" PORT=27017 USER="root" PWD="123" DB="amazon" TABLE="goods" ITEM_PIPELINES = { 'Amazon.pipelines.CustomPipeline': 200, } #二、pipelines.py class CustomPipeline(object): def __init__(self,host,port,user,pwd,db,table): self.host=host self.port=port self.user=user self.pwd=pwd self.db=db self.table=table @classmethod def from_crawler(cls, crawler): """ Scrapy會先經過getattr判斷咱們是否自定義了from_crawler,有則調它來完 成實例化 """ HOST = crawler.settings.get('HOST') PORT = crawler.settings.get('PORT') USER = crawler.settings.get('USER') PWD = crawler.settings.get('PWD') DB = crawler.settings.get('DB') TABLE = crawler.settings.get('TABLE') return cls(HOST,PORT,USER,PWD,DB,TABLE) def open_spider(self,spider): """ 爬蟲剛啓動時執行一次 """ self.client = MongoClient('mongodb://%s:%s@%s:%s' %(self.user,self.pwd,self.host,self.port)) def close_spider(self,spider): """ 爬蟲關閉時執行一次 """ self.client.close() def process_item(self, item, spider): # 操做並進行持久化 self.client[self.db][self.table].save(dict(item))
https://docs.scrapy.org/en/latest/topics/item-pipeline.html
九 Dowloader Middeware
下載中間件的用途 1、在process——request內，自定義下載，不用scrapy的下載 2、對請求進行二次加工，好比 設置請求頭 設置cookie 添加代理 scrapy自帶的代理組件： from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware from urllib.request import getproxies
class DownMiddleware1(object): def process_request(self, request, spider): """ 請求須要被下載時，通過全部下載器中間件的process_request調用 :param request: :param spider: :return: None,繼續後續中間件去下載； Response對象，中止process_request的執行，開始執行process_response Request對象，中止中間件的執行，將Request從新調度器 raise IgnoreRequest異常，中止process_request的執行，開始執行process_exception """ pass def process_response(self, request, response, spider): """ spider處理完成，返回時調用 :param response: :param result: :param spider: :return: Response 對象：轉交給其餘中間件process_response Request 對象：中止中間件，request會被從新調度下載 raise IgnoreRequest 異常：調用Request.errback """ print('response1') return response def process_exception(self, request, exception, spider): """ 當下載處理器(download handler)或 process_request() (下載中間件)拋出異常 :param response: :param exception: :param spider: :return: None：繼續交給後續中間件處理異常； Response對象：中止後續process_exception方法 Request對象：中止中間件，request將會被從新調用下載 """ return None
#一、與middlewares.py同級目錄下新建proxy_handle.py import requests def get_proxy(): return requests.get("http://127.0.0.1:5010/get/").text def delete_proxy(proxy): requests.get("http://127.0.0.1:5010/delete/?proxy={}".format(proxy)) #二、middlewares.py from Amazon.proxy_handle import get_proxy,delete_proxy class DownMiddleware1(object): def process_request(self, request, spider): """ 請求須要被下載時，通過全部下載器中間件的process_request調用 :param request: :param spider: :return: None,繼續後續中間件去下載； Response對象，中止process_request的執行，開始執行process_response Request對象，中止中間件的執行，將Request從新調度器 raise IgnoreRequest異常，中止process_request的執行，開始執行process_exception """ proxy="http://" + get_proxy() request.meta['download_timeout']=20 request.meta["proxy"] = proxy print('爲%s 添加代理%s ' % (request.url, proxy),end='') print('元數據爲',request.meta) def process_response(self, request, response, spider): """ spider處理完成，返回時調用 :param response: :param result: :param spider: :return: Response 對象：轉交給其餘中間件process_response Request 對象：中止中間件，request會被從新調度下載 raise IgnoreRequest 異常：調用Request.errback """ print('返回狀態嗎',response.status) return response def process_exception(self, request, exception, spider): """ 當下載處理器(download handler)或 process_request() (下載中間件)拋出異常 :param response: :param exception: :param spider: :return: None：繼續交給後續中間件處理異常； Response對象：中止後續process_exception方法 Request對象：中止中間件，request將會被從新調用下載 """ print('代理%s，訪問%s出現異常:%s' %(request.meta['proxy'],request.url,exception)) import time time.sleep(5) delete_proxy(request.meta['proxy'].split("//")[-1]) request.meta['proxy']='http://'+get_proxy() return request
十 Spider Middleware
class SpiderMiddleware(object): def process_spider_input(self,response, spider): """ 下載完成，執行，而後交給parse處理 :param response: :param spider: :return: """ pass def process_spider_output(self,response, result, spider): """ spider處理完成，返回時調用 :param response: :param result: :param spider: :return: 必須返回包含 Request 或 Item 對象的可迭代對象(iterable) """ return result def process_spider_exception(self,response, exception, spider): """ 異常調用 :param response: :param exception: :param spider: :return: None,繼續交給後續中間件處理異常；含 Response 或 Item 的可迭代對象(iterable)，交給調度器或pipeline """ return None def process_start_requests(self,start_requests, spider): """ 爬蟲啓動時調用 :param start_requests: :param spider: :return: 包含 Request 對象的可迭代對象 """ return start_requests
十一 自定義擴展
自定義擴展（與django的信號相似） 1、django的信號是django是預留的擴展，信號一旦被觸發，相應的功能就會執行 二、scrapy自定義擴展的好處是能夠在任意咱們想要的位置添加功能，而其餘組件中提供的功能只能在規定的位置執行
#一、在與settings同級目錄下新建一個文件，文件名能夠爲extentions.py,內容以下 from scrapy import signals class MyExtension(object): def __init__(self, value): self.value = value @classmethod def from_crawler(cls, crawler): val = crawler.settings.getint('MMMM') obj = cls(val) crawler.signals.connect(obj.spider_opened, signal=signals.spider_opened) crawler.signals.connect(obj.spider_closed, signal=signals.spider_closed) return obj def spider_opened(self, spider): print('=============>open') def spider_closed(self, spider): print('=============>close') #二、配置生效 EXTENSIONS = { "Amazon.extentions.MyExtension":200 }
十二 settings.py
# -*- coding: utf-8 -*- # Scrapy settings for step8_king project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # http://doc.scrapy.org/en/latest/topics/settings.html # http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html # http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html # 1. 爬蟲名稱 BOT_NAME = 'step8_king' # 2. 爬蟲應用路徑 SPIDER_MODULES = ['step8_king.spiders'] NEWSPIDER_MODULE = 'step8_king.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent # 3. 客戶端 user-agent請求頭 # USER_AGENT = 'step8_king (+http://www.yourdomain.com)' # Obey robots.txt rules # 4. 禁止爬蟲配置 # ROBOTSTXT_OBEY = False # Configure maximum concurrent requests performed by Scrapy (default: 16) # 5. 併發請求數 # CONCURRENT_REQUESTS = 4 # Configure a delay for requests for the same website (default: 0) # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs # 6. 延遲下載秒數 # DOWNLOAD_DELAY = 2 # The download delay setting will honor only one of: # 7. 單域名訪問併發數，而且延遲下次秒數也應用在每一個域名 # CONCURRENT_REQUESTS_PER_DOMAIN = 2 # 單IP訪問併發數，若是有值則忽略：CONCURRENT_REQUESTS_PER_DOMAIN，而且延遲下次秒數也應用在每一個IP # CONCURRENT_REQUESTS_PER_IP = 3 # Disable cookies (enabled by default) # 8. 是否支持cookie，cookiejar進行操做cookie # COOKIES_ENABLED = True # COOKIES_DEBUG = True # Disable Telnet Console (enabled by default) # 9. Telnet用於查看當前爬蟲的信息，操做爬蟲等... # 使用telnet ip port ，而後經過命令操做 # TELNETCONSOLE_ENABLED = True # TELNETCONSOLE_HOST = '127.0.0.1' # TELNETCONSOLE_PORT = [6023,] # 10. 默認請求頭 # Override the default request headers: # DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', # } # Configure item pipelines # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html # 11. 定義pipeline處理請求 # ITEM_PIPELINES = { # 'step8_king.pipelines.JsonPipeline': 700, # 'step8_king.pipelines.FilePipeline': 500, # } # 12. 自定義擴展，基於信號進行調用 # Enable or disable extensions # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html # EXTENSIONS = { # # 'step8_king.extensions.MyExtension': 500, # } # 13. 爬蟲容許的最大深度，能夠經過meta查看當前深度；0表示無深度 # DEPTH_LIMIT = 3 # 14. 爬取時，0表示深度優先Lifo(默認)；1表示廣度優先FiFo # 後進先出，深度優先 # DEPTH_PRIORITY = 0 # SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleLifoDiskQueue' # SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.LifoMemoryQueue' # 先進先出，廣度優先 # DEPTH_PRIORITY = 1 # SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue' # SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue' # 15. 調度器隊列 # SCHEDULER = 'scrapy.core.scheduler.Scheduler' # from scrapy.core.scheduler import Scheduler # 16. 訪問URL去重 # DUPEFILTER_CLASS = 'step8_king.duplication.RepeatUrl' # Enable and configure the AutoThrottle extension (disabled by default) # See http://doc.scrapy.org/en/latest/topics/autothrottle.html """ 17. 自動限速算法 from scrapy.contrib.throttle import AutoThrottle 自動限速設置 1. 獲取最小延遲 DOWNLOAD_DELAY 2. 獲取最大延遲 AUTOTHROTTLE_MAX_DELAY 3. 設置初始下載延遲 AUTOTHROTTLE_START_DELAY 4. 當請求下載完成後，獲取其"鏈接"時間 latency，即：請求鏈接到接受到響應頭之間的時間 5. 用於計算的... AUTOTHROTTLE_TARGET_CONCURRENCY target_delay = latency / self.target_concurrency new_delay = (slot.delay + target_delay) / 2.0 # 表示上一次的延遲時間 new_delay = max(target_delay, new_delay) new_delay = min(max(self.mindelay, new_delay), self.maxdelay) slot.delay = new_delay """ # 開始自動限速 # AUTOTHROTTLE_ENABLED = True # The initial download delay # 初始下載延遲 # AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies # 最大下載延遲 # AUTOTHROTTLE_MAX_DELAY = 10 # The average number of requests Scrapy should be sending in parallel to each remote server # 平均每秒併發數 # AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: # 是否顯示 # AUTOTHROTTLE_DEBUG = True # Enable and configure HTTP caching (disabled by default) # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings """ 18. 啓用緩存 目的用於將已經發送的請求或相應緩存下來，以便之後使用 from scrapy.downloadermiddlewares.httpcache import HttpCacheMiddleware from scrapy.extensions.httpcache import DummyPolicy from scrapy.extensions.httpcache import FilesystemCacheStorage """ # 是否啓用緩存策略 # HTTPCACHE_ENABLED = True # 緩存策略：全部請求均緩存，下次在請求直接訪問原來的緩存便可 # HTTPCACHE_POLICY = "scrapy.extensions.httpcache.DummyPolicy" # 緩存策略：根據Http響應頭：Cache-Control、Last-Modified 等進行緩存的策略 # HTTPCACHE_POLICY = "scrapy.extensions.httpcache.RFC2616Policy" # 緩存超時時間 # HTTPCACHE_EXPIRATION_SECS = 0 # 緩存保存路徑 # HTTPCACHE_DIR = 'httpcache' # 緩存忽略的Http狀態碼 # HTTPCACHE_IGNORE_HTTP_CODES = [] # 緩存存儲的插件 # HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' """ 19. 代理，須要在環境變量中設置 from scrapy.contrib.downloadermiddleware.httpproxy import HttpProxyMiddleware 方式一：使用默認 os.environ { http_proxy:http://root:woshiniba@192.168.11.11:9999/ https_proxy:http://192.168.11.11:9999/ } 方式二：使用自定義下載中間件 def to_bytes(text, encoding=None, errors='strict'): if isinstance(text, bytes): return text if not isinstance(text, six.string_types): raise TypeError('to_bytes must receive a unicode, str or bytes ' 'object, got %s' % type(text).__name__) if encoding is None: encoding = 'utf-8' return text.encode(encoding, errors) class ProxyMiddleware(object): def process_request(self, request, spider): PROXIES = [ {'ip_port': '111.11.228.75:80', 'user_pass': ''}, {'ip_port': '120.198.243.22:80', 'user_pass': ''}, {'ip_port': '111.8.60.9:8123', 'user_pass': ''}, {'ip_port': '101.71.27.120:80', 'user_pass': ''}, {'ip_port': '122.96.59.104:80', 'user_pass': ''}, {'ip_port': '122.224.249.122:8088', 'user_pass': ''}, ] proxy = random.choice(PROXIES) if proxy['user_pass'] is not None: request.meta['proxy'] = to_bytes（"http://%s" % proxy['ip_port']） encoded_user_pass = base64.encodestring(to_bytes(proxy['user_pass'])) request.headers['Proxy-Authorization'] = to_bytes('Basic ' + encoded_user_pass) print "**************ProxyMiddleware have pass************" + proxy['ip_port'] else: print "**************ProxyMiddleware no pass************" + proxy['ip_port'] request.meta['proxy'] = to_bytes("http://%s" % proxy['ip_port']) DOWNLOADER_MIDDLEWARES = { 'step8_king.middlewares.ProxyMiddleware': 500, } """ """ 20. Https訪問 Https訪問時有兩種狀況： 1. 要爬取網站使用的可信任證書(默認支持) DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory" DOWNLOADER_CLIENTCONTEXTFACTORY = "scrapy.core.downloader.contextfactory.ScrapyClientContextFactory" 2. 要爬取網站使用的自定義證書 DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory" DOWNLOADER_CLIENTCONTEXTFACTORY = "step8_king.https.MySSLFactory" # https.py from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory from twisted.internet.ssl import (optionsForClientTLS, CertificateOptions, PrivateCertificate) class MySSLFactory(ScrapyClientContextFactory): def getCertificateOptions(self): from OpenSSL import crypto v1 = crypto.load_privatekey(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.key.unsecure', mode='r').read()) v2 = crypto.load_certificate(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.pem', mode='r').read()) return CertificateOptions( privateKey=v1, # pKey對象 certificate=v2, # X509對象 verify=False, method=getattr(self, 'method', getattr(self, '_ssl_method', None)) ) 其餘： 相關類 scrapy.core.downloader.handlers.http.HttpDownloadHandler scrapy.core.downloader.webclient.ScrapyHTTPClientFactory scrapy.core.downloader.contextfactory.ScrapyClientContextFactory 相關配置 DOWNLOADER_HTTPCLIENTFACTORY DOWNLOADER_CLIENTCONTEXTFACTORY """ """ 21. 爬蟲中間件 class SpiderMiddleware(object): def process_spider_input(self,response, spider): ''' 下載完成，執行，而後交給parse處理 :param response: :param spider: :return: ''' pass def process_spider_output(self,response, result, spider): ''' spider處理完成，返回時調用 :param response: :param result: :param spider: :return: 必須返回包含 Request 或 Item 對象的可迭代對象(iterable) ''' return result def process_spider_exception(self,response, exception, spider): ''' 異常調用 :param response: :param exception: :param spider: :return: None,繼續交給後續中間件處理異常；含 Response 或 Item 的可迭代對象(iterable)，交給調度器或pipeline ''' return None def process_start_requests(self,start_requests, spider): ''' 爬蟲啓動時調用 :param start_requests: :param spider: :return: 包含 Request 對象的可迭代對象 ''' return start_requests 內置爬蟲中間件： 'scrapy.contrib.spidermiddleware.httperror.HttpErrorMiddleware': 50, 'scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware': 500, 'scrapy.contrib.spidermiddleware.referer.RefererMiddleware': 700, 'scrapy.contrib.spidermiddleware.urllength.UrlLengthMiddleware': 800, 'scrapy.contrib.spidermiddleware.depth.DepthMiddleware': 900, """ # from scrapy.contrib.spidermiddleware.referer import RefererMiddleware # Enable or disable spider middlewares # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html SPIDER_MIDDLEWARES = { # 'step8_king.middlewares.SpiderMiddleware': 543, } """ 22. 下載中間件 class DownMiddleware1(object): def process_request(self, request, spider): ''' 請求須要被下載時，通過全部下載器中間件的process_request調用 :param request: :param spider: :return: None,繼續後續中間件去下載； Response對象，中止process_request的執行，開始執行process_response Request對象，中止中間件的執行，將Request從新調度器 raise IgnoreRequest異常，中止process_request的執行，開始執行process_exception ''' pass def process_response(self, request, response, spider): ''' spider處理完成，返回時調用 :param response: :param result: :param spider: :return: Response 對象：轉交給其餘中間件process_response Request 對象：中止中間件，request會被從新調度下載 raise IgnoreRequest 異常：調用Request.errback ''' print('response1') return response def process_exception(self, request, exception, spider): ''' 當下載處理器(download handler)或 process_request() (下載中間件)拋出異常 :param response: :param exception: :param spider: :return: None：繼續交給後續中間件處理異常； Response對象：中止後續process_exception方法 Request對象：中止中間件，request將會被從新調用下載 ''' return None 默認下載中間件 { 'scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware': 100, 'scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware': 300, 'scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware': 350, 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400, 'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 500, 'scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware': 550, 'scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware': 580, 'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware': 590, 'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600, 'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware': 700, 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 750, 'scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware': 830, 'scrapy.contrib.downloadermiddleware.stats.DownloaderStats': 850, 'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900, } """ # from scrapy.contrib.downloadermiddleware.httpauth import HttpAuthMiddleware # Enable or disable downloader middlewares # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html # DOWNLOADER_MIDDLEWARES = { # 'step8_king.middlewares.DownMiddleware1': 100, # 'step8_king.middlewares.DownMiddleware2': 500, # }