爬蟲必備—Scrapy

時間 2019-11-12

原文原文鏈接

1、Scrapy簡介

Scrapy是一個爲了爬取網站數據，提取結構性數據而編寫的應用框架。其能夠應用在數據挖掘，信息處理或存儲歷史數據等一系列的程序中。
其最初是爲了頁面抓取 (更確切來講, 網絡抓取 )所設計的，也能夠應用在獲取API所返回的數據(例如 Amazon Associates Web Services ) 或者通用的網絡爬蟲。Scrapy用途普遍，能夠用於數據挖掘、監測和自動化測試。html

Scrapy 使用了 Twisted異步網絡庫來處理網絡通信。總體架構大體以下：python

Scrapy主要包括瞭如下組件：react

引擎(Scrapy)

　　用來處理整個系統的數據流處理, 觸發事務(框架核心)web

　　調度器(Scheduler)
　　用來接受引擎發過來的請求, 壓入隊列中, 並在引擎再次請求的時候返回. 能夠想像成一個URL（抓取網頁的網址或者說是連接）的優先隊列, 由它來決定下一個要抓取的網址是什麼, 同時去除重複的網址算法

下載器(Downloader)

　　用於下載網頁內容, 並將網頁內容返回給蜘蛛(Scrapy下載器是創建在twisted這個高效的異步模型上的)json

爬蟲(Spiders)

　　爬蟲是主要幹活的, 用於從特定的網頁中提取本身須要的信息, 即所謂的實體(Item)。用戶也能夠從中提取出連接,讓Scrapy繼續抓取下一個頁面緩存

項目管道(Pipeline)

　　負責處理爬蟲從網頁中抽取的實體，主要的功能是持久化實體、驗證明體的有效性、清除不須要的信息。當頁面被爬蟲解析後，將被髮送到項目管道，並通過幾個特定的次序處理數據。cookie

下載器中間件(Downloader Middlewares)

　　位於Scrapy引擎和下載器之間的框架，主要是處理Scrapy引擎與下載器之間的請求及響應。網絡

爬蟲中間件(Spider Middlewares)

　　介於Scrapy引擎和爬蟲之間的框架，主要工做是處理蜘蛛的響應輸入和請求輸出。架構

調度中間件(Scheduler Middewares)

　　介於Scrapy引擎和調度之間的中間件，從Scrapy引擎發送到調度的請求和響應。

Scrapy運行流程大概以下：

引擎從調度器中取出一個連接(URL)用於接下來的抓取
引擎把URL封裝成一個請求(Request)傳給下載器
下載器把資源下載下來，並封裝成應答包(Response)
爬蟲解析Response
解析出實體（Item）,則交給實體管道進行進一步的處理
解析出的是連接（URL）,則把URL交給調度器等待抓取

2、安裝

 1 Linux
 2       pip3 install scrapy
 3  
 4  
 5 Windows
 6       a. pip3 install wheel
 7       b. 下載twisted http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
 8       c. 進入下載目錄，執行 pip3 install Twisted‑17.1.0‑cp35‑cp35m‑win_amd64.whl
 9       d. pip3 install scrapy
10       e. 下載並安裝pywin32：https://sourceforge.net/projects/pywin32/files/

3、基本使用

1. 基本命令

1. scrapy startproject 項目名稱
   - 在當前目錄中建立中建立一個項目文件（相似於Django）
 
2. scrapy genspider [-t template] <name> <domain>
   - 建立爬蟲應用
   如：
      scrapy gensipider -t basic oldboy oldboy.com
      scrapy gensipider -t xmlfeed autohome autohome.com.cn
   PS:
      查看全部命令：scrapy gensipider -l
      查看模板命令：scrapy gensipider -d 模板名稱
 
3. scrapy list
   - 展現爬蟲應用列表
 
4. scrapy crawl 爬蟲應用名稱
   - 運行單獨爬蟲應用

2. 項目結構及爬蟲應用簡介

2.1. 目錄結構

project_name/
   scrapy.cfg
   project_name/
       __init__.py
       items.py
       pipelines.py
       settings.py
       spiders/
           __init__.py
           爬蟲1.py
           爬蟲2.py
           爬蟲3.py

2.2. 文件說明

scrapy.cfg 項目的主配置信息。（真正爬蟲相關的配置信息在settings.py文件中）
items.py 設置數據存儲模板，用於結構化數據，如：Django的Model
pipelines 數據處理行爲，如：通常結構化的數據持久化
settings.py 配置文件，如：遞歸的層數、併發數，延遲下載等
spiders 爬蟲目錄，如：建立文件，編寫爬蟲規則

注意：通常建立爬蟲文件時，以網站域名命名

2.3. 示例

 1 import scrapy
 2  
 3 class XiaoHuarSpider(scrapy.spiders.Spider):
 4     name = "xiaohuar"                            # 爬蟲名稱 *****
 5     allowed_domains = ["xiaohuar.com"]  # 容許的域名
 6     start_urls = [
 7         "http://www.xiaohuar.com/hua/",   # URL
 8     ]
 9  
10     def parse(self, response):
11         # 訪問起始URL並獲取結果後的回調函數

爬蟲1.py

若在Windows系統上若出現編碼錯誤，解決方法以下：

1 import sys,os
2 
3 sys.stdout=io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030')

3. 小試牛刀

 1 import scrapy
 2 from scrapy.selector import HtmlXPathSelector
 3 from scrapy.http.request import Request
 4  
 5  
 6 class DigSpider(scrapy.Spider):
 7     # 爬蟲應用的名稱，經過此名稱啓動爬蟲命令
 8     name = "dig"
 9  
10     # 容許的域名
11     allowed_domains = ["chouti.com"]
12  
13     # 起始URL
14     start_urls = [
15         'http://dig.chouti.com/',
16     ]
17  
18     has_request_set = {}
19  
20     def parse(self, response):
21         print(response.url)
22  
23         hxs = HtmlXPathSelector(response)
24         page_list = hxs.select('//div[@id="dig_lcpage"]//a[re:test(@href, "/all/hot/recent/\d+")]/@href').extract()
25         for page in page_list:
26             page_url = 'http://dig.chouti.com%s' % page
27             key = self.md5(page_url)
28             if key in self.has_request_set:
29                 pass
30             else:
31                 self.has_request_set[key] = page_url
32                 obj = Request(url=page_url, method='GET', callback=self.parse)
33                 yield obj
34  
35     @staticmethod
36     def md5(val):
37         import hashlib
38         ha = hashlib.md5()
39         ha.update(bytes(val, encoding='utf-8'))
40         key = ha.hexdigest()
41         return key

爬取抽屜網頁

執行此爬蟲文件，則在終端進入項目目錄執行以下命令：

1 scrapy crawl dig --nolog

對於上述代碼重要之處在於：

Request是一個封裝用戶請求的類，在回調函數中yield該對象表示繼續訪問
HtmlXpathSelector用於結構化HTML代碼並提供選擇器功能

4. 選擇器

 1 #!/usr/bin/env python
 2 # -*- coding:utf-8 -*-
 3 from scrapy.selector import Selector, HtmlXPathSelector
 4 from scrapy.http import HtmlResponse
 5 html = """<!DOCTYPE html>
 6 <html>
 7     <head lang="en">
 8         <meta charset="UTF-8">
 9         <title></title>
10     </head>
11     <body>
12         <ul>
13             <li class="item-"><a id='i1' href="link.html">first item</a></li>
14             <li class="item-0"><a id='i2' href="llink.html">first item</a></li>
15             <li class="item-1"><a href="llink2.html">second item<span>vv</span></a></li>
16         </ul>
17         <div><a href="llink2.html">second item</a></div>
18     </body>
19 </html>
20 """
21 response = HtmlResponse(url='http://example.com', body=html,encoding='utf-8')
22 # hxs = HtmlXPathSelector(response)
23 # print(hxs)
24 # hxs = Selector(response=response).xpath('//a')
25 # print(hxs)
26 # hxs = Selector(response=response).xpath('//a[2]')
27 # print(hxs)
28 # hxs = Selector(response=response).xpath('//a[@id]')
29 # print(hxs)
30 # hxs = Selector(response=response).xpath('//a[@id="i1"]')
31 # print(hxs)
32 # hxs = Selector(response=response).xpath('//a[@href="link.html"][@id="i1"]')
33 # print(hxs)
34 # hxs = Selector(response=response).xpath('//a[contains(@href, "link")]')
35 # print(hxs)
36 # hxs = Selector(response=response).xpath('//a[starts-with(@href, "link")]')
37 # print(hxs)
38 # hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]')
39 # print(hxs)
40 # hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]/text()').extract()
41 # print(hxs)
42 # hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]/@href').extract()
43 # print(hxs)
44 # hxs = Selector(response=response).xpath('/html/body/ul/li/a/@href').extract()
45 # print(hxs)
46 # hxs = Selector(response=response).xpath('//body/ul/li/a/@href').extract_first()
47 # print(hxs)
48  
49 # ul_list = Selector(response=response).xpath('//body/ul/li')
50 # for item in ul_list:
51 #     v = item.xpath('./a/span')
52 #     # 或
53 #     # v = item.xpath('a/span')
54 #     # 或
55 #     # v = item.xpath('*/a/span')
56 #     print(v)

View Code

5. 登錄抽屜實現自動點贊或取消贊

 1 # -*- coding: utf-8 -*-
 2 import scrapy
 3 import json
 4 import urllib.parse
 5 from scrapy.selector import Selector, HtmlXPathSelector
 6 from scrapy.http import Request
 7 
 8 
 9 class ChoutiSpider(scrapy.Spider):
10     name = 'chouti'
11     allowed_domains = ['dig.chouti.com']
12     start_urls = ['http://dig.chouti.com/']
13     cookie_dict = {}
14     news_dict = {}
15     type = 'undo'       # 取消點贊undo，點贊do
16 
17     def start_requests(self):
18         for url in self.start_urls:
19             yield Request(url, dont_filter=True, callback=self.login)
20 
21     def login(self, response):
22         print('login...\n')
23         from scrapy.http.cookies import CookieJar
24         ck_jar = CookieJar()
25         ck_jar.extract_cookies(response, response.request)  # 去響應中獲取cookies
26         for k, v in ck_jar._cookies.items():
27             for i, j in v.items():
28                 for m, n in j.items():
29                     self.cookie_dict[m] = n.value
30         print(self.cookie_dict,'\n')
31         post_dict = {
32             'phone': 123456789,
33             'password': '**********',
34             'oneMonth': 1,
35         }
36 
37         yield Request(
38             url='http://dig.chouti.com/login',
39             method='POST',
40             cookies=self.cookie_dict,
41             body=urllib.parse.urlencode(post_dict),
42             headers={'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'},
43             callback=self.get_news
44         )
45 
46     def get_news(self, response):
47         yield Request(url='http://dig.chouti.com/', cookies=self.cookie_dict, callback=self.do)
48 
49     def do(self, response):
50         hxs = Selector(response)
51         link_id_list = hxs.xpath('//div[@class="part2"]/@share-linkid').extract()
52         print(link_id_list)
53         undo_url = 'http://dig.chouti.com/vote/cancel/vote.do'
54         for link_id in link_id_list:
55             if self.type == 'undo':
56                 print('undo.........\n')
57                 post_dict = {'linksId': link_id}
58                 yield Request(
59                     undo_url, method='POST',
60                     cookies=self.cookie_dict,
61                     body=urllib.parse.urlencode(post_dict),
62                     headers={'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'},
63                     callback=self.showret)
64             else:
65                 print('dododod.......\n')
66                 thumb_url = "http://dig.chouti.com/link/vote?linksId=%s" % (link_id,)
67                 yield Request(thumb_url, method='POST', cookies=self.cookie_dict, callback=self.showret)
68         page_list = hxs.xpath('//a[@class="ct_pagepa"]/@href').extract()
69         for page in page_list:
70             # http://dig.chouti.com/all/hot/recent/2
71             page_url = 'http://dig.chouti.com%s' %(page,)
72             yield Request(page_url, method='GET', callback=self.do)
73 
74 
75     def showret(self, response):
76         print('show...\n', response.text)

抽屜網站自動點贊或取消贊

注意：settings.py中設置DEPTH_LIMIT = 1來指定「遞歸」的層數，即只操做當前頁。

6. 格式處理

上述實例只是簡單的處理，因此在parse方法中直接處理。若是對於想要獲取更多的數據處理，則能夠利用Scrapy的items將數據格式化，而後統一交由pipelines來處理。

 1 import scrapy
 2 from scrapy.selector import HtmlXPathSelector
 3 from scrapy.http.request import Request
 4 from scrapy.http.cookies import CookieJar
 5 from scrapy import FormRequest
 6 
 7 
 8 class XiaoHuarSpider(scrapy.Spider):
 9     # 爬蟲應用的名稱，經過此名稱啓動爬蟲命令
10     name = "xiaohuar"
11     # 容許的域名
12     allowed_domains = ["xiaohuar.com"]
13 
14     start_urls = [
15         "http://www.xiaohuar.com/list-1-1.html",
16     ]
17     # custom_settings = {
18     #     'ITEM_PIPELINES':{
19     #         'spider1.pipelines.JsonPipeline': 100
20     #     }
21     # }
22     has_request_set = {}
23 
24     def parse(self, response):
25         # 分析頁面
26         # 找到頁面中符合規則的內容（校花圖片），保存
27         # 找到全部的a標籤，再訪問其餘a標籤，一層一層的搞下去
28 
29         hxs = HtmlXPathSelector(response)
30 
31         items = hxs.select('//div[@class="item_list infinite_scroll"]/div')
32         for item in items:
33             src = item.select('.//div[@class="img"]/a/img/@src').extract_first()
34             name = item.select('.//div[@class="img"]/span/text()').extract_first()
35             school = item.select('.//div[@class="img"]/div[@class="btns"]/a/text()').extract_first()
36             url = "http://www.xiaohuar.com%s" % src
37             from ..items import XiaoHuarItem
38             obj = XiaoHuarItem(name=name, school=school, url=url)
39             yield obj
40 
41         urls = hxs.select('//a[re:test(@href, "http://www.xiaohuar.com/list-1-\d+.html")]/@href')
42         for url in urls:
43             key = self.md5(url)
44             if key in self.has_request_set:
45                 pass
46             else:
47                 self.has_request_set[key] = url
48                 req = Request(url=url,method='GET',callback=self.parse)
49                 yield req
50 
51     @staticmethod
52     def md5(val):
53         import hashlib
54         ha = hashlib.md5()
55         ha.update(bytes(val, encoding='utf-8'))
56         key = ha.hexdigest()
57         return key

spiders/xiahuar.py

1 import scrapy
2 
3 
4 class XiaoHuarItem(scrapy.Item):
5     name = scrapy.Field()
6     school = scrapy.Field()
7     url = scrapy.Field()

items

 1 import json
 2 import os
 3 import requests
 4 
 5 
 6 class JsonPipeline(object):
 7     def __init__(self):
 8         self.file = open('xiaohua.txt', 'w')
 9 
10     def process_item(self, item, spider):
11         v = json.dumps(dict(item), ensure_ascii=False)
12         self.file.write(v)
13         self.file.write('\n')
14         self.file.flush()
15         return item
16 
17 
18 class FilePipeline(object):
19     def __init__(self):
20         if not os.path.exists('imgs'):
21             os.makedirs('imgs')
22 
23     def process_item(self, item, spider):
24         response = requests.get(item['url'], stream=True)
25         file_name = '%s_%s.jpg' % (item['name'], item['school'])
26         with open(os.path.join('imgs', file_name), mode='wb') as f:
27             f.write(response.content)
28         return item

pipelines

1 ITEM_PIPELINES = {
2    'spider1.pipelines.JsonPipeline': 100,
3    'spider1.pipelines.FilePipeline': 300,
4 }
5 # 每行後面的整型值，肯定了他們運行的順序，item按數字從低到高的順序，經過pipeline，一般將這些數字定義在0-1000範圍內。

settings

對於pipeline能夠作更多，以下：

 1 from scrapy.exceptions import DropItem
 2 
 3 class CustomPipeline(object):
 4     def __init__(self,v):
 5         self.value = v
 6 
 7     def process_item(self, item, spider):
 8         # 操做並進行持久化
 9 
10         # return表示會被後續的pipeline繼續處理
11         return item
12 
13         # 表示將item丟棄，不會被後續pipeline處理
14         # raise DropItem()
15 
16 
17     @classmethod
18     def from_crawler(cls, crawler):
19         """
20         初始化時候，用於建立pipeline對象
21         :param crawler: 
22         :return: 
23         """
24         val = crawler.settings.getint('MMMM')
25         return cls(val)
26 
27     def open_spider(self,spider):
28         """
29         爬蟲開始執行時，調用
30         :param spider: 
31         :return: 
32         """
33         print('000000')
34 
35     def close_spider(self,spider):
36         """
37         爬蟲關閉時，被調用
38         :param spider: 
39         :return: 
40         """
41         print('111111')

自定義pipeline

7. 中間件

 1 class SpiderMiddleware(object):
 2 
 3     def process_spider_input(self,response, spider):
 4         """
 5         下載完成，執行，而後交給parse處理
 6         :param response: 
 7         :param spider: 
 8         :return: 
 9         """
10         pass
11 
12     def process_spider_output(self,response, result, spider):
13         """
14         spider處理完成，返回時調用
15         :param response:
16         :param result:
17         :param spider:
18         :return: 必須返回包含 Request 或 Item 對象的可迭代對象(iterable)
19         """
20         return result
21 
22     def process_spider_exception(self,response, exception, spider):
23         """
24         異常調用
25         :param response:
26         :param exception:
27         :param spider:
28         :return: None,繼續交給後續中間件處理異常；含 Response 或 Item 的可迭代對象(iterable)，交給調度器或pipeline
29         """
30         return None
31 
32 
33     def process_start_requests(self,start_requests, spider):
34         """
35         爬蟲啓動時調用
36         :param start_requests:
37         :param spider:
38         :return: 包含 Request 對象的可迭代對象
39         """
40         return start_requests

爬蟲中間件

 1 class DownMiddleware1(object):
 2     def process_request(self, request, spider):
 3         """
 4         請求須要被下載時，通過全部下載器中間件的process_request調用
 5         :param request: 
 6         :param spider: 
 7         :return:  
 8             None,繼續後續中間件去下載；
 9             Response對象，中止process_request的執行，開始執行process_response
10             Request對象，中止中間件的執行，將Request從新調度器
11             raise IgnoreRequest異常，中止process_request的執行，開始執行process_exception
12         """
13         pass
14 
15 
16 
17     def process_response(self, request, response, spider):
18         """
19         spider處理完成，返回時調用
20         :param response:
21         :param result:
22         :param spider:
23         :return: 
24             Response 對象：轉交給其餘中間件process_response
25             Request 對象：中止中間件，request會被從新調度下載
26             raise IgnoreRequest 異常：調用Request.errback
27         """
28         print('response1')
29         return response
30 
31     def process_exception(self, request, exception, spider):
32         """
33         當下載處理器(download handler)或 process_request() (下載中間件)拋出異常
34         :param response:
35         :param exception:
36         :param spider:
37         :return: 
38             None：繼續交給後續中間件處理異常；
39             Response對象：中止後續process_exception方法
40             Request對象：中止中間件，request將會被從新調用下載
41         """
42         return None

下載器中間件

8. 自定製命令

在spiders同級建立任意目錄，如：commands
在其中建立 crawlall.py 文件（此處文件名就是自定義的命令）

 1 from scrapy.commands import ScrapyCommand
 2     from scrapy.utils.project import get_project_settings
 3 
 4 
 5     class Command(ScrapyCommand):
 6 
 7         requires_project = True
 8 
 9         def syntax(self):
10             return '[options]'
11 
12         def short_desc(self):
13             return 'Runs all of the spiders'
14 
15         def run(self, args, opts):
16             spider_list = self.crawler_process.spiders.list()
17             for name in spider_list:
18                 self.crawler_process.crawl(name, **opts.__dict__)
19             self.crawler_process.start()

crawlall.py

在settings.py 中添加配置 COMMANDS_MODULE = '項目名稱.目錄名稱'
在項目目錄執行命令：scrapy crawlall

9. 自定義擴展

自定義擴展時，利用信號在指定位置註冊制定操做

 1 from scrapy import signals
 2 
 3 
 4 class MyExtension(object):
 5     def __init__(self, value):
 6         self.value = value
 7 
 8     @classmethod
 9     def from_crawler(cls, crawler):
10         val = crawler.settings.getint('MMMM')
11         ext = cls(val)
12 
13         crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
14         crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
15 
16         return ext
17 
18     def spider_opened(self, spider):
19         print('open')
20 
21     def spider_closed(self, spider):
22         print('close')

signals

10. 避免重複訪問

scrapy默認使用 scrapy.dupefilter.RFPDupeFilter 進行去重，相關配置有：

1 DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter'
2 DUPEFILTER_DEBUG = False
3 JOBDIR = "保存範文記錄的日誌路徑，如：/root/"  # 最終路徑爲 /root/requests.seen

 1 class RepeatUrl:
 2     def __init__(self):
 3         self.visited_url = set()
 4 
 5     @classmethod
 6     def from_settings(cls, settings):
 7         """
 8         初始化時，調用
 9         :param settings: 
10         :return: 
11         """
12         return cls()
13 
14     def request_seen(self, request):
15         """
16         檢測當前請求是否已經被訪問過
17         :param request: 
18         :return: True表示已經訪問過；False表示未訪問過
19         """
20         if request.url in self.visited_url:
21             return True
22         self.visited_url.add(request.url)
23         return False
24 
25     def open(self):
26         """
27         開始爬去請求時，調用
28         :return: 
29         """
30         print('open replication')
31 
32     def close(self, reason):
33         """
34         結束爬蟲爬取時，調用
35         :param reason: 
36         :return: 
37         """
38         print('close replication')
39 
40     def log(self, request, spider):
41         """
42         記錄日誌
43         :param request: 
44         :param spider: 
45         :return: 
46         """
47         print('repeat', request.url)

自定義URL去重操做

11. 其它

  1 # -*- coding: utf-8 -*-
  2 
  3 # Scrapy settings for step8_king project
  4 #
  5 # For simplicity, this file contains only settings considered important or
  6 # commonly used. You can find more settings consulting the documentation:
  7 #
  8 #     http://doc.scrapy.org/en/latest/topics/settings.html
  9 #     http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
 10 #     http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
 11 
 12 # 1. 爬蟲名稱
 13 BOT_NAME = 'step8_king'
 14 
 15 # 2. 爬蟲應用路徑
 16 SPIDER_MODULES = ['step8_king.spiders']
 17 NEWSPIDER_MODULE = 'step8_king.spiders'
 18 
 19 # Crawl responsibly by identifying yourself (and your website) on the user-agent
 20 # 3. 客戶端 user-agent請求頭
 21 # USER_AGENT = 'step8_king (+http://www.yourdomain.com)'
 22 
 23 # Obey robots.txt rules
 24 # 4. 禁止爬蟲配置
 25 # ROBOTSTXT_OBEY = False
 26 
 27 # Configure maximum concurrent requests performed by Scrapy (default: 16)
 28 # 5. 併發請求數
 29 # CONCURRENT_REQUESTS = 4
 30 
 31 # Configure a delay for requests for the same website (default: 0)
 32 # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
 33 # See also autothrottle settings and docs
 34 # 6. 延遲下載秒數
 35 # DOWNLOAD_DELAY = 2
 36 
 37 
 38 # The download delay setting will honor only one of:
 39 # 7. 單域名訪問併發數，而且延遲下次秒數也應用在每一個域名
 40 # CONCURRENT_REQUESTS_PER_DOMAIN = 2
 41 # 單IP訪問併發數，若是有值則忽略：CONCURRENT_REQUESTS_PER_DOMAIN，而且延遲下次秒數也應用在每一個IP
 42 # CONCURRENT_REQUESTS_PER_IP = 3
 43 
 44 # Disable cookies (enabled by default)
 45 # 8. 是否支持cookie，cookiejar進行操做cookie
 46 # COOKIES_ENABLED = True
 47 # COOKIES_DEBUG = True
 48 
 49 # Disable Telnet Console (enabled by default)
 50 # 9. Telnet用於查看當前爬蟲的信息，操做爬蟲等...
 51 #    使用telnet ip port ，而後經過命令操做
 52 # TELNETCONSOLE_ENABLED = True
 53 # TELNETCONSOLE_HOST = '127.0.0.1'
 54 # TELNETCONSOLE_PORT = [6023,]
 55 
 56 
 57 # 10. 默認請求頭
 58 # Override the default request headers:
 59 # DEFAULT_REQUEST_HEADERS = {
 60 #     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 61 #     'Accept-Language': 'en',
 62 # }
 63 
 64 
 65 # Configure item pipelines
 66 # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
 67 # 11. 定義pipeline處理請求
 68 # ITEM_PIPELINES = {
 69 #    'step8_king.pipelines.JsonPipeline': 700,
 70 #    'step8_king.pipelines.FilePipeline': 500,
 71 # }
 72 
 73 
 74 
 75 # 12. 自定義擴展，基於信號進行調用
 76 # Enable or disable extensions
 77 # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
 78 # EXTENSIONS = {
 79 #     # 'step8_king.extensions.MyExtension': 500,
 80 # }
 81 
 82 
 83 # 13. 爬蟲容許的最大深度，能夠經過meta查看當前深度；0表示無深度
 84 # DEPTH_LIMIT = 3
 85 
 86 # 14. 爬取時，0表示深度優先Lifo(默認)；1表示廣度優先FiFo
 87 
 88 # 後進先出，深度優先
 89 # DEPTH_PRIORITY = 0
 90 # SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleLifoDiskQueue'
 91 # SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.LifoMemoryQueue'
 92 # 先進先出，廣度優先
 93 
 94 # DEPTH_PRIORITY = 1
 95 # SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue'
 96 # SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
 97 
 98 # 15. 調度器隊列
 99 # SCHEDULER = 'scrapy.core.scheduler.Scheduler'
100 # from scrapy.core.scheduler import Scheduler
101 
102 
103 # 16. 訪問URL去重
104 # DUPEFILTER_CLASS = 'step8_king.duplication.RepeatUrl'
105 
106 
107 # Enable and configure the AutoThrottle extension (disabled by default)
108 # See http://doc.scrapy.org/en/latest/topics/autothrottle.html
109 
110 """
111 17. 自動限速算法
112     from scrapy.contrib.throttle import AutoThrottle
113     自動限速設置
114     1. 獲取最小延遲 DOWNLOAD_DELAY
115     2. 獲取最大延遲 AUTOTHROTTLE_MAX_DELAY
116     3. 設置初始下載延遲 AUTOTHROTTLE_START_DELAY
117     4. 當請求下載完成後，獲取其"鏈接"時間 latency，即：請求鏈接到接受到響應頭之間的時間
118     5. 用於計算的... AUTOTHROTTLE_TARGET_CONCURRENCY
119     target_delay = latency / self.target_concurrency
120     new_delay = (slot.delay + target_delay) / 2.0 # 表示上一次的延遲時間
121     new_delay = max(target_delay, new_delay)
122     new_delay = min(max(self.mindelay, new_delay), self.maxdelay)
123     slot.delay = new_delay
124 """
125 
126 # 開始自動限速
127 # AUTOTHROTTLE_ENABLED = True
128 # The initial download delay
129 # 初始下載延遲
130 # AUTOTHROTTLE_START_DELAY = 5
131 # The maximum download delay to be set in case of high latencies
132 # 最大下載延遲
133 # AUTOTHROTTLE_MAX_DELAY = 10
134 # The average number of requests Scrapy should be sending in parallel to each remote server
135 # 平均每秒併發數
136 # AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
137 
138 # Enable showing throttling stats for every response received:
139 # 是否顯示
140 # AUTOTHROTTLE_DEBUG = True
141 
142 # Enable and configure HTTP caching (disabled by default)
143 # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
144 
145 
146 """
147 18. 啓用緩存
148     目的用於將已經發送的請求或相應緩存下來，以便之後使用
149     
150     from scrapy.downloadermiddlewares.httpcache import HttpCacheMiddleware
151     from scrapy.extensions.httpcache import DummyPolicy
152     from scrapy.extensions.httpcache import FilesystemCacheStorage
153 """
154 # 是否啓用緩存策略
155 # HTTPCACHE_ENABLED = True
156 
157 # 緩存策略：全部請求均緩存，下次在請求直接訪問原來的緩存便可
158 # HTTPCACHE_POLICY = "scrapy.extensions.httpcache.DummyPolicy"
159 # 緩存策略：根據Http響應頭：Cache-Control、Last-Modified 等進行緩存的策略
160 # HTTPCACHE_POLICY = "scrapy.extensions.httpcache.RFC2616Policy"
161 
162 # 緩存超時時間
163 # HTTPCACHE_EXPIRATION_SECS = 0
164 
165 # 緩存保存路徑
166 # HTTPCACHE_DIR = 'httpcache'
167 
168 # 緩存忽略的Http狀態碼
169 # HTTPCACHE_IGNORE_HTTP_CODES = []
170 
171 # 緩存存儲的插件
172 # HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
173 
174 
175 """
176 19. 代理，須要在環境變量中設置
177     from scrapy.contrib.downloadermiddleware.httpproxy import HttpProxyMiddleware
178     
179     方式一：使用默認
180         os.environ
181         {
182             http_proxy:http://root:woshiniba@192.168.11.11:9999/
183             https_proxy:http://192.168.11.11:9999/
184         }
185     方式二：使用自定義下載中間件
186     
187     def to_bytes(text, encoding=None, errors='strict'):
188         if isinstance(text, bytes):
189             return text
190         if not isinstance(text, six.string_types):
191             raise TypeError('to_bytes must receive a unicode, str or bytes '
192                             'object, got %s' % type(text).__name__)
193         if encoding is None:
194             encoding = 'utf-8'
195         return text.encode(encoding, errors)
196         
197     class ProxyMiddleware(object):
198         def process_request(self, request, spider):
199             PROXIES = [
200                 {'ip_port': '111.11.228.75:80', 'user_pass': ''},
201                 {'ip_port': '120.198.243.22:80', 'user_pass': ''},
202                 {'ip_port': '111.8.60.9:8123', 'user_pass': ''},
203                 {'ip_port': '101.71.27.120:80', 'user_pass': ''},
204                 {'ip_port': '122.96.59.104:80', 'user_pass': ''},
205                 {'ip_port': '122.224.249.122:8088', 'user_pass': ''},
206             ]
207             proxy = random.choice(PROXIES)
208             if proxy['user_pass'] is not None:
209                 request.meta['proxy'] = to_bytes（"http://%s" % proxy['ip_port']）
210                 encoded_user_pass = base64.encodestring(to_bytes(proxy['user_pass']))
211                 request.headers['Proxy-Authorization'] = to_bytes('Basic ' + encoded_user_pass)
212                 print "**************ProxyMiddleware have pass************" + proxy['ip_port']
213             else:
214                 print "**************ProxyMiddleware no pass************" + proxy['ip_port']
215                 request.meta['proxy'] = to_bytes("http://%s" % proxy['ip_port'])
216     
217     DOWNLOADER_MIDDLEWARES = {
218        'step8_king.middlewares.ProxyMiddleware': 500,
219     }
220     
221 """
222 
223 """
224 20. Https訪問
225     Https訪問時有兩種狀況：
226     1. 要爬取網站使用的可信任證書(默認支持)
227         DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory"
228         DOWNLOADER_CLIENTCONTEXTFACTORY = "scrapy.core.downloader.contextfactory.ScrapyClientContextFactory"
229         
230     2. 要爬取網站使用的自定義證書
231         DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory"
232         DOWNLOADER_CLIENTCONTEXTFACTORY = "step8_king.https.MySSLFactory"
233         
234         # https.py
235         from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
236         from twisted.internet.ssl import (optionsForClientTLS, CertificateOptions, PrivateCertificate)
237         
238         class MySSLFactory(ScrapyClientContextFactory):
239             def getCertificateOptions(self):
240                 from OpenSSL import crypto
241                 v1 = crypto.load_privatekey(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.key.unsecure', mode='r').read())
242                 v2 = crypto.load_certificate(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.pem', mode='r').read())
243                 return CertificateOptions(
244                     privateKey=v1,  # pKey對象
245                     certificate=v2,  # X509對象
246                     verify=False,
247                     method=getattr(self, 'method', getattr(self, '_ssl_method', None))
248                 )
249     其餘：
250         相關類
251             scrapy.core.downloader.handlers.http.HttpDownloadHandler
252             scrapy.core.downloader.webclient.ScrapyHTTPClientFactory
253             scrapy.core.downloader.contextfactory.ScrapyClientContextFactory
254         相關配置
255             DOWNLOADER_HTTPCLIENTFACTORY
256             DOWNLOADER_CLIENTCONTEXTFACTORY
257 
258 """
259 
260 
261 
262 """
263 21. 爬蟲中間件
264     class SpiderMiddleware(object):
265 
266         def process_spider_input(self,response, spider):
267             '''
268             下載完成，執行，而後交給parse處理
269             :param response: 
270             :param spider: 
271             :return: 
272             '''
273             pass
274     
275         def process_spider_output(self,response, result, spider):
276             '''
277             spider處理完成，返回時調用
278             :param response:
279             :param result:
280             :param spider:
281             :return: 必須返回包含 Request 或 Item 對象的可迭代對象(iterable)
282             '''
283             return result
284     
285         def process_spider_exception(self,response, exception, spider):
286             '''
287             異常調用
288             :param response:
289             :param exception:
290             :param spider:
291             :return: None,繼續交給後續中間件處理異常；含 Response 或 Item 的可迭代對象(iterable)，交給調度器或pipeline
292             '''
293             return None
294     
295     
296         def process_start_requests(self,start_requests, spider):
297             '''
298             爬蟲啓動時調用
299             :param start_requests:
300             :param spider:
301             :return: 包含 Request 對象的可迭代對象
302             '''
303             return start_requests
304     
305     內置爬蟲中間件：
306         'scrapy.contrib.spidermiddleware.httperror.HttpErrorMiddleware': 50,
307         'scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware': 500,
308         'scrapy.contrib.spidermiddleware.referer.RefererMiddleware': 700,
309         'scrapy.contrib.spidermiddleware.urllength.UrlLengthMiddleware': 800,
310         'scrapy.contrib.spidermiddleware.depth.DepthMiddleware': 900,
311 
312 """
313 # from scrapy.contrib.spidermiddleware.referer import RefererMiddleware
314 # Enable or disable spider middlewares
315 # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
316 SPIDER_MIDDLEWARES = {
317    # 'step8_king.middlewares.SpiderMiddleware': 543,
318 }
319 
320 
321 """
322 22. 下載中間件
323     class DownMiddleware1(object):
324         def process_request(self, request, spider):
325             '''
326             請求須要被下載時，通過全部下載器中間件的process_request調用
327             :param request:
328             :param spider:
329             :return:
330                 None,繼續後續中間件去下載；
331                 Response對象，中止process_request的執行，開始執行process_response
332                 Request對象，中止中間件的執行，將Request從新調度器
333                 raise IgnoreRequest異常，中止process_request的執行，開始執行process_exception
334             '''
335             pass
336     
337     
338     
339         def process_response(self, request, response, spider):
340             '''
341             spider處理完成，返回時調用
342             :param response:
343             :param result:
344             :param spider:
345             :return:
346                 Response 對象：轉交給其餘中間件process_response
347                 Request 對象：中止中間件，request會被從新調度下載
348                 raise IgnoreRequest 異常：調用Request.errback
349             '''
350             print('response1')
351             return response
352     
353         def process_exception(self, request, exception, spider):
354             '''
355             當下載處理器(download handler)或 process_request() (下載中間件)拋出異常
356             :param response:
357             :param exception:
358             :param spider:
359             :return:
360                 None：繼續交給後續中間件處理異常；
361                 Response對象：中止後續process_exception方法
362                 Request對象：中止中間件，request將會被從新調用下載
363             '''
364             return None
365 
366     
367     默認下載中間件
368     {
369         'scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware': 100,
370         'scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware': 300,
371         'scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware': 350,
372         'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400,
373         'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 500,
374         'scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware': 550,
375         'scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware': 580,
376         'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware': 590,
377         'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600,
378         'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware': 700,
379         'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 750,
380         'scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware': 830,
381         'scrapy.contrib.downloadermiddleware.stats.DownloaderStats': 850,
382         'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900,
383     }
384 
385 """
386 # from scrapy.contrib.downloadermiddleware.httpauth import HttpAuthMiddleware
387 # Enable or disable downloader middlewares
388 # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
389 # DOWNLOADER_MIDDLEWARES = {
390 #    'step8_king.middlewares.DownMiddleware1': 100,
391 #    'step8_king.middlewares.DownMiddleware2': 500,
392 # }

settings配置

12. TinyScrapy

  1 # -*- coding: utf-8 -*-
  2 
  3 from twisted.web.client import getPage, defer
  4 from twisted.internet import reactor
  5 import queue
  6 
  7 
  8 class Response(object):
  9     def __init__(self, body, request):
 10         self.body = body
 11         self.request = request
 12         self.url = request.url
 13 
 14     @property
 15     def text(self):
 16         return self.body.decode('utf-8')
 17 
 18 
 19 class Request(object):
 20     def __init__(self, url, callback=None):
 21         self.url = url
 22         self.callback = callback
 23 
 24 
 25 class Scheduler(object):
 26     def __init__(self, engine):
 27         self.q = queue.Queue()
 28         self.engine = engine
 29 
 30     def enqueue_request(self, request):
 31         self.q.put(request)
 32 
 33     def next_request(self):
 34         try:
 35             req = self.q.get(block=False)
 36         except Exception as e:
 37             req = None
 38         return req
 39 
 40     def size(self):
 41         return self.q.qsize()
 42 
 43 
 44 class ExecutionEngine(object):
 45     def __init__(self):
 46         self._closewait = None
 47         self.running = True
 48         self.start_requests = None
 49         self.scheduler = Scheduler(self)
 50         self.inprogress = set()
 51 
 52     def check_empty(self, response):
 53         if not self.running:
 54             self._closewait.callback()
 55 
 56     def _nex_request(self):
 57         while self.start_requests:
 58             try:
 59                 request = next(self.start_requests)
 60             except StopIteration:
 61                 self.start_requests = None
 62             else:
 63                 self.scheduler.enqueue_request(request)
 64         while len(self.inprogress) < 5 and self.scheduler.size() > 0:   # 最大併發數爲5
 65             request = self.scheduler.next_request()
 66             if not request:
 67                 break
 68             self.inprogress.add(request)
 69             d = getPage(bytes(request.url, encoding='utf-8'))
 70             d.addBoth(self._handle_downloader_output, request)
 71             d.addBoth(lambda x, req: self.inprogress.remove(req), request)
 72             d.addBoth(lambda x: self._nex_request())
 73         if not len(self.inprogress) and not self.scheduler.size():
 74             self._closewait.callback(None)
 75 
 76     def _handle_downloader_output(self, body, request):
 77         """
 78         獲取內容，執行回調函數，而且把回調函數中的返回值獲取，並添加到隊列中
 79         :param body:
 80         :param request:
 81         :return:
 82         """
 83         import types
 84         response = Response(body, request)
 85         func = request.callback or self.spider.parse
 86         gen = func(response)
 87         if isinstance(gen, types.GeneratorType):
 88             for req in gen:
 89                 self.scheduler.enqueue_request(req)
 90 
 91     @defer.inlineCallbacks
 92     def start(self):
 93         self._closewait = defer.Deferred()
 94         yield self._closewait
 95 
 96     @defer.inlineCallbacks
 97     def open_spider(self, spider, start_requests):
 98         self.start_requests = start_requests
 99         self.spider = spider
100         yield None
101         reactor.callLater(0, self._nex_request)
102 
103 
104 class Crawler(object):
105     def __init__(self, spidercls):
106         self.spidercls = spidercls
107         self.spider = None
108         self.engine = None
109 
110     @defer.inlineCallbacks
111     def crawl(self):
112         self.engine = ExecutionEngine()
113         self.spider = self.spidercls()
114         start_requests = iter(self.spider.start_requests())
115         yield self.engine.open_spider(self.spider, start_requests)
116         yield self.engine.start()
117 
118 
119 class CrawlerProcess(object):
120     def __init__(self):
121         self._active = set()
122         self.crawlers = set()
123 
124     def crawl(self, spidercls, *args, **kwargs):
125         crawler = Crawler(spidercls)
126         self.crawlers.add(crawler)
127         d = crawler.crawl(*args, **kwargs)
128         self._active.add(d)
129         return d
130 
131     def start(self):
132         d1 = defer.DeferredList(self._active)
133         d1.addBoth(self._stop_reactor)
134         reactor.run()
135 
136     def _stop_reactor(self, _=None):
137         reactor.stop()
138 
139 
140 class Spider(object):
141     def start_requests(self):
142         for url in self.start_urls:
143             yield Request(url)
144 
145 
146 class ChoutiSpider(Spider):
147     name = "chouti"
148     start_urls = [
149         'http://dig.chouti.com/',
150     ]
151 
152     def parse(self, response):
153         print(response.text)
154 
155 
156 class CnblogsSpider(Spider):
157     name = "cnblogs"
158     start_urls = [
159         'http://www.cnblogs.com/',
160     ]
161 
162     def parse(self, response):
163         print(response.text)
164 
165 
166 if __name__ == '__main__':
167     spider_cls_list = [ChoutiSpider, CnblogsSpider]
168     crawler_process = CrawlerProcess()
169     for spider_cls in spider_cls_list:
170         crawler_process.crawl(spider_cls)
171     crawler_process.start()