Scrapy

目錄

1、簡介

2、安裝和基本使用

三. 選擇器

四. 格式化和持久化處理(item+pipelines)

五.爬蟲中間件

六. 自定製命令

七. 自定義擴展

八. 避免重複訪問

九.配置文件和示例

1、簡介

Scrapy是一個爲了爬取網站數據，提取結構性數據而編寫的應用框架。 其能夠應用在數據挖掘，信息處理或存儲歷史數據等一系列的程序中。
其最初是爲了頁面抓取 (更確切來講, 網絡抓取 )所設計的， 也能夠應用在獲取API所返回的數據(例如 Amazon Associates Web Services ) 或者通用的網絡爬蟲。Scrapy用途普遍，能夠用於數據挖掘、監測和自動化測試。

Scrapy 使用了 Twisted異步網絡庫來處理網絡通信。總體架構大體以下

一、Scrapy主要包括瞭如下組件：

• 引擎(Scrapy)
  用來處理整個系統的數據流處理, 觸發事務(框架核心)
• 調度器(Scheduler)
  用來接受引擎發過來的請求, 壓入隊列中, 並在引擎再次請求的時候返回. 能夠想像成一個URL（抓取網頁的網址或者說是連接）的優先隊列, 由它來決定下一個要抓取的網址是什麼, 同時去除重複的網址
• 下載器(Downloader)
  用於下載網頁內容, 並將網頁內容返回給蜘蛛(Scrapy下載器是創建在twisted這個高效的異步模型上的)
• 爬蟲(Spiders)
  爬蟲是主要幹活的, 用於從特定的網頁中提取本身須要的信息, 即所謂的實體(Item)。用戶也能夠從中提取出連接,讓Scrapy繼續抓取下一個頁面
• 項目管道(Pipeline)
  負責處理爬蟲從網頁中抽取的實體，主要的功能是持久化實體、驗證明體的有效性、清除不須要的信息。當頁面被爬蟲解析後，將被髮送到項目管道，並通過幾個特定的次序處理數據。
• 下載器中間件(Downloader Middlewares)
  位於Scrapy引擎和下載器之間的框架，主要是處理Scrapy引擎與下載器之間的請求及響應。
• 爬蟲中間件(Spider Middlewares)
  介於Scrapy引擎和爬蟲之間的框架，主要工做是處理蜘蛛的響應輸入和請求輸出。
• 調度中間件(Scheduler Middewares)
  介於Scrapy引擎和調度之間的中間件，從Scrapy引擎發送到調度的請求和響應。

二、Scrapy運行流程大概以下：

1. 引擎從調度器中取出一個連接(URL)用於接下來的抓取
2. 引擎把URL封裝成一個請求(Request)傳給下載器
3. 下載器把資源下載下來，並封裝成應答包(Response)
4. 爬蟲解析Response
5. 解析出實體（Item）,則交給實體管道進行進一步的處理
6. 解析出的是連接（URL）,則把URL交給調度器等待抓取

2、安裝和基本使用

Linux
      pip3 install scrapy
 
 
Windows
      a. pip3 install wheel
      b. 下載twisted http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
      c. 進入下載目錄，執行 pip3 install Twisted‑17.1.0‑cp35‑cp35m‑win_amd64.whl
      d. pip3 install scrapy
      e. 下載並安裝pywin32：https://sourceforge.net/projects/pywin32/files/

1. 基本命令

1. scrapy startproject 項目名稱
   - 在當前目錄中建立中建立一個項目文件（相似於Django）
 
2. scrapy genspider [-t template] <name> <domain>
   - 建立爬蟲應用
   如：
      scrapy gensipider -t basic oldboy oldboy.com
      scrapy gensipider -t xmlfeed autohome autohome.com.cn
   PS:
      查看全部命令：scrapy gensipider -l
      查看模板命令：scrapy gensipider -d 模板名稱
 
3. scrapy list
   - 展現爬蟲應用列表
 
4. scrapy crawl 爬蟲應用名稱
   - 運行單獨爬蟲應用

2.項目結構以及爬蟲應用簡介

project_name/
   scrapy.cfg
   project_name/
       __init__.py
       items.py
       pipelines.py
       settings.py
       spiders/
           __init__.py
           爬蟲1.py
           爬蟲2.py
           爬蟲3.py

文件說明：

• scrapy.cfg  項目的主配置信息。（真正爬蟲相關的配置信息在settings.py文件中）
• items.py    設置數據存儲模板，用於結構化數據，如：Django的Model
• pipelines    數據處理行爲，如：通常結構化的數據持久化
• settings.py 配置文件，如：遞歸的層數、併發數，延遲下載等
• spiders      爬蟲目錄，如：建立文件，編寫爬蟲規則

注意：通常建立爬蟲文件時，以網站域名命名

• 爬蟲1.py

import scrapy
 
class XiaoHuarSpider(scrapy.spiders.Spider):
    name = "xiaohuar"                            # 爬蟲名稱 *****
    allowed_domains = ["xiaohuar.com"]  # 容許的域名，只能在這個域名內爬取
    start_urls = [
        "http://www.xiaohuar.com/hua/",   # 其實URL
    ]
 
    def parse(self, response):
        # 訪問起始URL並獲取結果後的回調函數

 三、 response參數中封裝了不少參數

response.url
response.text
response.body
response.meta['depth']  # 當前深度,能夠經過判斷當前深度到了哪一層，來選擇中止時間

四、關於Windows編碼，若是在Windows上執行代碼，輸出response.text爲空的話，就加上這一段代碼

import sys,io
sys.stdout=io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030')

5. 小試牛刀

import scrapy
from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
 
 
class DigSpider(scrapy.Spider):
    # 爬蟲應用的名稱，經過此名稱啓動爬蟲命令
    name = "dig"
 
    # 容許的域名
    allowed_domains = ["chouti.com"]
 
    # 起始URL
    start_urls = [
        'http://dig.chouti.com/',
    ]
 
    has_request_set = {}
 
    def parse(self, response):
        print(response.url)
 
        hxs = HtmlXPathSelector(response)
        page_list = hxs.select('//div[@id="dig_lcpage"]//a[re:test(@href, "/all/hot/recent/\d+")]/@href').extract()
        for page in page_list:
            page_url = 'http://dig.chouti.com%s' % page
            key = self.md5(page_url)
            if key in self.has_request_set:
                pass
            else:
                self.has_request_set[key] = page_url
                obj = Request(url=page_url, method='GET', callback=self.parse)
                yield obj
 
    @staticmethod
    def md5(val):
        import hashlib
        ha = hashlib.md5()
        ha.update(bytes(val, encoding='utf-8'))
        key = ha.hexdigest()
        return key

• 執行此爬蟲文件，則在終端進入項目目錄執行以下命令：

scrapy crawl dig --nolog
# 記得在settings.py文件中修改ROBOTSTXT_OBEY = False，不然什麼都不打印

　　對於上述代碼重要之處在於：

• Request是一個封裝用戶請求的類，在回調函數中yield該對象表示繼續訪問
• HtmlXpathSelector用於結構化HTML代碼並提供選擇器功能

三. 選擇器

設置遞歸層數：

• settings.py中設置DEPTH_LIMIT = 1來指定「遞歸」的層數，0表示沒有限制，1表示在當前頁的基礎上再多訪問一層，好比首頁10個頁碼，1就是訪問1-14個頁碼

經常使用選擇器：

import scrapy
import hashlib
from scrapy.http import Request
from scrapy.selector import Selector,HtmlXPathSelector

class ChoutiSpider(scrapy.Spider):
    name = 'chouti'
    allowed_domains = ['chouti.com']
    start_urls = ['http://dig.chouti.com/']
    visited_urls = set() # 對URL去重

    # 程序初始化的時候會自動執行self.parse方法
    # 若是你想修改parse方法，就要加上這一段源碼，而後從新寫parse方法
    # def start_requests(self):
    #     for url in self.start_urls:
    #         yield Request(url=url,callback=self.parse)


    def parse(self, response):
        # 找到全部的A標籤對象列表
        # hxs = Selector(response=response).xpath('//a').extract()
        # for i in hxs:
        #     print(i)
        # //從全部文檔標籤中找，/從全部兒子中找，@後面加屬性參數
        # hxs = Selector(response=response).xpath('//div[@id="content-list"]/div[@class="item"]')
        # for obj in hxs:
            # print(obj)
            # .//表示從當前對象的子孫中尋找，./表示從當前對象的兒子中找，若是有兩個class都要寫上，不然找不到
            # /text()獲取文本，
            # extract（）把列表中每個對象轉換成字符串,
            # extract_first()把列表中每個對象轉換成字符串,並返回第一個元素
            # a = obj.xpath('.//a[@class="show-content color-chag"]/text()').extract_first()
            # print(a.strip())
        # 獲取當前頁的全部頁碼,/@屬性名--獲取標籤對象的屬性，/ul/li/a能夠直接寫//a
        # hxs = Selector(response=response).xpath('//div[@id="dig_lcpage"]//a/@href').extract()
        # starts-with尋找標籤屬性值是以什麼開頭的
        # hxs = Selector(response=response).xpath('//a[starts-with(@href, "/all/hot/recent/")]/@href').extract()
        # 正則表達式匹配re:test
        hxs = Selector(response=response).xpath('//a[re:test(@href, "/all/hot/recent/\d+")]/@href').extract()
        for url in hxs:
            md5_url = self.md5(url)
            if md5_url in self.visited_urls:
                # 訪問過的無論
                pass
            else:
                self.visited_urls.add(md5_url)
                print(url)# /all/hot/recent/10
                url = 'http://dig.chouti.com/%s'%url
                # 遞歸爬取全部頁碼，必須加yield，引擎幫咱們將新要訪問的URL放入調度器
                # 一次下載完成以後回調函數也能夠是其餘函數，可是就沒法實現遞歸了，因此仍是用parse
                yield Request(url=url,callback=self.parse)

    def md5(self,url):
        """若是URL特別長，數據庫中創建索引費空間，因此先加密再放入"""
        obj = hashlib.md5()
        obj.update(bytes(url,encoding='utf-8'))
        return obj.hexdigest()

更多選擇器：

from scrapy.selector import Selector, HtmlXPathSelector
from scrapy.http import HtmlResponse

html = """<!DOCTYPE html>
<html>
    <head lang="en">
        <meta charset="UTF-8">
        <title></title>
    </head>
    <body>
        <ul>
            <li class="item-"><a id='i1' href="link.html">first item</a></li>
            <li class="item-0"><a id='i2' href="llink.html">first item</a></li>
            <li class="item-1"><a href="llink2.html">second item<span>vv</span></a></li>
        </ul>
        <div><a href="llink2.html">second item</a></div>
    </body>
</html>
"""
response = HtmlResponse(url='http://example.com', body=html, encoding='utf-8')
hxs = HtmlXPathSelector(response)
print(hxs)
hxs = Selector(response=response).xpath('//a')# 找到全部的a標籤
print(hxs)
hxs = Selector(response=response).xpath('//a[2]')# 找到列表裏索引爲2的標籤
print(hxs)
hxs = Selector(response=response).xpath('//a[@id]')# 找到a標籤中有id屬性的全部標籤
print(hxs)
hxs = Selector(response=response).xpath('//a[@id="i1"]')# 找到a標籤id=i1的全部標籤
print(hxs)
hxs = Selector(response=response).xpath('//a[@href="link.html"][@id="i1"]')# 寫兩個條件就是且的意思
print(hxs)
hxs = Selector(response=response).xpath('//a[contains(@href, "link")]')# a標籤的href屬性中包含link
print(hxs)
hxs = Selector(response=response).xpath('//a[starts-with(@href, "link")]')# a標籤的href屬性以xx開頭
print(hxs)
hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]') # 正則表達式
print(hxs)
hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]/text()').extract() # 取標籤對象的文本
print(hxs)
hxs = Selector(response=response).xpath('//a[re:test(@id, "i\d+")]/@href').extract()# 取標籤對象的href屬性
print(hxs)
hxs = Selector(response=response).xpath('/html/body/ul/li/a/@href').extract()# /一個孩子一個孩子的找，不會越級
print(hxs)
hxs = Selector(response=response).xpath('//body/ul/li/a/@href').extract_first()
print(hxs)

ul_list = Selector(response=response).xpath('//body/ul/li')
for item in ul_list:
    v = item.xpath('./a/span')# 從當前對象的兒子中找
    # 或
    # v = item.xpath('a/span')
    # 或
    # v = item.xpath('*/a/span')
    print(v)

四. 格式化和持久化處理(item+pipelines)

一、上述實例只是簡單的處理，因此在parse方法中直接處理。若是對於想要獲取更多的數據處理，則能夠利用Scrapy的items將數據格式化，而後統一交由pipelines來處理。

• Spider01/spiders/xiaohuar.py

# -*- coding: utf-8 -*-
import scrapy
from scrapy.selector import Selector
from ..items import XiaohuaItem
import hashlib
from scrapy.http import Request

class XiaohuarSpider(scrapy.Spider):
    name = 'xiaohuar'
    allowed_domains = ['xiaohuar.com']
    start_urls = ['http://www.xiaohuar.com/hua']
    visited_urls = set()

    def parse(self, response):
        items = Selector(response=response).xpath('//div[@class="item_list infinite_scroll"]/div')
        for obj in items:
            src = obj.xpath('.//div[@class="img"]/a/img/@src').extract_first()
            name = obj.xpath('.//div[@class="img"]/span/text()').extract_first()
            school = obj.xpath('.//div[@class="img"]/div/a/text()').extract_first()
            # print(school)
            if 'http' in src:
                url = src
            else:
                url = 'http://www.xiaohuar.com%s' % src
            item_obj = XiaohuaItem(name=name,url=url,school=school)
            yield item_obj

        urls = Selector(response=response).xpath('//a[re:test(@href,"http://www.xiaohuar.com/list-1-\d+.html")]/@href').extract()
        for url in urls:
            md5_url = self.md5(url)
            if md5_url in self.visited_urls:
                pass
            else:
                self.visited_urls.add(md5_url)
                yield Request(url=url,callback=self.parse)


    def md5(self, url):
        """若是URL特別長，數據庫中創建索引費空間，因此先加密再放入"""
        obj = hashlib.md5()
        obj.update(bytes(url, encoding='utf-8'))
        return obj.hexdigest()

• items

class XiaohuaItem(scrapy.Item):
    name = scrapy.Field()
    school = scrapy.Field()
    url = scrapy.Field()

• piplines

import requests
import os

class JsonPipeline(object):
    """保存到json文件"""
    def process_item(self, item, spider):
        # if spider.name == 'chouti':
        tpl = '%s\n%s\n%s\n\n'%(item['school'],item['name'],item['url'])
        # print(tpl)
        file = open('files/xiaohuar.json', 'a')
        file.write(tpl)
        file.close()
        return item

class FilePipeline(object):
    """保存圖片"""
    def __init__(self):
        if not os.path.exists('imgs'):
            os.makedirs('imgs')
    def process_item(self, item, spider):
        response = requests.get(url=item['url'],stream=True)
        file_name = '%s-%s.jpg'%(item['school'],item['name'])
        with open(os.path.join('imgs',file_name),'wb') as f:
            f.write(response.content)
        return item

• settings.py

ITEM_PIPELINES = {
   'spider1.pipelines.JsonPipeline': 100,
   'spider1.pipelines.FilePipeline': 300,
}
# 每行後面的整型值，肯定了他們運行的順序，item按數字從低到高的順序，經過pipeline，一般將這些數字定義在0-1000範圍內。

二、對於pipeline能夠作更多，自定義pipeline：

import requests
import os
from scrapy.exceptions import DropItem

class JsonPipeline(object):
    """保存到json文件"""

    def __init__(self, conn_str):
        self.conn_str = conn_str
        self.conn = None

    def process_item(self, item, spider):
        """
        操做並進行持久化
        return item 表示會被後續的pipeline繼續處理
        raise DropItem() 表示將item丟棄，不會被後續pipeline處理
        要麼返回item，要麼拋出丟棄異常
        """
        tpl = '%s\n%s\n\n'%(item['title'],item['href'])
        self.conn.write(tpl)
        return item

    @classmethod
    def from_crawler(cls, crawler):
        """
        初始化時候，用於建立pipeline對象,拿到配置文件中的文件路徑
        :param crawler:
        :return:
        """
        conn_str = crawler.settings.get('DB')
        return cls(conn_str)

    def open_spider(self, spider):
        """
        爬蟲開始執行時，調用，打開文件，只須要打開一次
        :param spider:
        :return:
        """
        self.conn = open(self.conn_str,'a')
        print('open...')

    def close_spider(self, spider):
        """
        爬蟲關閉時，被調用，關閉文件
        :param spider:
        :return:
        """
        self.conn.close()
        print('close...')

# 配置文件中寫上，DB必須大寫
DB = 'files/chouti.json'

五.爬蟲中間件

• 爬蟲中間件

class SpiderMiddleware(object):

    def process_spider_input(self,response, spider):
        """
        下載完成，執行，而後交給parse處理
        :param response: 
        :param spider: 
        :return: 
        """
        pass

    def process_spider_output(self,response, result, spider):
        """
        spider處理完成，返回時調用
        :param response:
        :param result:
        :param spider:
        :return: 必須返回包含 Request 或 Item 對象的可迭代對象(iterable)
        """
        return result

    def process_spider_exception(self,response, exception, spider):
        """
        異常調用
        :param response:
        :param exception:
        :param spider:
        :return: None,繼續交給後續中間件處理異常；含 Response 或 Item 的可迭代對象(iterable)，交給調度器或pipeline
        """
        return None


    def process_start_requests(self,start_requests, spider):
        """
        爬蟲啓動時調用
        :param start_requests:
        :param spider:
        :return: 包含 Request 對象的可迭代對象
        """
        return start_requests

View Code

• 下載器中間件

class DownMiddleware1(object):
    def process_request(self, request, spider):
        """
        請求須要被下載時，通過全部下載器中間件的process_request調用
        :param request: 
        :param spider: 
        :return:  
            None,繼續後續中間件去下載；
            Response對象，中止process_request的執行，開始執行process_response
            Request對象，中止中間件的執行，將Request從新調度器
            raise IgnoreRequest異常，中止process_request的執行，開始執行process_exception
        """
        pass



    def process_response(self, request, response, spider):
        """
        spider處理完成，返回時調用
        :param response:
        :param result:
        :param spider:
        :return: 
            Response 對象：轉交給其餘中間件process_response
            Request 對象：中止中間件，request會被從新調度下載
            raise IgnoreRequest 異常：調用Request.errback
        """
        print('response1')
        return response

    def process_exception(self, request, exception, spider):
        """
        當下載處理器(download handler)或 process_request() (下載中間件)拋出異常
        :param response:
        :param exception:
        :param spider:
        :return: 
            None：繼續交給後續中間件處理異常；
            Response對象：中止後續process_exception方法
            Request對象：中止中間件，request將會被從新調用下載
        """
        return None

View Code

六. 自定製命令

• 在spiders同級建立任意目錄，如：commands
• 在其中建立 crawlall.py 文件 （此處文件名就是自定義的命令）

from scrapy.commands import ScrapyCommand
from scrapy.utils.project import get_project_settings

    class Command(ScrapyCommand):

        requires_project = True

        def syntax(self):
            return '[options]'

        def short_desc(self):
            return 'Runs all of the spiders'

        def run(self, args, opts):
            """
            self.crawler_process是CrawlerProcess的對象，包含全部的爬蟲和配置文件
            執行CrawlerProcess的構造方法，
            獲取全部的爬蟲列表
            """
            # spider_list = ['chouti','cbblogs']能夠隨意控制運行的爬蟲
            spider_list = self.crawler_process.spiders.list()
            for name in spider_list:
                self.crawler_process.crawl(name, **opts.__dict__)
            self.crawler_process.start()    

• 在settings.py 中添加配置 COMMANDS_MODULE = '項目名稱.目錄名稱'

• 在項目目錄執行命令：scrapy crawlall 

# 單個爬蟲
import sys
from scrapy.cmdline import execute

if __name__ == '__main__':
    execute(["scrapy","github","--nolog"])

七. 自定義擴展

自定義擴展時，利用信號在指定位置註冊指定操做

修改配置文件：

EXTENSIONS = {
   'Spider01.extention.MyExtension': 300,
}

from scrapy import signals


class MyExtension(object):
    def __init__(self, value):
        self.value = value

    @classmethod
    def from_crawler(cls, crawler):
        val = crawler.settings.getint('MMMM')
        obj = cls(val)

        crawler.signals.connect(obj.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(obj.spider_closed, signal=signals.spider_closed)

        return obj

    def spider_opened(self, spider):
        print('open')

    def spider_closed(self, spider):
        print('close')

八. 避免重複訪問

scrapy默認使用 scrapy.dupefilter.RFPDupeFilter 進行去重，相關配置有：

DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter'
DUPEFILTER_DEBUG = False
JOBDIR = "保存範文記錄的日誌路徑，如：/root/"  # 最終路徑爲 /root/requests.seen

•   自定義URL去重操做

class RepeatUrl:
    def __init__(self):
        self.visited_url = set()

    @classmethod
    def from_settings(cls, settings):
        """
        初始化時，調用
        :param settings: 
        :return: 
        """
        return cls()

    def request_seen(self, request):
        """
        檢測當前請求是否已經被訪問過
        :param request: 
        :return: True表示已經訪問過；False表示未訪問過
        """
        if request.url in self.visited_url:
            return True
        self.visited_url.add(request.url)
        return False

    def open(self):
        """
        開始爬去請求時，調用
        :return: 
        """
        print('open replication')

    def close(self, reason):
        """
        結束爬蟲爬取時，調用
        :param reason: 
        :return: 
        """
        print('close replication')

    def log(self, request, spider):
        """
        記錄日誌
        :param request: 
        :param spider: 
        :return: 
        """
        print('repeat', request.url)

View Code

九.配置文件和示例

• settings.py

# 1. 爬蟲名稱
BOT_NAME = 'step8_king'

# 2. 爬蟲應用路徑
SPIDER_MODULES = ['step8_king.spiders']
NEWSPIDER_MODULE = 'step8_king.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
# 3. 客戶端 user-agent請求頭
# USER_AGENT = 'step8_king (+http://www.yourdomain.com)'
# 能夠改爲瀏覽器
#USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'


# Obey robots.txt rules
# 4. 禁止爬蟲配置，就是不遵循爬蟲限制規則
# ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
# 5. 併發請求數，最大32個爬蟲一塊兒
# CONCURRENT_REQUESTS = 4

# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# 6. 延遲下載秒數，每2秒執行一個，太快容易被封號
# DOWNLOAD_DELAY = 2


# The download delay setting will honor only one of:
# 7. 單域名訪問併發數，而且延遲下次秒數也應用在每一個域名
# CONCURRENT_REQUESTS_PER_DOMAIN = 2
# 單IP訪問併發數，若是有值則忽略：CONCURRENT_REQUESTS_PER_DOMAIN，而且延遲下次秒數也應用在每一個IP
# CONCURRENT_REQUESTS_PER_IP = 3

# Disable cookies (enabled by default)
# 8. 是否支持cookie，cookiejar進行操做cookie
# COOKIES_ENABLED = True 默認表示支持
# COOKIES_DEBUG = True 是不是調試模式

# Disable Telnet Console (enabled by default)
# 9. Telnet用於查看當前爬蟲的信息，操做爬蟲等...
#    使用telnet ip port ，而後經過命令操做
# TELNETCONSOLE_ENABLED = True
# TELNETCONSOLE_HOST = '127.0.0.1'
# TELNETCONSOLE_PORT = [6023,] #scrapy默認端口


# 10. 默認請求頭
# Override the default request headers:
# DEFAULT_REQUEST_HEADERS = {
#     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#     'Accept-Language': 'en',
# }


# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
# 11. 定義pipeline處理請求的優先級，權重越大優先級越低
# ITEM_PIPELINES = {
#    'step8_king.pipelines.JsonPipeline': 700,後執行
#    'step8_king.pipelines.FilePipeline': 500, 先執行
# }



# 12. 自定義擴展，基於信號進行調用
# 自定義類，重寫signals中規定的方法，把本身的方法註冊到signals
# Enable or disable extensions
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
# EXTENSIONS = {
#     # 'step8_king.extensions.MyExtension': 500,
# }


# 13. 爬蟲容許的最大深度，能夠經過meta查看當前深度；0表示無深度
# DEPTH_LIMIT = 3

# 14. 爬取時，0表示深度優先Lifo(默認)；1表示廣度優先FiFo

# 後進先出，深度優先
# DEPTH_PRIORITY = 0
# SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleLifoDiskQueue'
# SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.LifoMemoryQueue'
# 先進先出，廣度優先

# DEPTH_PRIORITY = 1
# SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue'
# SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'

# 15. 調度器隊列
# SCHEDULER = 'scrapy.core.scheduler.Scheduler'
# from scrapy.core.scheduler import Scheduler


# 16. 訪問URL去重
# DUPEFILTER_CLASS = 'step8_king.duplication.RepeatUrl'


# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html

"""
17. 自動限速算法
    from scrapy.contrib.throttle import AutoThrottle
    自動限速設置
    1. 獲取最小延遲 DOWNLOAD_DELAY
    2. 獲取最大延遲 AUTOTHROTTLE_MAX_DELAY
    3. 設置初始下載延遲 AUTOTHROTTLE_START_DELAY
    4. 當請求下載完成後，獲取其"鏈接"時間 latency，即：請求鏈接到接受到響應頭之間的時間
    5. 用於計算的... AUTOTHROTTLE_TARGET_CONCURRENCY
    target_delay = latency / self.target_concurrency
    new_delay = (slot.delay + target_delay) / 2.0 #計算下一次的延遲時間
    new_delay = max(target_delay, new_delay)
    new_delay = min(max(self.mindelay, new_delay), self.maxdelay)
    slot.delay = new_delay
"""

# 開始自動限速
# AUTOTHROTTLE_ENABLED = True
# The initial download delay
# 初始下載延遲
# AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
# 最大下載延遲
# AUTOTHROTTLE_MAX_DELAY = 10
# The average number of requests Scrapy should be sending in parallel to each remote server
# 平均每秒併發數
# AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

# Enable showing throttling stats for every response received:
# 是否顯示
# AUTOTHROTTLE_DEBUG = True

# Enable and configure HTTP caching (disabled by default)
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings


"""
18. 啓用緩存
    目的用於將已經發送的請求或相應緩存下來，以便之後使用
    
    from scrapy.downloadermiddlewares.httpcache import HttpCacheMiddleware
    from scrapy.extensions.httpcache import DummyPolicy
    from scrapy.extensions.httpcache import FilesystemCacheStorage
"""
# 是否啓用緩存策略
# HTTPCACHE_ENABLED = True

# 緩存策略：全部請求均緩存，下次在請求直接訪問原來的緩存便可
# HTTPCACHE_POLICY = "scrapy.extensions.httpcache.DummyPolicy"
# 緩存策略：根據Http響應頭：Cache-Control、Last-Modified 等進行緩存的策略
# HTTPCACHE_POLICY = "scrapy.extensions.httpcache.RFC2616Policy"

# 緩存超時時間
# HTTPCACHE_EXPIRATION_SECS = 0

# 緩存保存路徑
# HTTPCACHE_DIR = 'httpcache'

# 緩存忽略的Http狀態碼
# HTTPCACHE_IGNORE_HTTP_CODES = []

# 緩存存儲的插件
# HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'


"""
19. 代理，須要在環境變量中設置
    from scrapy.contrib.downloadermiddleware.httpproxy import HttpProxyMiddleware
    
    方式一：使用默認
        os.environ
        {
            http_proxy:http://root:woshiniba@192.168.11.11:9999/
            https_proxy:http://192.168.11.11:9999/
        }
    方式二：使用自定義下載中間件
    
    def to_bytes(text, encoding=None, errors='strict'):
        if isinstance(text, bytes):
            return text
        if not isinstance(text, six.string_types):
            raise TypeError('to_bytes must receive a unicode, str or bytes '
                            'object, got %s' % type(text).__name__)
        if encoding is None:
            encoding = 'utf-8'
        return text.encode(encoding, errors)
        
    class ProxyMiddleware(object):
        def process_request(self, request, spider):
            PROXIES = [
                {'ip_port': '111.11.228.75:80', 'user_pass': ''},
                {'ip_port': '120.198.243.22:80', 'user_pass': ''},
                {'ip_port': '111.8.60.9:8123', 'user_pass': ''},
                {'ip_port': '101.71.27.120:80', 'user_pass': ''},
                {'ip_port': '122.96.59.104:80', 'user_pass': ''},
                {'ip_port': '122.224.249.122:8088', 'user_pass': ''},
            ]
            proxy = random.choice(PROXIES)
            if proxy['user_pass'] is not None:
                request.meta['proxy'] = to_bytes（"http://%s" % proxy['ip_port']）
                encoded_user_pass = base64.encodestring(to_bytes(proxy['user_pass']))
                request.headers['Proxy-Authorization'] = to_bytes('Basic ' + encoded_user_pass)
                print "**************ProxyMiddleware have pass************" + proxy['ip_port']
            else:
                print "**************ProxyMiddleware no pass************" + proxy['ip_port']
                request.meta['proxy'] = to_bytes("http://%s" % proxy['ip_port'])
    
    DOWNLOADER_MIDDLEWARES = {
       'step8_king.middlewares.ProxyMiddleware': 500,
    }
    
"""

"""
20. Https訪問
    Https訪問時有兩種狀況：
    1. 要爬取網站使用的可信任證書(默認支持)
        DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory"
        DOWNLOADER_CLIENTCONTEXTFACTORY = "scrapy.core.downloader.contextfactory.ScrapyClientContextFactory"
        
    2. 要爬取網站使用的自定義證書
        DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory"
        DOWNLOADER_CLIENTCONTEXTFACTORY = "step8_king.https.MySSLFactory"
        
        # https.py
        from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
        from twisted.internet.ssl import (optionsForClientTLS, CertificateOptions, PrivateCertificate)
        
        class MySSLFactory(ScrapyClientContextFactory):
            def getCertificateOptions(self):
                from OpenSSL import crypto
                v1 = crypto.load_privatekey(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.key.unsecure', mode='r').read())
                v2 = crypto.load_certificate(crypto.FILETYPE_PEM, open('/Users/wupeiqi/client.pem', mode='r').read())
                return CertificateOptions(
                    privateKey=v1,  # pKey對象
                    certificate=v2,  # X509對象
                    verify=False,
                    method=getattr(self, 'method', getattr(self, '_ssl_method', None))
                )
    其餘：
        相關類
            scrapy.core.downloader.handlers.http.HttpDownloadHandler
            scrapy.core.downloader.webclient.ScrapyHTTPClientFactory
            scrapy.core.downloader.contextfactory.ScrapyClientContextFactory
        相關配置
            DOWNLOADER_HTTPCLIENTFACTORY
            DOWNLOADER_CLIENTCONTEXTFACTORY

"""



"""
21. 爬蟲中間件
    class SpiderMiddleware(object):

        def process_spider_input(self,response, spider):
            '''
            下載完成，執行，而後交給parse處理
            :param response: 
            :param spider: 
            :return: 
            '''
            pass
    
        def process_spider_output(self,response, result, spider):
            '''
            spider處理完成，返回時調用
            :param response:
            :param result:
            :param spider:
            :return: 必須返回包含 Request 或 Item 對象的可迭代對象(iterable)
            '''
            return result
    
        def process_spider_exception(self,response, exception, spider):
            '''
            異常調用
            :param response:
            :param exception:
            :param spider:
            :return: None,繼續交給後續中間件處理異常；含 Response 或 Item 的可迭代對象(iterable)，交給調度器或pipeline
            '''
            return None
    
    
        def process_start_requests(self,start_requests, spider):
            '''
            爬蟲啓動時調用
            :param start_requests:
            :param spider:
            :return: 包含 Request 對象的可迭代對象
            '''
            return start_requests
    
    內置爬蟲中間件：
        'scrapy.contrib.spidermiddleware.httperror.HttpErrorMiddleware': 50,
        'scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware': 500,
        'scrapy.contrib.spidermiddleware.referer.RefererMiddleware': 700,
        'scrapy.contrib.spidermiddleware.urllength.UrlLengthMiddleware': 800,
        'scrapy.contrib.spidermiddleware.depth.DepthMiddleware': 900,

"""
# from scrapy.contrib.spidermiddleware.referer import RefererMiddleware
# Enable or disable spider middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
SPIDER_MIDDLEWARES = {
   # 'step8_king.middlewares.SpiderMiddleware': 543,
}


"""
22. 下載中間件
    class DownMiddleware1(object):
        def process_request(self, request, spider):
            '''
            請求須要被下載時，通過全部下載器中間件的process_request調用
            :param request:
            :param spider:
            :return:
                None,繼續後續中間件去下載；
                Response對象，中止process_request的執行，開始執行process_response
                Request對象，中止中間件的執行，將Request從新調度器
                raise IgnoreRequest異常，中止process_request的執行，開始執行process_exception
            '''
            pass
    
    
    
        def process_response(self, request, response, spider):
            '''
            spider處理完成，返回時調用
            :param response:
            :param result:
            :param spider:
            :return:
                Response 對象：轉交給其餘中間件process_response
                Request 對象：中止中間件，request會被從新調度下載
                raise IgnoreRequest 異常：調用Request.errback
            '''
            print('response1')
            return response
    
        def process_exception(self, request, exception, spider):
            '''
            當下載處理器(download handler)或 process_request() (下載中間件)拋出異常
            :param response:
            :param exception:
            :param spider:
            :return:
                None：繼續交給後續中間件處理異常；
                Response對象：中止後續process_exception方法
                Request對象：中止中間件，request將會被從新調用下載
            '''
            return None

    
    默認下載中間件
    {
        'scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware': 100,
        'scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware': 300,
        'scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware': 350,
        'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400,
        'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 500,
        'scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware': 550,
        'scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware': 580,
        'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware': 590,
        'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600,
        'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware': 700,
        'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 750,
        'scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware': 830,
        'scrapy.contrib.downloadermiddleware.stats.DownloaderStats': 850,
        'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900,
    }

"""
# from scrapy.contrib.downloadermiddleware.httpauth import HttpAuthMiddleware
# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# DOWNLOADER_MIDDLEWARES = {
#    'step8_king.middlewares.DownMiddleware1': 100,
#    'step8_king.middlewares.DownMiddleware2': 500,
# }

View Code

十一、給抽屜點贊（帶上Cookie）

• DEPTH_LIMIT不能等於1，會返回空，不知道爲何

# -*- coding: utf-8 -*-
import scrapy
from scrapy.http import Request
from scrapy.selector import Selector
from scrapy.http.cookies import CookieJar


class ChoutiSpider(scrapy.Spider):
    name = 'chouti'
    allowed_domains = ['chouti.com']
    start_urls = ['https://dig.chouti.com/']
    cookie_dict = None


    def parse(self, response):
        """
        response.request表示當前的請求對象
        cookie_obj._cookies表示獲取的全部的cookies
        """
        cookie_obj = CookieJar()
        cookie_obj.extract_cookies(response,response.request)
        self.cookie_dict = cookie_obj._cookies
        # print(cookie_obj._cookies)
        # 帶上cookie+用戶名密碼登錄,給任務調度器發送請求
        yield Request(
            url='https://dig.chouti.com/login',
            method='POST',
            body="phone=86155&password=123&oneMonth=1",# 不支持字典格式
            cookies=self.cookie_dict,
            headers= { 'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',},
            callback=self.check_login,
        )

    def check_login(self,response):
        print(response.text)
        #{"result":{"code":"9999", "message":"", "data":{"complateReg":"0","destJid":"cdu_55306581825"}}}
        # 登錄成功後訪問首頁,訪問首頁不用帶cookies
        yield Request(
            url='https://dig.chouti.com/',
            callback=self.good,
        )

    def good(self,response):
        """
        給全部指定頁碼的新聞點贊
        注意：DEPTH_LIMIT不能等於1，會返回空，不知道爲何
        """
        # 先找到全部的新聞ID列表
        id_list = Selector(response=response).xpath('//div[@share-linkid]/@share-linkid').extract()
        print(id_list)
        for nid in id_list:
            url = 'https://dig.chouti.com/link/vote?linksId=%s'%nid
            yield Request(
                url=url,
                cookies=self.cookie_dict,
                method='POST',
                callback=self.show,
            )

        # 遞歸獲取當前頁的全部頁碼，執行點贊請求
        page_urls = Selector(response=response).xpath('//div[@id="dig_lcpage"]//a/@href').extract()
        for page in page_urls:
            url = 'http://dig.chouti.com%s' % page
            yield Request(url=url, callback=self.good)

    def show(self,response):
        """顯示點贊結果"""
        print(response.text)
        #{"result":{"code":"30010", "message":"你已經推薦過了", "data":""}}

View Code

12. 大文件下載

示例文件：猛擊下載

13.TinyScrapy

 

from twisted.web.client import getPage  # 用於建立socket對象(若是下載完成，自動從事件循環中移除）
from twisted.internet import reactor   # 事件循環（全部的socket都已經移除，纔會終止循環）
from twisted.internet import defer   # defer.Deferred特殊的socket對象，不會發請求，須要手動移除


def callback(arg):
    print('回來一個', arg)

@defer.inlineCallbacks
def task(url):
    # 1.利用getPage建立socket對象
    ret = getPage(bytes(url, encoding='utf8'))
    # 添加回調函數
    ret.addCallback(callback)
    # 2.將socket添加到事件循環
    yield ret

def stop(arg):
    print('已經所有如今完畢', arg)
    reactor.stop()

#3.開始事件循環
url_list = ['http://www.bing.com', 'http://www.baidu.com', ]
defer_list = []
for url in url_list:
    ret = task(url)
    defer_list.append(ret)

# 全部的socket請求完成後，終止循環
d = defer.DeferredList(defer_list)
d.addBoth(stop)

reactor.run()

twisted示例一

 

from twisted.internet import defer
from twisted.web.client import getPage
from twisted.internet import reactor
import threading


def _next_request():
    _next_request_from_scheduler()


def _next_request_from_scheduler():
    ret = getPage(bytes('http://www.chouti.com', encoding='utf8'))
    ret.addCallback(callback)
    ret.addCallback(lambda _: reactor.callLater(0, _next_request))


_closewait = None

@defer.inlineCallbacks
def engine_start():
    global _closewait
    _closewait = defer.Deferred()
    yield _closewait


@defer.inlineCallbacks
def task(url):
    reactor.callLater(0, _next_request)
    yield engine_start()


counter = 0
def callback(arg):
    global counter
    counter +=1
    if counter == 10:
        _closewait.callback(None)
    print('one', len(arg))


def stop(arg):
    print('all done', arg)
    reactor.stop()


if __name__ == '__main__':
    url = 'http://www.cnblogs.com'

    defer_list = []
    deferObj = task(url)
    defer_list.append(deferObj)

    v = defer.DeferredList(defer_list)
    v.addBoth(stop)
    reactor.run()

twisted示例二

 

from twisted.web.client import getPage, defer
from twisted.internet import reactor
import queue


class Response(object):
    def __init__(self, body, request):
        self.body = body
        self.request = request
        self.url = request.url

    @property
    def text(self):
        return self.body.decode('utf-8')


class Request(object):
    def __init__(self, url, callback=None):
        self.url = url
        self.callback = callback


class Scheduler(object):
    def __init__(self, engine):
        self.q = queue.Queue()
        self.engine = engine

    def enqueue_request(self, request):
        self.q.put(request)

    def next_request(self):
        try:
            req = self.q.get(block=False)
        except Exception as e:
            req = None

        return req

    def size(self):
        return self.q.qsize()


class ExecutionEngine(object):
    def __init__(self):
        self._closewait = None
        self.running = True
        self.start_requests = None
        self.scheduler = Scheduler(self)

        self.inprogress = set()

    def check_empty(self, response):
        if not self.running:
            self._closewait.callback('......')

    def _next_request(self):
        while self.start_requests:
            try:
                request = next(self.start_requests)
            except StopIteration:
                self.start_requests = None
            else:
                self.scheduler.enqueue_request(request)

        while len(self.inprogress) < 5 and self.scheduler.size() > 0:  # 最大併發數爲5

            request = self.scheduler.next_request()
            if not request:
                break

            self.inprogress.add(request)
            d = getPage(bytes(request.url, encoding='utf-8'))
            d.addBoth(self._handle_downloader_output, request)
            d.addBoth(lambda x, req: self.inprogress.remove(req), request)
            d.addBoth(lambda x: self._next_request())

        if len(self.inprogress) == 0 and self.scheduler.size() == 0:
            self._closewait.callback(None)

    def _handle_downloader_output(self, body, request):
        """
        獲取內容，執行回調函數，而且把回調函數中的返回值獲取，並添加到隊列中
        :param response: 
        :param request: 
        :return: 
        """
        import types

        response = Response(body, request)
        func = request.callback or self.spider.parse
        gen = func(response)
        if isinstance(gen, types.GeneratorType):
            for req in gen:
                self.scheduler.enqueue_request(req)

    @defer.inlineCallbacks
    def start(self):
        self._closewait = defer.Deferred()
        yield self._closewait

    def open_spider(self, spider, start_requests):
        self.start_requests = start_requests
        self.spider = spider
        reactor.callLater(0, self._next_request)


class Crawler(object):
    def __init__(self, spidercls):
        self.spidercls = spidercls

        self.spider = None
        self.engine = None

    @defer.inlineCallbacks
    def crawl(self):
        self.engine = ExecutionEngine()
        self.spider = self.spidercls()
        start_requests = iter(self.spider.start_requests())
        start_requests = iter(start_requests)
        self.engine.open_spider(self.spider, start_requests)
        yield self.engine.start()


class CrawlerProcess(object):
    def __init__(self):
        self._active = set()
        self.crawlers = set()

    def crawl(self, spidercls, *args, **kwargs):
        crawler = Crawler(spidercls)

        self.crawlers.add(crawler)
        d = crawler.crawl(*args, **kwargs)
        self._active.add(d)
        return d

    def start(self):
        dl = defer.DeferredList(self._active)
        dl.addBoth(self._stop_reactor)
        reactor.run()

    def _stop_reactor(self, _=None):
        reactor.stop()


class Spider(object):
    def start_requests(self):
        for url in self.start_urls:
            yield Request(url)


class ChoutiSpider(Spider):
    name = "chouti"
    start_urls = [
        'http://dig.chouti.com/',
    ]

    def parse(self, response):
        print(response.text)


class CnblogsSpider(Spider):
    name = "cnblogs"
    start_urls = [
        'http://www.cnblogs.com/',
    ]

    def parse(self, response):
        print(response.text)


if __name__ == '__main__':

    spider_cls_list = [ChoutiSpider, CnblogsSpider]

    crawler_process = CrawlerProcess()
    for spider_cls in spider_cls_list:
        crawler_process.crawl(spider_cls)

    crawler_process.start()

模擬scrapy框架

 

import types
from twisted.internet import defer
from twisted.web.client import getPage
from twisted.internet import reactor



class Request(object):
    def __init__(self, url, callback):
        self.url = url
        self.callback = callback
        self.priority = 0


class HttpResponse(object):
    def __init__(self, content, request):
        self.content = content
        self.request = request


class ChouTiSpider(object):

    def start_requests(self):
        url_list = ['http://www.cnblogs.com/', 'http://www.bing.com']
        for url in url_list:
            yield Request(url=url, callback=self.parse)

    def parse(self, response):
        print(response.request.url)
        # yield Request(url="http://www.baidu.com", callback=self.parse)




from queue import Queue
Q = Queue()


class CallLaterOnce(object):
    def __init__(self, func, *a, **kw):
        self._func = func
        self._a = a
        self._kw = kw
        self._call = None

    def schedule(self, delay=0):
        if self._call is None:
            self._call = reactor.callLater(delay, self)

    def cancel(self):
        if self._call:
            self._call.cancel()

    def __call__(self):
        self._call = None
        return self._func(*self._a, **self._kw)


class Engine(object):
    def __init__(self):
        self.nextcall = None
        self.crawlling = []
        self.max = 5
        self._closewait = None

    def get_response(self,content, request):
        response = HttpResponse(content, request)
        gen = request.callback(response)
        if isinstance(gen, types.GeneratorType):
            for req in gen:
                req.priority = request.priority + 1
                Q.put(req)


    def rm_crawlling(self,response,d):
        self.crawlling.remove(d)

    def _next_request(self,spider):
        if Q.qsize() == 0 and len(self.crawlling) == 0:
            self._closewait.callback(None)

        if len(self.crawlling) >= 5:
            return
        while len(self.crawlling) < 5:
            try:
                req = Q.get(block=False)
            except Exception as e:
                req = None
            if not req:
                return
            d = getPage(req.url.encode('utf-8'))
            self.crawlling.append(d)
            d.addCallback(self.get_response, req)
            d.addCallback(self.rm_crawlling,d)
            d.addCallback(lambda _: self.nextcall.schedule())


    @defer.inlineCallbacks
    def crawl(self):
        spider = ChouTiSpider()
        start_requests = iter(spider.start_requests())
        flag = True
        while flag:
            try:
                req = next(start_requests)
                Q.put(req)
            except StopIteration as e:
                flag = False

        self.nextcall = CallLaterOnce(self._next_request,spider)
        self.nextcall.schedule()

        self._closewait = defer.Deferred()
        yield self._closewait

    @defer.inlineCallbacks
    def pp(self):
        yield self.crawl()

_active = set()
obj = Engine()
d = obj.crawl()
_active.add(d)

li = defer.DeferredList(_active)
li.addBoth(lambda _,*a,**kw: reactor.stop())

reactor.run()

參考版

 點擊下載TinyScrapy

1四、源碼閱讀

 

 更多文檔參見：http://scrapy-chs.readthedocs.io/zh_CN/latest/index.html

文章轉載自https://www.cnblogs.com/wupeiqi/articles/6229292.html