爬蟲-scrapy的中間件

時間  2019-12-09
標籤 爬蟲 scrapy 中間件 欄目 網絡爬蟲 简体版
原文   原文鏈接

scrapy的中間件

  • 下載中間件php

    • 做用:
      • 處於引擎和下載器之間,所以該中間件能夠批量攔截整個工程中發起全部的請求和響應
    • 攔截請求可進行的操做
      • 進行代理IP
        • request.meta['proxy'] = 'http://ip:port'
      • 進行UA假裝
        • request.headers['User-Agent'] = 'xxxx'
    • 攔截響應可進行的操做
      • 篡改響應數據(通常不用)
      • 更換響應對象
    • 在scrapy中使用selenium
      • 爬蟲類中定義一個bro的屬性(selenium實例化的一個瀏覽器對象)
      • 爬蟲類中重寫父類的一個方法closed(self,spider),在該方法中關閉瀏覽器對象
      • 在中間件中的process_response中經過spider參數獲取爬蟲類中的bro屬性
      • 在中間件中編寫相關的瀏覽器自動化的操做獲取頁面源碼數據
      • 將頁面源碼數據做爲新的響應對象的響應數據
      • 將新的響應對象返回

  • 基於crawlSpier的全站數據爬取html

    • crawlSpier和Spider之間的關聯?
      • crawlSpier是SPider的一個子類
    • 建立一個基於CrawlSpider的爬蟲文件
      • scrapy genspider -t crawl xxx www.xxx.com

  • 一個簡單的IP端口測試python

    • 項目運行文件設置
    import scrapy

class MidSpider(scrapy.Spider):
    name = 'mid'
    # allowed_domains = ['www.xxx.com']
    start_urls = ['http://www.baidu.com/s?wd=ip']

    def parse(self, response):
        page_text = response.text
        with open('./ip.html','w',encoding='utf-8') as fp:
            fp.write(page_text)
    • item中不須要任何操做
    • pipelines文件代碼
    class MiddleproPipeline(object):
    def process_item(self, item, spider):
        return item
    • middlewares文件中代碼
    rom scrapy import signals
import random
#可被選用的代理IP
PROXY_http = [
    '153.180.102.104:80',
    '195.208.131.189:56055',
]
PROXY_https = [
    '120.83.49.90:9000',
    '95.189.112.214:35508',
]
#UA池的彙總
user_agent_list = [
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "
        "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
        "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "
        "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "
        "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "
        "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "
        "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "
        "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
        "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "
        "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
        "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
class MiddleproDownloaderMiddleware(object):

    #攔截正常請求
    def process_request(self, request, spider):
        #UA假裝:UA池,每次都使用隨機的UA
        request.headers['User-Agent'] = random.choice(user_agent_list)

        #使用假裝的代理IP訪問
        request.meta['proxy'] = 'http://39.137.77.66:8080'
    #攔截全部的響應
    def process_response(self, request, response, spider):
        # Called with the response returned from the downloader.

        # Must either;
        # - return a Response object
        # - return a Request object
        # - or raise IgnoreRequest
        return response
    #攔截髮生異常的請求
    def process_exception(self, request, exception, spider):
        #代理ip的設定
        if request.url.split(':')[0] == 'http':
            request.meta['proxy'] = random.choice(PROXY_http)
        else:
            request.meta['proxy'] = random.choice(PROXY_https)

        return request  #將修正以後的請求對象進行從新發送
    • 此時settings文件中須要將中間件打開
    DOWNLOADER_MIDDLEWARES = {
   'middlePro.middlewares.MiddleproDownloaderMiddleware': 543,
}

  • 爬取網易新聞的內容web

    • 運行文件內容
    import scrapy

from selenium import webdriver
class WangyiSpider(scrapy.Spider):
    name = 'wangyi'
    # allowed_domains = ['www.xxx.com']
    start_urls = ['https://news.163.com/world/']

    #實例化一個瀏覽器對象,須要添加chromedriver插件
    bro = webdriver.Chrome(executable_path=r'D:\爬蟲\谷歌訪問助手\chromedriver.exe')
    def parse(self, response):
        div_list = response.xpath('/html/body/div/div[3]/div[4]/div[1]/div/div/ul/li/div/div')
        for div in div_list:
        #xpath表達式須要跳過body標籤,遇到body標籤常使用//跳過
            title = div.xpath('.//div[@class="news_title"]//a/text()').extract_first()
            detail_url = div.xpath('.//div[@class="news_title"]//a/@href').extract_first()
            yield scrapy.Request(detail_url,self.parse_detail)
            print(title,detail_url)

    def parse_detail(self,response):
        content = response.xpath('//*[@id="endText"]//text()').extract()
        #使用join將列表數據組成字符串
        content = ''.join(content)
        print(content)
    def closed(self,spider):
        self.bro.quit()
    • items文件不須要操做
    • pipeline文件的操做
    class WangyiproPipeline(object):
    def process_item(self, item, spider):
        return item
    • middleware文件中的操做
    from scrapy import signals

from scrapy.http import HtmlResponse
from time import sleep



class WangyiproDownloaderMiddleware(object):

    def process_request(self, request, spider):
        # Called for each request that goes through the downloader
        # middleware.

        # Must either:
        # - return None: continue processing this request
        # - or return a Response object
        # - or return a Request object
        # - or raise IgnoreRequest: process_exception() methods of
        #   installed downloader middleware will be called
        return None
    #spider表示的就是爬蟲類實例化的對象
    def process_response(self, request, response, spider):
        #將不符合需求的響應對象修改爲符合需求的
        #body:響應數據
        #如何獲取爬蟲類中生成的瀏覽器對象呢?
        if request.url == 'https://news.163.com/world/':
            bro = spider.bro
            bro.get('https://news.163.com/world/')
            sleep(2)
  #這裏使用了js自動下滑處理動態加載數據          bro.excute_script('window.scrollTo(0,document.body.scrollHeight)')
            sleep(1)
            bro.excute_script('window.scrollTo(0,document.body.scrollHeight)')
            sleep(1)
            bro.excute_script('window.scrollTo(0,document.body.scrollHeight)')
            sleep(1)
            page_text = bro.page_source

            new_response = HtmlResponse(url=bro.current_url,body=page_text,encoding='utf-8',request=request)

            return new_response
        else:
            return response


    def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        pass
    • 最後在settings文件中加上中間件的配置
    DOWNLOADER_MIDDLEWARES = {
   'wangyiPro.middlewares.WangyiproDownloaderMiddleware': 543,
}

  • crawlspider全站數據爬取案例chrome

    • 執行文件內容
    import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from sunShinePro.items import SunshineproItem,sunConetent

# http://wz.sun0769.com/index.php/question/questionType?type=4&page=30
class SunSpider(CrawlSpider):
    name = 'sun'
    # allowed_domains = ['www.xxx.com']

    start_urls = ['http://wz.sun0769.com/index.php/question/questionType?type=4&page=']
    #連接提取器
        #做用:根據 指定的規則(allow:正則) 提取頁面源碼中指定的鏈接,進行對頁面中的頁碼鏈接提取
    link = LinkExtractor(allow=r'type=4&page=\d+')
    link_detail = LinkExtractor(allow='question/\d+/\d+\.shtml')
    rules = (
        #規則解析器:將鏈接提取器提取到的鏈接對應的頁面源碼數據 根據指定規則(callback) 進行數據解析
        Rule(link, callback='parse_item', follow=False),
        Rule(link_detail, callback='parse_detail'),
    )
    def parse_detail(self,response):
        content = response.xpath('/html/body/div[9]/table[2]//tr[1]/td/div[2]//text()').extract()
        content = ''.join(content)

        item = sunConetent()
        item['content'] = content

        yield item

    def parse_item(self, response):
        #注意:若是xpath定位的標籤中存在tbody,則須要跳過tbody
        tr_list = response.xpath('//*[@id="morelist"]/div/table[2]//tr/td/table//tr')
        for tr in tr_list:
            title = tr.xpath('./td[2]/a[2]/text()').extract_first()
            status = tr.xpath('./td[3]/span/text()').extract_first()
            item = SunshineproItem()
            item['title'] = title
            item['status'] = status

            yield item
    • items文件內容
    import scrapy


class SunshineproItem(scrapy.Item):
    # define the fields for your item here like:
    title = scrapy.Field()
    status = scrapy.Field()
    # pass
class sunConetent(scrapy.Item):
    content = scrapy.Field()
    • pipelines文件內容
    class SunshineproPipeline(object):
    def process_item(self, item, spider):
        if item.__class__.__name__ == 'SunshineproItem':
            print(item['title'], item['status'])

        else:
            print(item['content'])
        return item
    • 此案例未使用中間件配置,所以settings文件中只作之間的配置,而且middleware文件不須要變更

  • 爬取網易新聞中的5個模塊內的全部內容瀏覽器

    • 建立一個工程:scrapy startproject wangyiPro
    • cd wangyiPro

    • 建立一個爬蟲文件:scrapy genspider wangyi www.xxx.com(指定的url,必需要先CD到項目中,保證爬蟲文件在spiders目錄中)app

    • wangyi文件中內容dom

    import scrapy
from wangyiPro.items import WangyiproItem
from selenium import webdriver
class WangyiSpider(scrapy.Spider):
    name = 'wangyi'
    # allowed_domains = ['www.xxx.com']
    # 根據起始url解析出五大板塊對應的詳情頁的url
    start_urls = ['https://news.163.com/']
    model_detail_urls = []#五個板塊詳情頁的url

    bro = webdriver.Chrome(executable_path=r'D:\爬蟲\谷歌訪問助手\chromedriver.exe')
    def parse(self, response):
        #解析出五大板塊對應的詳情頁的url
        model_list = []  #存儲的就是五個板塊對應的li標籤
        indexs = [3,4,6,7,8]
        li_list = response.xpath('//*[@id="index2016_wrap"]/div[1]/div[2]/div[2]/div[2]/div[2]/div/ul/li')

        for index in indexs:
            model_list.append(li_list[index])

        for li in model_list:
            #五個板塊對應詳情頁的url
            detail_url = li.xpath('./a/@href').extract_first()
            self.model_detail_urls.append(detail_url)
            yield scrapy.Request(detail_url,callback=self.parse_detail)
    #解析:新聞標題和新聞詳情頁的url
    def parse_detail(self,response):
        div_list = response.xpath('/html/body/div/div[3]/div[4]/div[1]/div/div/ul/li/div/div')
        for div in div_list:
            title = div.xpath('.//div[@class="news_title"]/h3/a/text()').extract_first()
            new_detail_url = div.xpath('.//div[@class="news_title"]/h3/a/@href').extract_first()

            item = WangyiproItem()
            item['title'] = title
            #須要經過請求傳參將item傳遞給news_parse
            yield scrapy.Request(new_detail_url,callback=self.news_parse,meta={'item':item})
    #用來解析新聞內容
    def news_parse(self,response):
        content = response.xpath('//*[@id="endText"]//text()').extract()
        content = ''.join(content)

        item = response.meta['item']
        item['content'] = content

        yield item
    def closed(self,spider):
        self.bro.quit()
    • items中定義字段
    import scrapy


class WangyiproItem(scrapy.Item):
    # define the fields for your item here like:
    title = scrapy.Field()
    content = scrapy.Field()
    pass
    • pipelines文件內容
    class WangyiproPipeline(object):
    fp = None
    def open_spider(self,spider):
        self.fp = open('./wangyi.txt','w',encoding='utf-8')
    def process_item(self, item, spider):
        self.fp.write(item['title']+':'+item['content']+'\n')
        return item
    def close_spider(self,spider):
        self.fp.close()
    • middlewares文件內容
    from scrapy.http import HtmlResponse
from time import sleep


class WangyiproDownloaderMiddleware(object):


    def process_request(self, request, spider):

        return None

    # 能夠攔截全部的響應對象:該方法攔截5個指定的響應對象且進行替換操做,其餘的響應對象不作處理
    def process_response(self, request, response, spider):
        #定位到5個指定的響應對象
        if request.url in spider.model_detail_urls:
            print(request.url)
            bro = spider.bro
            bro.get(request.url)
            sleep(2)
            bro.execute_script('window.scrollTo(0,document.body.scrollHeight)')
            sleep(2)
            bro.execute_script('window.scrollTo(0,document.body.scrollHeight)')
            sleep(2)
            bro.execute_script('window.scrollTo(0,document.body.scrollHeight)')
            sleep(2)
            page_text = bro.page_source
            new_response = HtmlResponse(url=request.url,body=page_text,encoding='utf-8',request=request)

            return new_response
        else:
            return response #將原始的響應對象進行返回

    def process_exception(self, request, exception, spider):

        pass
    • 最後在settings文件中作以下配置
    #不聽從爬蟲Robots協議
ROBOTSTXT_OBEY = False
#UA假裝
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
#開啓下載中間件
DOWNLOADER_MIDDLEWARES = {
   'wangyiPro.middlewares.WangyiproDownloaderMiddleware': 543,
}
#開啓管道
ITEM_PIPELINES = {
   'wangyiPro.pipelines.WangyiproPipeline': 300,
}
#顯示打印錯誤信息
LOG_LEVEL = 'ERROR'
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程