Scrapy爬取豆瓣小組圖片

時間 2019-11-10

標籤 scrapy 豆瓣小組圖片欄目 Python 简体版

原文原文鏈接

狗年開工毫無工做心情，胡思亂想後決定爬取豆瓣上的一下信息打發時間，畢竟以前基本沒接觸過爬蟲，仍是挺感興趣的。python

Scrapy簡介

首先簡單介紹一下Scrapy爬蟲框架，主要是架構方面，這方面能快速理解scrapy是如何工做的。web

Scrapy的數據流由執行引擎（Engine）控制，其基本過程以下：數據庫

引擎從Spider中獲取到初始Requests。
引擎將該Requests放入調度器，並請求下一個要爬取的Requests。
調度器返回下一個要爬取的Requests給引擎
引擎將Requests經過下載器中間件轉發給下載器(Downloader)。
一旦頁面下載完畢，下載器生成一個該頁面的Response，並將其經過下載中間件(返回(response)方向)發送給引擎。
引擎從下載器中接收到Response並經過Spider中間件(輸入方向)發送給Spider處理。
Spider處理Response並返回爬取到的Item及(跟進的)新的Request給引擎。
引擎將(Spider返回的)爬取到的Item交給ItemPipeline處理，將(Spider返回的)Request交給調度器，並請求下一個Requests（若是存在的話）。
(從第一步)重複直到調度器中沒有更多地Request。

使用Scrapy建立的項目架構以下json

其中：api

spider 文件夾編寫本身的爬蟲；
settings.py 配置爬蟲的默認信息，功能開關，中間件執行順序等；
middlewares.py 中間件，主要是對功能的拓展，添加自定義功能，好比user-agent和proxy
item.py 定義抓取處理的字段
piplines.py 管道文件，處理item

爬取豆瓣小組

豆瓣小組的帖子主要核心內容是圖片，所以要按不一樣的帖子分類下載。cookie

settings.py

設置了user-agent，指定了中間件和piplines架構

BOT_NAME = 'douban'

SPIDER_MODULES = ['douban.spiders']
NEWSPIDER_MODULE = 'douban.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'douban (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

MY_USER_AGENT = [
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",
    "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
    "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
    "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",
    "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",
    "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
    "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
    "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
    "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
    "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",
    "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",
    "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER",
    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)",
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 LBBROWSER",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)",
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)",
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
    "Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5",
    "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13pre",
    "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",
    "Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
    ]
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'douban.middlewares.MyUserAgentMiddleware': 400,
}
COOKIES_ENABLES = True
DOWNLOAD_DELAY=1
ITEM_PIPELINES = {
   'douban.pipelines.DoubanPipeline': 1,
}

item.py

定義字段，包括做者，帖子名稱，做者主頁地址，圖片地址app

class DoubanItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title=scrapy.Field()
    author=scrapy.Field()
    author_homepage=scrapy.Field()
    img_url=scrapy.Field()
    pass

middlewares.py

設置user-agent框架

class MyUserAgentMiddleware(UserAgentMiddleware):
    '''
    設置User-Agent
    '''

    def __init__(self, user_agent, ip):
        self.user_agent = user_agent
        self.ip=ip

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            user_agent=crawler.settings.get('MY_USER_AGENT')
            , ip=crawler.settings.get('PROXIES')
        )

    def process_request(self, request, spider):
        agent = random.choice(self.user_agent)
        request.headers['User-Agent'] = agent

spiders/douban_spider.py

爬蟲的處理代碼，先登陸而後爬取，若是有驗證碼，下載圖片而後輸入驗證碼dom

import urllib

import scrapy
from scrapy import Request, FormRequest

from douban.items import DoubanItem
import json

class DoubanSpider(scrapy.Spider):
    name = 'douban'
    allowed_domains = ['douban.com']

    start_urls = []

    def start_requests(self):
        yield Request("https://www.douban.com/login", callback=self.parse, meta={"cookiejar":1})

    def parse(self, response):
        captcha = response.xpath('//img[@id="captcha_image"]/@src').extract()
        if len(captcha)>0:
            print("此時有驗證碼")
            localpath = "E:/spider/douban/captchar.jpg"
            urllib.request.urlretrieve(captcha[0],filename=localpath)
            print("請查看本地驗證碼圖片並輸入驗證碼")
            captcha_value=input()

            data = {
                "form_email": "*******@126.com",
                "form_password": "*******",
                "captcha-solution": str(captcha_value),
                "redir": "https://www.douban.com/group/haixiuzu/discussion?start=0"  # 登陸後要返回的頁面
            }
        else:
            print("此時沒有驗證碼")
            data = {
                "form_email": "nofree1990@126.com",
                "form_password": "8296926",
                # "redir": "https://www.douban.com/group/haixiuzu/discussion?start=0"  # 登陸後要返回的頁面
            }
        print("登錄中...")
        yield FormRequest.from_response(response,meta={"cookiejar": response.meta["cookiejar"]}, formdata=data, callback=self.parse_redirect)

    def parse_redirect(self, response):
        print("已登陸豆瓣")
        title = response.xpath('//title//text()').extract()

        baseurl='https://www.douban.com/group/haixiuzu/discussion?start='
        for i in range(0, 625, 25):
            pageUrl=baseurl+str(i)
            yield Request(url=pageUrl, callback=self.parse_process,dont_filter = True)

    def parse_process(self, response):
        title = response.xpath('//title//text()').extract()
        items = response.xpath('//td//a/@href').extract()
        for item in items:
            if 'topic' in item:
                url=item
                yield Request(url=item,callback=self.parse_img)

    def parse_img(self,response):
        img = DoubanItem()
        title=response.xpath('//title//text()').extract()
        img['title']=title
        author=response.xpath('//div[@class="topic-doc"]//h3//a//text()').extract()
        img['author']=author
        author_homepage = response.xpath('//div[@class="topic-doc"]//h3//a/@href').extract()
        img['author_homepage'] = author_homepage
        img_url = response.xpath('//div[@class="image-wrapper"]//img/@src').extract()
        img['img_url'] = img_url
        yield img

piplines.py

在此保存帖子信息，沒有使用自帶的保存圖片的類主要緣由是不夠靈活。

class DoubanPipeline(object):
    def process_item(self, item, spider):
        author=item["author"][0]
        title=item["title"][0].replace('\n','').strip()
        author_homepage=item["author_homepage"][0]
        #路徑
        dir="E:/spider/douban/img/"
        if not os.path.exists(dir):
            os.mkdir(dir)
        author_dir=dir+title
        if not os.path.exists(author_dir):
            os.mkdir(author_dir)
        #用戶信息txt
        info=open(author_dir+"/用戶信息.txt", "w")
        info.write(author+'\n'+author_homepage)
        info.close()
        #保存圖片
        count=1
        for url in item["img_url"]:
            path=author_dir+"/"+str(count)+".jpg"
            urllib.request.urlretrieve(url, filename=path)
            count += 1
        return item

爬蟲結果

遇到的問題

主要問題就是爬取太頻繁而被禁止，登陸豆瓣也是想減小被禁止機率，可是發現沒什麼用。網上有不少解決方案，仍是要僞造一些user-agent，使用proxy代理。也爬取過一些proxy存到數據庫中，可是proxy比較慢，遂放棄。

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。