使用scrapy框架爬取豆瓣影評

最近幾天在看平凡歲月 感受挺不錯的一部生活劇，想看看豆瓣

對該據評論，使用scrapy爬取，基於python2.7進行實現

1. 建立scrapy項目

scrapy startproject doubanmovie

2.建立爬蟲

scrapy genspider -t crawl dbm "movie.douban.com"

須要注意的是 爬蟲名稱不能和項目名稱重複

3. 編寫item

4.編寫爬蟲（spiders-->dbm.py）

# -*- coding: utf-8 -*-
import scrapy

from scrapy.linkextractors import LinkExtractor

from scrapy.spiders import CrawlSpider, Rule

from doubanmovie.items import DoubanmovieItem

class DbmSpider(CrawlSpider):
    name = 'dbm'
    allowed_domains = ['movie.douban.com']
    start_urls = ['https://movie.douban.com/subject/26795042/reviews?start=0']

    rules = (
        Rule(LinkExtractor(allow=r'start=\d+'), follow=True),
        Rule(LinkExtractor(allow=r'https://movie.douban.com/review/\d+/'), callback='parse_item', follow=False),
    )

    def parse_item(self, response):
        item = DoubanmovieItem()
        item['username'] = response.xpath("//header[@class='main-hd']/a/span/text()").extract()[0]
        title = response.xpath("//span[@property='v:summary']/text()").extract()[0]
        if len(title):
            item['title'] = title
        else:
            item['title'] = 'NULL'
        content = response.xpath("//div[@class='review-content clearfix']/p/text()").extract()
        if len(content):
            item['content'] = ''.join(content).strip()
        else:
            content = response.xpath("//div[@class='review-content clearfix']/text()").extract()
            if len(content):
                item['content'] = ''.join(content).strip()
            else:
                item['content'] = 'NULL'
        item['url'] = response.url
        yield item

5.編寫管道文件(該案例使用json保存) doubanmovie->pipelines.py

import json
class YouyuanPipeline(object):
    def __init__(self):
        self.filename = open('youyuan.json','w')

    def process_item(self, item, spider):
        content = json.dumps(dict(item),ensure_ascii=False) + ',\n'
        self.filename.write(content.encode('utf-8'))
        return item

    def close_spider(self,spider):
        self.filename.close()

6.修改setting文件

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.54 Safari/536.5'  
DOWNLOAD_DELAY = 3
ITEM_PIPELINES = {
   'doubanmovie.pipelines.DoubanmoviePipeline': 300,
}

7- 執行爬蟲

scrapy crawl dbm