python scrapy簡單爬蟲記錄(實現簡單爬取知乎)

以前寫了個scrapy的學習記錄，只是簡單的介紹了下scrapy的一些內容，並無實際的例子，如今開始記錄例子

使用的環境是python2.7， scrapy1.2.0

首先建立項目

在要創建項目的目錄下執行命令scrapy startproject tutorial

scrapy會幫你創建好項目，接着建立蜘蛛scrapy genspider　　zhuhuSpider zhihu.com

當前的文件結構是

--tutorial

　　--spiders

　　　　--__init__.py

　　　　--zhihuSpider.py

　　--__init__.py

　　--items.py

　　--pipelines.py

　　--settings.py

主要用到的文件有三個zhihuspider.py,items.py,settings.py

zhihuspider是用來寫爬行的蜘蛛的，items用來保存爬取的數據，settings修改配置

# -*- coding: utf-8 -*-
from scrapy.spiders import Spider
from scrapy.selector import Selector
from tutorial.items import ZhihuItem

#蜘蛛的主體
class ZhihuSpider(Spider):
    name = "zhihuSpider"
    allowed_domains = ['zhihu.com']
    start_urls = []

　　#獲取要爬取的url，我把關鍵詞放在zhihu.txt中
    def start_requests(self):
        url_head = 'https://www.zhihu.com/search?type=content&q='
        with open('zhihu.txt', 'r') as f:
            datas = f.readlines()
        for data in datas:
            url = url_head+data
            print url
            self.start_urls.append(url)

        for url in self.start_urls:
            yield self.make_requests_from_url(url)

　　#對返回的response進行解析，該步驟能夠配合瀏覽器進行xpath的查找
    def parse(self, response):
　　　　#刪除全部的<em>
        response = response.replace(body=response.body.replace('<em>', ''))
        hxs = Selector(response)

        contents = hxs.xpath('//*[@class="zu-main-content"]//*[contains(@class, "list")]')
        item = ZhihuItem()
        for content in contents:
            item['search_title'] = content.xpath('//*[@class="title"]/a/text()').extract()
            item['search_title_link'] = content.xpath('//*[@class="title"]/a/@href').extract()
            item['search_answer'] = content.xpath('//*[@class="content"]//*[contains(@class, "entry-content")]//*[contains(@class, "summary")]/text()').extract()
            item['search_answer_link'] = content.xpath('//*[@class="content"]//*[contains(@class, "entry-content")]//*[contains(@class, "summary")]/a/@href').extract()
            item['search_answer_writer'] = content.xpath('//*[@class="content"]//*[contains(@class, "entry-meta")]//a[contains(@class, "author")]/text()').extract()
            print item
            yield item

from scrapy.item import Item, Field

class ZhihuItem(Item):
    search_title = Field()
    search_title_link = Field()
    search_answer = Field()
    search_answer_link = Field()
    search_answer_writer = Field()

該items文件存放數據

由於知乎對爬蟲有限制，因此須要加入反爬蟲機制

基本上有4種，添加useragent，添加代理，禁用cookie以及爬取時間限制

在當前項目下添加python包middlewares，和setttings在同一個目錄下，下面只有一個__init__.py文件，新建一個RandomUserAgent.py文件

#coding:utf-8
import random

class RandomUserAgent(object):

    def __init__(self, agents):
        self.agents = agents

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist('USER_AGENTS'))

    def process_request(self, request, spider):
        request.headers.setdefault('User-Agent', random.choice(self.agents))

簡單的加入user-agent

最後在settings中設置

DOWNLOADER_MIDDLEWARES = {
   # 'tutorial.middlewares.MyCustomDownloaderMiddleware': 543,
    'tutorial.middlewares.RandomUserAgent.RandomUserAgent': 1,
}
USER_AGENTS = [

    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",

    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",

    "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",

    "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",

    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",

    "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",

    "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",

    "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",

    "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",

    "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",

    "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",

    "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",

    "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",

    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",

    "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER",

    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)",

    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 LBBROWSER",

    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",

    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)",

    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",

    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)",

    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",

    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",

    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",

    "Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5",

    "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13pre",

    "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0",

    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",

    "Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10"

]
COOKIES_ENABLED=False
DOWNLOAD_DELAY=3

整個項目完成了，scrapy crawl zhihuSpider -o zhihu.json導出數據。就是這麼簡單 

若是想要不喜歡用-o來導出數據，能夠讓scrapy本身保存數據，編輯pipelines

import json
import codecs


class xxxPipeline(object):
    def __init__(self):
        self.file = codecs.open('xxx.csv', 'w', encoding='utf-8')

    def process_item(self, item, spider):
        datas = json.dumps(dict(item), ensure_ascii=False) + '\n'
        self.file.write(datas)
        return item

    def spider_closed(self, spider):
        self.file.close()

而後在setting中啓用

ITEM_PIPELINES = {
   'tutorial.pipelines.xxxPipeline': 300, 
}

把對應的pipeline修改爲本身的，最後那個數字表示啓動順序，越低順序越快。

最後說一句，爬取數據時要注意對爬取時間的設置，別給服務器形成負擔

項目已經提交到github上地址是https://github.com/lin344902118/doubanSpider.git

裏面還有個對豆瓣的爬蟲