scrapy學習筆記一

時間 2019-12-14

標籤 scrapy 學習筆記欄目 Python 简体版

原文原文鏈接

scrapy學習筆記

下面以爬取1919網站爲例子，完成對一整個網站數據爬取的scrapy項目建立。html

建立一個scrapy文件

在任意目錄下輸入命令mysql

scrapy startproject OneNine (文件名)

將會獲得以下目錄的文件sql

OneNine/
    scrapy.cfg            # 部署配置文件

    OneNine/           # Python模塊，你全部的代碼都放這裏面
        __init__.py

        items.py          # Item定義文件

        pipelines.py      # pipelines定義文件

        settings.py       # 配置文件

        spiders/          # 全部爬蟲spider都放這個文件夾下面
            __init__.py
            ...

接着建立一個spider文件用來編寫爬取規則數據庫

cd OneNine
scrape genspider onenine onenine.com

此時在spiders文件夾下就會生成一個onenine.py文件，咱們將在這個文件中編寫爬蟲規則json

定義Item

在items.py文件中須要編寫咱們要爬取的字段內容。session

import scrapy

class OnenineItem(scrapy.Item):
    url = scrapy.Field()
    good_name = scrapy.Field()
    actual_price = scrapy.Field()
    details = scrapy.Field()
    year = scrapy.Field()
    month = scrapy.Field()
    plateform = scrapy.Field()
    cat_lv_one = scrapy.Field()
    cat_lv_two = scrapy.Field()
    shop_id = scrapy.Field()
    shop_name = scrapy.Field()
    shop_area = scrapy.Field()
    shop_province = scrapy.Field()
    shop_city = scrapy.Field()
    good_id = scrapy.Field()
    brand = scrapy.Field()
    size = scrapy.Field()
    percent = scrapy.Field()
    country = scrapy.Field()
    area = scrapy.Field()
    type = scrapy.Field()
    grape_type = scrapy.Field()
    num = scrapy.Field()
    name_price = scrapy.Field()
    bottle_price = scrapy.Field()
    comments = scrapy.Field()
    accumulate_sales = scrapy.Field()
    month_sales = scrapy.Field()
    month_bottle_sales = scrapy.Field()
    month_sale_amounts = scrapy.Field()

scrapy.Field的屬性的字段能夠直接在後期直接生成你要的文件格式。app

spider文件

在OneNine/spiders文件夾下的onenine.py文件中咱們編寫了對於網站爬取規則的編寫。dom

在編寫爬取規則前，咱們要先繼承一個scrapy.Spider類，並定義一些屬性：scrapy

name：Spider名稱，必須惟一
allowed_domains：定義網頁的篩選規則
start_urls：起始爬取的網址

  1 # -*- coding: utf-8 -*-
  2 import scrapy
  3 from ..items import OnenineItem
  4 from scrapy.linkextractors import LinkExtractor
  5 from scrapy.spiders import CrawlSpider
  6 import requests,re
  7 
  8 class OnenineSpider(scrapy.Spider):
  9     name = 'onenine'
 10     allowed_domains = ['www.1919.cn']
 11     start_urls = ['https://www.1919.cn/search.html?sort=DEFAULT_SORT&page='+str(x) +'&size=16&kw=%E7%99%BD%E9%85%92'
 12                    for x in range(0,27)]    #使用列表生成式完成翻頁處理
 13 
 14     def parse(self, response):
 15 
 16         result = response.xpath('//div[@class="ml-info ml-rpb12"]')
 17         for i in result:
 18             item = OnenineItem()
 19             item['good_name'] = i.xpath('p[@class="ml-pdtname"]/a/text()').extract()[0]  # 商品名
 20             item['name_price'] =  i.xpath('p[@class="ml-pdtpri"]/span[@class="ml-pri"]/text()').extract()[0].replace('.','')# 商品價格
 21             item['url'] = i.xpath('p[@class="ml-pdtname"]/a/@href').extract()[0]   # 商品url
 22             url = response.urljoin(item['url'])
 23             yield scrapy.Request(url,meta={'item':item},callback=self.good_detail)
 24 
 25     def good_detail(self,response):
 26         # item = OnenineItem()
 27         item = response.meta['item']
 28         result = response.xpath('//div[@class="intro-cont com-size"]')
 29         li_list = []
 30         for i in result:
 31             result2 = i.xpath('span/text()').extract()
 32             li_list.append(''.join(result2))
 33 
 34 
 35 
 36         item['year'] = 2018
 37         item['month'] = 2
 38         item['plateform'] = '1919'
 39         item['cat_lv_one'] = '酒水'
 40         item['cat_lv_two'] = '白酒'
 41 
 42         shop_url = response.xpath('//a[@class="dt-mainRedColor"]/@href').extract()[0]
 43         panter = re.compile('v/(.*?)\.', re.S)
 44         item['shop_id'] = re.findall(panter,shop_url)[0]
 45 
 46         item['shop_name'] = response.xpath('//input[@name="vendorName"]/@value').extract()[0]
 47 
 48         item['brand'] = response.xpath('//input[@name="brandName"]/@value').extract()[0]
 49 
 50         item['good_id'] = response.xpath('//input[@name="productCode"]/@value').extract()[0]
 51 
 52         item['actual_price'] = response.xpath('//em[@class="details-pri"]/text()').extract()[0].replace('.','')
 53 
 54         details = ','.join(li_list) + ','
 55 
 56         item['grape_type'] = ''
 57         item['country'] = ''
 58         item['area'] = ''
 59         item['type'] = ''
 60 
 61         if '葡萄品種' in details:
 62             panter = re.compile('葡萄品種：(.*?),', re.S)
 63             results8 = re.findall(panter, details)
 64             if results8 != []:
 65                 item['grape_type'] = results8[0]
 66 
 67         if '產國' in details:
 68             panter = re.compile('產國：(.*?),', re.S)
 69             results8 = re.findall(panter, details)
 70             if results8 != []:
 71                 item['country'] = results8[0]
 72 
 73         if '產地' in details:
 74             panter = re.compile('產地：(.*?),', re.S)
 75             results8 = re.findall(panter, details)
 76             if results8 != []:
 77                 item['area'] = results8[0]
 78 
 79         if '產區' in details:
 80             panter = re.compile('產區：(.*?),', re.S)
 81             results8 = re.findall(panter, details)
 82             if results8 != []:
 83                 item['area'] = results8[0]
 84 
 85         #針對 葡萄酒 白酒
 86         if '型：' in details:
 87             panter = re.compile('型：(.*?),', re.S)
 88             results8 = re.findall(panter, details)
 89             if results8 != []:
 90                 item['type'] =  results8[0]
 91 
 92         #針對洋酒
 93         if '品類' in details:
 94             panter = re.compile('品類：(.*?),', re.S)
 95             results8 = re.findall(panter, details)
 96             if results8 != []:
 97                 item['type'] = results8[0]
 98 
 99 
100 
101         item['details'] = details
102 
103         #評論數據是js渲染後的頁面，經過抓包的方式找到信息
104         #在spider中使用requests爬取會致使進程阻塞
105         pro_url = 'https://www.1919.cn/product/commentData?productCode=' + item['good_id'] + '&productId=346840940029284375&page=1&vendorId=346833407843635201'
106         contents = requests.get(pro_url).text
107         panter = re.compile('<span class="ass-num">(.*?)</span>', re.S)
108         results = re.findall(panter, contents)
109         if results != []:
110             item['comments'] = results[0]
111 
112         yield item

View Code

在這個文件中定義了深度爬取和一個翻頁的方法，經過一個requests請求來解決js渲染的問題。ide

關聯數據庫

拿到數據後，咱們要將數據持久化保存到數據庫。scrapy支持多個數據庫。在此，以mysql舉例。

先建立一個models.py文件來鏈接數據庫。

#Auther: Xiaoliuer Li

from sqlalchemy import Column, String , Integer,BIGINT,TEXT,DECIMAL
from sqlalchemy.ext.declarative import declarative_base

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker


Base = declarative_base()

engine = create_engine('mysql+pymysql://root:@localhost/drinking?charset=utf8')
DBSession = sessionmaker(bind=engine)

class OneNinedata(Base):
    __tablename__ = 'ecommerce_data'

    id = Column(Integer, primary_key=True)
    year = Column(Integer)
    month = Column(Integer)
    plateform = Column(String(20))
    cat_lv_one = Column(String(20))
    cat_lv_two = Column(String(20))
    shop_id = Column(String(20))
    shop_name = Column(String(100))
    shop_area = Column(String(50))
    shop_province = Column(String(20))
    shop_city = Column(String(20))
    good_id = Column(String(20))
    good_name = Column(String(100))
    brand = Column(String(50))
    size = Column(Integer)
    percent = Column(DECIMAL)
    country = Column(String(50))
    area = Column(String(50))
    type = Column(String(20))
    grape_type = Column(String(50))
    num = Column(Integer)
    name_price = Column(Integer)
    actual_price = Column(Integer)
    bottle_price = Column(Integer)
    comments = Column(Integer)
    accumulate_sales = Column(Integer)
    month_sales = Column(Integer)
    month_bottle_sales = Column(Integer)
    month_sale_amounts = Column(BIGINT)
    url = Column(String(256))
    details = Column(TEXT)

View Code

文件中使用的是SQLAlchemy來保存數據庫，不清楚的同志瞭解一下。

在pipelines.py文件中，咱們編寫管道，讓scrapy明確知道要接收哪些數據。

from scrapy.exceptions import DropItem
from .models import OnesNinedata,DBSession


class OneninePipeline(object):

    def open_spider(self, spider):
        self.session = DBSession()

    def process_item(self, item, spider):
        a =OneNinedata(
            year=item['year'], month=item['month'], plateform=item['plateform'], cat_lv_one=item['cat_lv_one'],
            cat_lv_two=item['cat_lv_two'], brand=item['brand'],type=item['type'],name_price=item['name_price'],
            url=item['url'], shop_id=item['shop_id'], shop_name=item['shop_name'],area=item['area'],
            good_name=item['good_name'],grape_type=item['grape_type'],country=item['country'],
            good_id=item['good_id'], actual_price=item['actual_price'], details=item['details'],
            comments=item['comments'])
        self.session.add(a)
        self.session.commit()

    def close_spider(self,spider):
        self.session.close()

View Code

修改settings.py文件，告訴scrapy咱們要將數據保存到數據庫。

ITEM_PIPELINES = {
   'OneNine.pipelines.OneninePipeline': 300,
}

運行scrapy

在命令行中輸入

scrapy crawl onenine

打開數據庫，就能夠看見數據保存在數據中了。

同時咱們還能夠將數據以其餘格式保存在本地。

scrapy crawl onenine -o items.json

上面是例子是以json格式把數據保存在了本地。

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。