Python 2.7_利用xpath語法爬取豆瓣圖書top250信息_20170129

大年初二,忙完家裏一些事,順帶有人交流爬取豆瓣圖書top250

一、構造urls列表 urls=['https://book.douban.com/top250?start={}'.format(str(i) for i in range(0, 226, 25))]

二、模塊 requests獲取網頁源代碼 lxml 解析網頁 xpath提取

三、提取信息

四、能夠封裝成函數 此處沒有封裝調用

python代碼：

#coding:utf-8
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
from lxml import etree
import requests
urls=['https://book.douban.com/top250?start={}'.format(str(i) for i in range(0, 226, 25))]
for url in urls:
    html=requests.get(url).content
    selector=etree.HTML(html)
    infos=selector.xpath('//tr[@class="item"]')
    for info in infos:
        book_name = info.xpath('td/div/a/@title')[0]
        book_url = info.xpath('td/div/a/@href')[0]
        published_infos = str(info.xpath('td/p/text()')[0])
        splitlistinfos = published_infos.split('/')
        #print splitlistinfos
        published_date=str(splitlistinfos[-2])
        #print published_date
        price = str(splitlistinfos[-1])
        #print price
        rate = info.xpath('td/div/span[2]/text()')[0]
        # comment_nums = info.xpath('td/div/span[3]/text()')[0]
        # print comment_nums
        comment_nums = info.xpath('td/div/span[3]/text()')[0].strip('(').strip().strip(')').strip().strip('人評價').strip() +'人評價'
        introduceinfo = info.xpath('td/p/span/text()')
        print book_name,book_url,published_date,price,rate,comment_nums,introduceinfo[0] if len(introduceinfo) > 0 else ''