Scrapy 入門

Scrapy

https://docs.scrapy.org/en/latest/intro/overview.html

Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

Even though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.

安裝

一、 安裝python 2.7

二、安裝 pip

https://jingyan.baidu.com/article/e73e26c0d94e0524adb6a7ff.html

進入命令行，而後把目錄切換到python的安裝目錄下的Script文件夾下，運行 easy_inatall pip

三、 安裝scrpay

https://docs.scrapy.org/en/latest/intro/install.html

pip install Scrapy

若是安裝不成功，多嘗試兩次

運行報錯：

ImportError： No module named win32api

解決辦法：

https://blog.csdn.net/jianhong1990/article/details/46406499

下載安裝： http://sourceforge.net/projects/pywin32/files%2Fpywin32/

運行

保存以下腳本爲

quotes_spider.py

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/tag/humor/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.xpath('span/small/text()').extract_first(),
            }

        next_page = response.css('li.next a::attr("href")').extract_first()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

在此目錄下，運行

scrapy runspider quotes_spider.py -o quotes.json

相關知識

CSS選擇器

http://www.w3school.com.cn/cssref/css_selectors.asp

XPATH

http://www.cnblogs.com/fdszlzl/archive/2009/06/02/1494836.html

Python yield

https://www.ibm.com/developerworks/cn/opensource/os-cn-python-yield/