win7安裝python爬蟲框架scrapy全紀錄

一、安裝Python（建議32位）

建議安裝Python2.7.x，3.x貌似還不支持。
安裝完了記得配置環境，將python目錄和python目錄下的Scripts目錄添加到系統環境變量的Path裏。
在cmd中輸入python若是出現版本信息說明配置完畢。

二、安裝setuptools

用來安裝egg文件，點擊這裏下載python2.7的對應版本的setuptools。

三、安裝lxml

lxml是一種使用 Python 編寫的庫，能夠迅速、靈活地處理 XML。點擊這裏選擇對應的Python版本安裝。也能夠命令行下輸入

easy-install lxml

四、安裝zope.interface

可使用第三步下載的setuptools來安裝egg文件，

easy-install zope.interface

如今也有exe版本，點擊這裏下載。

五、安裝Twisted

Twisted是用Python實現的基於事件驅動的網絡引擎框架，點擊這裏下載。

easy-install Twisted

六、安裝pyOpenSSL

pyOpenSSL是Python的OpenSSL接口，點擊這裏下載。也能夠

easy_install pyOpenSSL==0.13

七、安裝win32py

提供win32api，點擊這裏下載
安裝win32api可能遇到python version 2.7 required 錯誤，這是須要將以下代碼保存爲一個文件register.py

#
# script to register Python 2.0 or later for use with win32all
# and other extensions that require Python registry settings
#
# written by Joakim Loew for Secret Labs AB / PythonWare
#
# source:
# http://www.pythonware.com/products/works/articles/regpy20.htm
#
# modified by Valentine Gogichashvili as described in http://www.mail-archive.com/distutils-sig@python.org/msg10512.html
 
import sys
 
from _winreg import *
 
# tweak as necessary
version = sys.version[:3]
installpath = sys.prefix
 
regpath = "SOFTWARE\\Python\\Pythoncore\\%s\\" % (version)
installkey = "InstallPath"
pythonkey = "PythonPath"
pythonpath = "%s;%s\\Lib\\;%s\\DLLs\\" % (
    installpath, installpath, installpath
)
 
def RegisterPy():
    try:
        reg = OpenKey(HKEY_CURRENT_USER, regpath)
    except EnvironmentError as e:
        try:
            reg = CreateKey(HKEY_CURRENT_USER, regpath)
            SetValue(reg, installkey, REG_SZ, installpath)
            SetValue(reg, pythonkey, REG_SZ, pythonpath)
            CloseKey(reg)
        except:
            print "*** Unable to register!"
            return
        print "--- Python", version, "is now registered!"
        return
    if (QueryValue(reg, installkey) == installpath and
        QueryValue(reg, pythonkey) == pythonpath):
        CloseKey(reg)
        print "=== Python", version, "is already registered!"
        return
    CloseKey(reg)
    print "*** Unable to register!"
    print "*** You probably have another Python installation!"
 
if __name__ == "__main__":
    RegisterPy()

而後命令行下執行

python register.py

便可註冊python2.7

八、安裝Scrapy

終於到了激動人心的時候了！安裝了那麼多小部件以後終於輪到主角登場。
直接在cmd中輸入easy_install scrapy回車便可。
出現以下錯誤

error: Setup script exited with error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from
http://aka.ms/vcpython27

解決辦法的前提是，系統中必須安裝了vs的相關版本。

For Windows installations:

While running setup.py for package installations, Python 2.7 searches for an installed Visual Studio 2008. You can trick Python to use a newer Visual Studio by setting the correct path in VS90COMNTOOLS environment variable before calling setup.py.

If you have Visual Studio 2010 installed, execute

SET VS90COMNTOOLS=%VS100COMNTOOLS%

or with Visual Studio 2012 installed (Visual Studio Version 11)

SET VS90COMNTOOLS=%VS110COMNTOOLS%

or with Visual Studio 2013 installed (Visual Studio Version 12)

SET VS90COMNTOOLS=%VS120COMNTOOLS%

九、檢查安裝

打開一個cmd窗口，在任意位置執行scrapy命令，獲得下列頁面，表示環境配置成功。

Scrapy 0.24.4 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  fetch         Fetch a URL using the Scrapy downloader
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command