Mac安裝scrapy並建立項目

最近想要學習Python3+Scrapy的爬蟲技術，須要先安裝Python3和Scrapy。由於Mac是自帶Python2.7的。安裝Python3.6版本有兩種方法，一種是升級，一種是額外安裝3.6版本。

安裝Python3.6版本

其實安裝3.6版本也就是在官網上直接下載以後安裝，和普通的mac軟件安裝方式是同樣的~~

www.python.org/downloads/r…

安裝完成以後，不會覆蓋原來的Python，新安裝的Python3.6版本會在 /usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/bin/python3.6 文件中

此時在終端直接輸入 python 會執行python2.7版本

$ python
Python 2.7.10 (default, Jul 15 2017, 17:16:57) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
複製代碼

輸入 python3 則會執行Python3.6版本

$ python3
Python 3.6.2 (default, Sep 11 2017, 16:24:44) 
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
複製代碼

安裝Scrapy

接下來就能夠開始安裝scrapy了

python3.6中自帶 pip，因此不須要額外安裝，能夠直接在終端輸入 pip3 --version查看版本和路徑

$ pip3 --version
pip 9.0.1 from /usr/local/lib/python3.6/site-packages (python 3.6)
複製代碼

使用 pip3 安裝scrapy

$ pip3 install Scrapy
複製代碼

這裏的Scrapy必定要首字母大寫，否則會在安裝的過程當中報錯~~

Collecting scrapy  
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x103aa2c88>: Failed to establish a new connection: [Errno 61] Connection refused',)': /simple/scrapy/  
Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x103aa29e8>: Failed to establish a new connection: [Errno 61] Connection refused',)': /simple/scrapy/  
Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x103aa2630>: Failed to establish a new connection: [Errno 61] Connection refused',)': /simple/scrapy/  
Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x103aa2f28>: Failed to establish a new connection: [Errno 61] Connection refused',)': /simple/scrapy/  
Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x103aa2be0>: Failed to establish a new connection: [Errno 61] Connection refused',)': /simple/scrapy/  
Could not find a version that satisfies the requirement scrapy (from versions: )  
No matching distribution found for scrapy  
複製代碼

安裝成功以後，能夠直接在終端上輸入 scrapy 查看版本號及使用

$ scrapy
Scrapy 1.4.0 - no active project
    
Usage:
   scrapy <command> [options] [args]
    
Available commands:
   bench         Run quick benchmark test
   fetch         Fetch a URL using the Scrapy downloader
   genspider     Generate new spider using pre-defined templates
   runspider     Run a self-contained spider (without creating a project)
   settings      Get settings values
   shell         Interactive scraping console
   startproject  Create new project
   version       Print Scrapy version
   view          Open URL in browser, as seen by Scrapy
    
   [ more ]      More commands available when run from project directory
    
 Use "scrapy <command> -h" to see more info about a command
複製代碼

建立Scrapy項目

在pycharm中是沒有直接建立scrapy項目的，能夠使用 scrapy 命令手動新建項目（ArticleSpider爲項目名稱）

$ scrapy startproject ArticleSpider
複製代碼