Scrapy安轉遇到問題

最近嘗試使用Scrapy進行數據抓取，並嘗試在windows7 64位系統上安裝scrapy，下面總結記錄遇到兩個問題和解決方法：

 

scrapy官網的地址爲： http://scrapy.org/

一、首先按照官網的說明，直接pip安裝scrapy，報如下錯誤，提示缺乏VC++9.0，報錯信息有給出具體的說明和解決方法。

>>pip install scrapy

error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from  http://aka.ms/vcpython27

解決方法：直接點擊下載VCForPython27.msi，也直接訪問下面的鏈接https://www.microsoft.com/en-us/download/details.aspx?id=44266

 

二、下載並安裝VCForPython27.msi，而後再運行pip install scrapy，再次報錯，提示找不到libxml2庫。

>>pip install scrapy

c:\users\zjn3645\appdata\local\temp\xmlXPathInit7hkp2z.c(1) : fatal error C1083: Cannot open include file: 'libxml/xpath.h': No such file or directory

*********************************************************************************

Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?

*********************************************************************************

error: command 'C:\\Users\\zjn3645\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\cl.exe' failed with exit status 2

 

解決方法：

使用easy_install安裝lxml，而後再次安裝pip install scrapy，成功~

>>easy_install lxml

 

三、scrapy安裝完成，運行官網首頁的樣例報錯，緣由是缺乏pywin32

>>scrapy runspider myspider.py

exceptions.ImportError: No module named win32api

2016-03-09 10:17:49 [twisted] CRITICAL:

 

解決方法：按照官方文檔的說明安裝

Install  pywin32 from  https://sourceforge.net/projects/pywin32/files/pywin32/

這裏有詳細的說明 http://doc.scrapy.org/en/latest/intro/install.html#intro-install-platform-notes

pywin32有32位和64位版本，

對於64位的windows 7，若是python環境變量已經正確安裝，pywin32安裝的時候仍然報找不到python的錯誤，嘗試pywin32的32位版本。

 

總結：在安裝和使用新的工具以前必定要先看看官方文檔的說明，搞清楚安裝和使用的前提條件！

四、關閉代理

默認使用代理，有些網頁使用本地代理沒法訪問！

2016-03-09 15:18:21 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023

2016-03-09 15:18:21 [scrapy] DEBUG: Crawled (403) <GET http://xxx.com.cn/xxx.html>

(referer: None)

2016-03-09 15:18:21 [scrapy] DEBUG: Ignoring response <403 http://xxx.com.cn/xxx.html>: HTTP status code is not handled or not allowed

關閉代理

修改settings.py以下

DOWNLOADER_MIDDLEWARES = {

    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,

}

 

 

 

python scrapy 安裝 lxml 報 ERROR: 'xslt-config' 不是內部或外部命令，也不是可運行的程序的解決辦法。

 

1  解決辦法 

手動安裝 lxml包

 

去http://pypi.python.org/simple/lxml/，下載win7 x64的(注意下載對應你係統版本)：

https://pypi.python.org/packages/2.7/l/lxml/lxml-2.2.8.win-amd64-py2.7.exe#md5=cfcf7f07a5016a5934271cddde4bbcbe

而後從新打開cmd去pip install Scrapy，解決了此xml的問題了

 

 

 mac安裝遇到問題：

I resolved a problem ,when you you install scrapy-----{mac os system}, maybe you will get error like:

'''
sted>=10.0.0->Scrapy)
Installing collected packages: six, w3lib, parsel, PyDispatcher, Twisted, Scrapy
  Found existing installation: six 1.4.1
    DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-8.1.1-py2.7.egg/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-8.1.1-py2.7.egg/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/Library/Python/2.7/site-packages/pip-8.1.1-py2.7.egg/pip/req/req_set.py", line 726, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-8.1.1-py2.7.egg/pip/req/req_install.py", line 746, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-8.1.1-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-8.1.1-py2.7.egg/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
    copy2(src, real_dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
    copystat(src, dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-ZVi5QO-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
LuoTimdeMacBook-Pro-2:~ luotim$ sudo pip install Scrapy --ingnore-installed six

'''

Six is a Python 2 and 3 compatibility library.

frist thanks google and what's fuck baidu ! so you should be do this to resolved the problem:
一、Download the six-1.10.0.tar.gz package
wget https://pypi.python.org/packages/b3/b2/238e2590826bfdd113244a40d9d3eb26918bd798fc187e2360a8367068db/six-1.10.0.tar.gz#md5=34eed507548117b2ab523ab14b2f8b55

2、UnZip software package
tar -zxvf six-1.10.0.tar.gz

3、Use this command to install it.
cd cd six-1.10.0
sudo python setup.py install

http://stackoverflow.com/questions/29485741/unable-to-upgrade-python-six-package-in-mac-osx-10-10-2