python3.6爬蟲須要安裝的模塊

庫的安裝：

---

    內置庫

        urllib

        re

    須要安裝的庫

        requests

            pip3 install requests

        selenium

            pip3 install  selenium

        chromedriver

            下載驅動，放在配置好的環境變量下

            http://npm.taobao.org/mirrors/chromedriver/

        phantomjs(×××面瀏覽器)

            下載，而且配置環境變量

            http://phantomjs.org/

        lxml

            pip3 install lxml

            可能下載失敗

            用下面的方式安裝

            https://pypi.python.org/pypi/lxml

            

            

            首先安裝

            pip3 install wheel

            而後載pip3 install 下載的文件路徑和名字

        beautifulsoup4（依賴lxml）

            pip3 install beautifulsoup4

        pyquery

            pip3 install pyquery

    存儲庫：

        pymysql

            pip3 install pymysql

        pymongo

            pip3 install pymongo

        redis

            pip3 install redis

        flask(代理IP的庫)

            pip3 install flask

        django

            pip3 install django

        jupyter(強大的記事本)

            pip3 install jupyter

            jupyter notebook（啓動服務）

    框架：

        PySpider

            pip3 install PySpider

        scrapy

            1. wheel

                pip install wheel

            2. lxml

                http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml

            3. PyOpenssl

                https://pypi.python.org/pypi/pyOpenSSL#downloads

            4. Twisted

                http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted

                連接：https://pan.baidu.com/s/1oAh2Dse 密碼：okk0

            5. Pywin32

                https://sourceforge.net/projects/pywin32/files/pywin32/Build%20220/

                https://sourceforge.net/projects/pywin32/files/pywin32/Build%20221/pywin32-221.win-amd64-py3.6.exe/download

                軟件和出現問題修復腳本，

                連接：https://pan.baidu.com/s/1mj2VjxI 密碼：ls54

            6. Scrapy

                pip3 install scrapy

---

 

MongoDB 的安裝：

---

   

啓動服務：

    D:\Program Files\MongoDB\Server\3.6\bin>"D:\Program Files\MongoDB\Server\3.6\bin\mongod.exe" --dbpath ../data/db

配置可視化服務：

    創建日誌文件在c盤建立C:\data\db文件夾和C:\data\logs\logs.txt文件

    D:\Program Files\MongoDB\Server\3.6\bin>mongod --bind_ip 0.0.0.0 --logpath C:\data\logs\logs.txt  --logappend --dbpath C:\data\db --port 27017 --serviceName "mongodb" --serviceDisplayName "mongodb" --install

 

圖形化管理頁面軟件

    https://download.robomongo.org/1.2.1/windows/robo3t-1.2.1-windows-x86_64-3e50a65.exe

---

redis數據庫安裝：

---

參考上篇博客：    

    https://blog.51cto.com/tdcqvip/2072845