python爬蟲入門--urllib

一，抓百度網頁對象，輸出網頁內容

from urllib import request 
req = request.urlopen("http://www.baidu.com")
print(req.read().decode("utf-8"))

1,其中from urllib import request，這個再cmd中檢查是否安裝，直接輸入from urllib import request，若是沒安裝則輸入：pip install urllib

2,req.read().decode("utf-8"),讀取網頁對象內容，以utf-8編碼讀取

 

二，模擬真實瀏覽器

from urllib import request 
resq = request.Request("http://www.baidu.com");
resq.add_header("User-Agent","Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Mobile Safari/537.36")
req = request.urlopen(resq)
print(req.read().decode("utf-8"))

這樣作是由於有些網站會不讓爬蟲，因此咱們要模擬真實的的請求。

 

 

三，發送post請求

1，導包 from urllib import parse

from urllib import request 
from urllib import parse

resq = request.Request("http://www.thsrc.com.tw/tw/TimeTable/SearchResult");
resq.add_header("User-Agent","Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Mobile Safari/537.36")
resq.add_header("Origin","http://www.thsrc.com.tw")
postDate = parse.urlencode([
        ("StartStation","2f940836-cedc-41ef-8e28-c2336ac8fe68"),
        ("EndStation","977abb69-413a-4ccf-a109-0272c24fd490"),
        ("SearchDate","2017/12/09"),
        ("SearchTime","21:30"),
        ("SearchWay","DepartureInMandarin")
        ]);
req = request.urlopen(resq,data=postDate.encode("utf-8"))
print(req.read().decode("utf-8"))

2，模擬請求臺灣高鐵，獲取高鐵班次信息。