簡單爬蟲爬取知乎日報並保存日報網頁到本地

知乎日報爬蟲

# coding=utf-8

import requests
from lxml import html


def spider_zhihudaily():
    url = "http://daily.zhihu.com/"
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"}
    response = requests.get(url, headers = headers)
    # print(response.encoding)
    url_data = response.text
    selector = html.fromstring(url_data)
    ul_list = selector.xpath('//div[@class="main-content-wrap"]/div[@class="row"]//a')

    for ul in ul_list:
        title = ul.xpath("span/text()")[0].replace("?","")
        print(title)
        link = ul.xpath("@href")
        print("https://daily.zhihu.com" + link[0])

        img_url = ul.xpath("img/@src")
        print(img_url[0])

        content = requests.get("https://daily.zhihu.com" + link[0], headers = headers).text
        f = open('./zhihu_html/{0}.html'.format(title), 'w', encoding='utf-8')
        f.write(content)

        print("--------------------------")


if __name__ == "__main__":
    spider_zhihudaily()

# 代碼運行結果

C:\Users\ws\.virtualenvs\pytools\Scripts\python.exe D:/python/pytools/chapter01_spider_book/spider_zhihudaily.py
阿杜的亞文化再研究
https://daily.zhihu.com/story/9713313
https://pic1.zhimg.com/v2-36cbffea9cdf1590fd90a1cfe4184bec.jpg
--------------------------
老一輩人常說的「腳氣在土裏走走好了」，有什麼原理？
https://daily.zhihu.com/story/9713335
https://pic1.zhimg.com/v2-bd24d473b75d213a015f4fe9b6080840.jpg
--------------------------
若是三代蜘蛛俠對決，誰勝算更大？
https://daily.zhihu.com/story/9713283
https://pic4.zhimg.com/v2-3d06d6821d6db4bde782be5ae7b692f3.jpg
--------------------------
威少和保羅互換東家，能盤活火箭和雷霆嗎？
https://daily.zhihu.com/story/9713376
https://pic2.zhimg.com/v2-cfd8375cb3af92e6fb265814faf0ab01.jpg
--------------------------
爲何大部分人會對喝本身的口水感到噁心？
https://daily.zhihu.com/story/9713245
https://pic4.zhimg.com/v2-ca9cb3863923fc574aadd9332fc980c3.jpg
--------------------------
「無反相機」是如何發展起來的？
https://daily.zhihu.com/story/9713258
https://pic3.zhimg.com/v2-53d720366dee5c6c061f2a292248508e.jpg
--------------------------
瞎扯 · 如何正確地吐槽
https://daily.zhihu.com/story/9713369
https://pic1.zhimg.com/v2-88354bf286e898517b6a91db45ab6ec4.jpg
--------------------------
《長安十二時辰》中有哪些值得剖析的細節和彩蛋？
https://daily.zhihu.com/story/9713242
https://pic4.zhimg.com/v2-a3ba34ca7385c9a04fcfa4df2c95b3bb.jpg
--------------------------
未檢疫水果入境，這事到底多嚴重？
https://daily.zhihu.com/story/9713230
https://pic4.zhimg.com/v2-1c82311ae5b9eb6133819cb566b7066b.jpg
--------------------------
如何看待遊戲《戀與製做人》宣佈動畫化？
https://daily.zhihu.com/story/9713296
https://pic3.zhimg.com/v2-bf3a3bfecd18aad6d0525943ec034e22.jpg
--------------------------
怎樣避免早上起牀時的口臭？
https://daily.zhihu.com/story/9713234
https://pic3.zhimg.com/v2-5641bc119681279f94ad00b780492d12.jpg
--------------------------
瞎扯 · 如何正確地吐槽
https://daily.zhihu.com/story/9713164
https://pic1.zhimg.com/v2-29b8253fc73d604e9c3b3331d07507f0.jpg
--------------------------
女朋友婚前，讓我過戶婚前房子的一半給她，該怎麼辦？
https://daily.zhihu.com/story/9713262
https://pic1.zhimg.com/v2-e201ee5ec7aa14b3cbdb78bb6466e34c.jpg
--------------------------
人的聽覺系統是怎樣對聲音進行定位的？
https://daily.zhihu.com/story/9713210
https://pic3.zhimg.com/v2-95e4632978715690366e4cc65adf75ba.jpg
--------------------------
兒子的玩伴很聰明也算頗有心機，還應該讓孩子跟他一塊兒玩嗎？
https://daily.zhihu.com/story/9713204
https://pic3.zhimg.com/v2-17325b8d8847598b78b6a87923973042.jpg
--------------------------
爲何滑石粉做爲一種已知致癌物，還被添加進化妝品裏？
https://daily.zhihu.com/story/9713224
https://pic4.zhimg.com/v2-1bdfd07e9a202258c1b2adebee504a27.jpg
--------------------------
瞎扯 · 如何正確地吐槽
https://daily.zhihu.com/story/9713298
https://pic4.zhimg.com/v2-0e0b61ed9b282eeb180b132b663a9aa3.jpg
--------------------------
如何評價《怪奇物語》第三季
https://daily.zhihu.com/story/9713215
https://pic1.zhimg.com/v2-70d018a670e42f857dc38903f3148d6c.jpg
--------------------------
開普勒是如何得出開普勒三大定律的？
https://daily.zhihu.com/story/9713194
https://pic2.zhimg.com/v2-caca938de9a4fd46175f31630107fa51.jpg
--------------------------
中國有什麼 ACG 愛好者聖地巡禮的地方？
https://daily.zhihu.com/story/9713184
https://pic4.zhimg.com/v2-fca6cb00b1da7d6ec0244c78d20d74af.jpg
--------------------------
爲何把魚放進可樂和雪碧中浸泡 30 天，魚沒了？
https://daily.zhihu.com/story/9713253
https://pic4.zhimg.com/v2-a1a6a6286dcb6249abb0f12cba7e0a7f.jpg
--------------------------
瞎扯 · 如何正確地吐槽
https://daily.zhihu.com/story/9713107
https://pic3.zhimg.com/v2-c3e0252364705040c480e4cd2900503e.jpg
--------------------------
爲何最近的地震如此頻繁？
https://daily.zhihu.com/story/9713203
https://pic3.zhimg.com/v2-8fcae8e7e92a40d67002cff00b0e0e42.jpg
--------------------------
如何看待倫納德與快船達成 4 年 1.42 億美圓簽約協議？
https://daily.zhihu.com/story/9713169
https://pic2.zhimg.com/v2-77848b0003dc703c89a2d7d8a1578b6d.jpg
--------------------------
對科學執着追求的人，能夠整晚打遊戲嗎？
https://daily.zhihu.com/story/9713115
https://pic3.zhimg.com/v2-f73ebfeb032d7a12e1e01aa77e40343e.jpg
--------------------------
怎樣快速去除嘴裏的蒜味？
https://daily.zhihu.com/story/9713140
https://pic3.zhimg.com/v2-ac354902d2f1572c03d6501d21c1f6be.jpg
--------------------------
瞎扯 · 如何正確地吐槽
https://daily.zhihu.com/story/9713158
https://pic1.zhimg.com/v2-0a84aa7f699d493504e7e1cce5d50374.jpg
--------------------------
小事 · 醫生，個人丈夫，還有多久才……死？
https://daily.zhihu.com/story/9713139
https://pic2.zhimg.com/v2-b7ca3f6664db39c41b8d78607e553fe9.jpg
--------------------------
原生動物是如何演化成後生動物的？
https://daily.zhihu.com/story/9713001
https://pic4.zhimg.com/v2-43f652b7055ea70c646eb4a4c484c4c3.jpg
--------------------------
爲何有時候吃了油炸物就會喉嚨痛？
https://daily.zhihu.com/story/9713137
https://pic2.zhimg.com/v2-38033e1a5fe2dc2357928c23f0fa6e61.jpg
--------------------------

Process finished with exit code 0