今天開始嘗試爬取一下知乎,看一下這個網站都有什麼好玩的內容能夠爬取到,可能斷斷續續會寫幾篇文章,今天首先爬取最簡單的,單一文章的全部回答,爬取這個沒有什麼難度。mongodb
找到咱們要爬取的頁面,我隨便選了一個編程
https://www.zhihu.com/question/292393947
json
1084
個回答,數據量能夠說很是小了,就爬取它吧。api
爬取使用requests
存儲使用 mongodb 就能夠了工具
爬取地址通過分析以後,找到了一個能夠返回json的數據接口網站
提取連接,看一下各參數的意思,方便咱們程序模擬編碼
https://www.zhihu.com/api/v4/questions/292393947/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics&limit=5&offset=10&sort_by=default
上面的鏈接進行了URL編碼,去找個解碼工具解析一下,編程下面的URL就比較好解釋了,answers
後面跟了一堆的參數,應該是返回的關鍵字,找到limit
每頁顯示的數據量,offset
偏移量,咱們下拉滾動條,發現這個在不斷的疊加+5,sort_by
就是排序。url
https://www.zhihu.com/api/v4/questions/292393947/answers?include=data[*].is_normal,admin_closed_comment,reward_info,is_collapsed,annotation_action,annotation_detail,collapse_reason,is_sticky,collapsed_by,suggest_edit,comment_count,can_comment,content,editable_content,voteup_count,reshipment_settings,comment_permission,created_time,updated_time,review_info,relevant_info,question,excerpt,relationship.is_authorized,is_author,voting,is_thanked,is_nothelp;data[*].mark_infos[*].url;data[*].author.follower_count,badge[*].topics&limit=5&offset=10&sort_by=default
作好上面的工做,接下來就是爬取了,我簡化了一下爬取的地址,只保留了一些關鍵的信息code
https://www.zhihu.com/api/v4/questions/292393947/answers?include=comment_count,content,voteup_count,reshipment_settings,is_author,voting,is_thanked,is_nothelp;data[*].mark_infos[*].url;data[*].author.follower_count,badge[*].topics&limit=5&offset=0&sort_by=default
分析完畢以後,發現代碼很是簡單了orm
import requests from fake_useragent import UserAgent ############## 數據存儲 import pymongo import time DATABASE_IP = '127.0.0.1' DATABASE_PORT = 27017 DATABASE_NAME = 'sun' client = pymongo.MongoClient(DATABASE_IP,DATABASE_PORT) db = client.sun db.authenticate("dba", "dba") collection = db.zhihuone # 準備插入數據 ################################## class ZhihuOne(object): def __init__(self,totle): self._offset = 0 self._totle = totle #self._ua = UserAgent() def run(self): print("正在抓取 {} 數據".format(self._offset)) headers = { "upgrade-insecure-requests":"1", "user-agent": "Mozilla/5.0 (Windows NT 10.0; WOW64)" } with requests.Session() as s: try: with s.get("https://www.zhihu.com/api/v4/questions/292393947/answers?include=comment_count,content,voteup_count,reshipment_settings,is_author,voting,is_thanked,is_nothelp;data[*].mark_infos[*].url;data[*].author.follower_count,badge[*].topics&limit=5&offset={}&sort_by=default".format(self._offset),headers=headers,timeout=3) as rep: data = rep.json() if data: collection.insert_many(data["data"]) except Exception as e: print(e.args) finally: if self._offset <= self._totle: self._offset = self._offset + 5 # 每次+5 print("防止被辦,休息3s") time.sleep(3) self.run() else: print("全部數據獲取完畢") if __name__ == '__main__': # 偏移量是0,5,10 i=1 (i-1)*5 zhi = ZhihuOne(1084) zhi.run()
上面主程序入口中,我寫了個1084
,這個偷懶,就硬編碼了,數據固然也能夠經過爬取獲取,沒有任何問題
本篇文章是知乎文章爬取器之一,接下來完善的功能