使用python抓取App數據

App接口爬取數據過程
使用抓包工具
手機使用代理，app全部請求經過抓包工具
得到接口，分析接口
反編譯apk獲取key
突破反爬限制
須要的工具：
夜神模擬器
Fiddler
Pycharm
實現過程
首先下載夜神模擬器模擬手機也能夠用真機，而後下載Fiddler抓取手機APP數據包，分析接口完成之後使用Python實現爬蟲程序

Fiddler安裝配置過程
第一步：下載神器Fiddler
Fiddler下載完成以後，傻瓜式的安裝一下！

第二步：設置Fiddler
打開Fiddler, Tools-> Fiddler Options (配置完後記得要重啓Fiddler)
選中」Decrpt HTTPS traffic」, Fiddler就能夠截獲HTTPS請求
選中」Allow remote computers to connect」. 是容許別的機器把HTTP/HTTPS請求發送到Fiddler上來
 

 

記住這個端口號是:8888

夜神模擬器安裝配置過程

第一步:下載安裝

夜神模擬器下載完成以後，傻瓜式的安裝一下！

第二步:配置橋接 實現互通

首先將當前手機網絡橋接到本電腦網絡 實現互通 

安裝完成橋接驅動後配置IP地址，要配成和本機互通的網段，配置完成後打開主機cmd終端ping通ok

第三步:配置代理

1. 打開主機cmd 

1. 輸入ipconfig查看本機IP 
2. 配置代理 
   進入夜神模擬器–打開設置–打開WLAN 
    
    
3. 點擊修改網絡–配置代理 以下圖： 
   
   配置完後保存 
   到這裏咱們就設置好全部的值，下面就來測試一下，打開手機的超級課程表APP 
   
4. 1. 在夜神模擬器上下載你想爬取得App使用Fiddler抓包分析api後使用python進行爬取就能夠了

   爬取充電網APP實例

   爬取部份內容截圖： 
   

5. 部分python代碼分享：

   import requests
   import city
   import json
   import jsonpath
   import re

   city_list = city.jsons
   tags_list = city.Tag

   def city_func(city_id):
   try:
   city = jsonpath.jsonpath(city_list, '$..sub[?(@.code=={})]'.format(int(city_id)))[0]["name"]
   except:
   city = jsonpath.jsonpath(city_list, '$[?(@.code=={})]'.format(int(city_id)))[0]["name"]
   return city

   def tags_func(tags_id):
   tags_join = []
   if tags_id:
   for tags in tags_id:
   t = jsonpath.jsonpath(tags_list,'$..spotFilterTags[?(@.id=={})]'.format(int(tags)))
   tags_join.append(t[0]["title"])

   return ('-'.join(tags_join))

   def split_n(ags):
   return re.sub('\n',' ',ags)

   
   def request(page):
   print('開始下載第%d頁'%page)
   url = 'https://app-api.chargerlink.com/spot/searchSpot'
   two_url = "https://app-api.chargerlink.com/spot/getSpotDetail?spotId={d}"
   head = {
   "device": "client=android&cityName=%E5%8C%97%E4%BA%AC%E5%B8%82&cityCode=110106&lng=116.32154281224254&device_id=8A261C9D60ACEBDED7CD3706C92DD68E&ver=3.7.7&lat=39.895024107858724&network=WIFI&os_version=19",
   "appId": "20171010",
   "timestamp": "1532342711477",
   "signature": "36daaa33e7b0d5d29ac9c64a2ce6c4cf",
   "forcecheck": "1",
   "Content-Type": "application/x-www-form-urlencoded",
   "Content-Length": "68",
   "Host": "app-api.chargerlink.com",
   "Connection": "Keep-Alive",
   "User-Agent": "okhttp/3.2.0"
   }

   
   data = {
   "userFilter[operateType]": 2,
   "cityCode": 110000,
   "sort": 1,
   "page": page,
   "limit": 10,
   }

   response = requests.post(url,data=data,headers=head)
   #獲取數據
   data = response.json()
   for i in data['data']:
   c = []
   id = i['id']
   name = i["name"] #充電樁名
   phone = i["phone"] #手機號
   num = i['quantity'] #有幾個充電樁
   city = city_func(i["provinceCode"]) #城市
   tags =tags_func(i["tags"].split(','))#標籤
   message = c + [id,name,phone,num,city,tags]
   parse_info(two_url.format(d=id),message)

   def parse_info(url,message):

   #打開文件
   with open('car.csv','a',encoding='utf-8')as c:
   head = {
   "device": "client=android&cityName=&cityCode=&lng=116.32154281224254&device_id=8A261C9D60ACEBDED7CD3706C92DD68E&ver=3.7.7&lat=39.895024107858724&network=WIFI&os_version=19",
   "TOKEN": "036c8e24266c9089db50899287a99e65dc3bf95f",
   "appId": "20171010",
   "timestamp": "1532357165598",
   "signature": "734ecec249f86193d6e54449ec5e8ff6",
   "forcecheck": "1",
   "Host": "app-api.chargerlink.com",
   "Connection": "Keep-Alive",
   "User-Agent": "okhttp/3.2.0",
   }
   #發起詳情請求
   res = requests.get(url,headers=head)
   price = split_n(jsonpath.jsonpath(json.loads(res.text),'$..chargingFeeDesc')[0]) #價錢
   payType = jsonpath.jsonpath(json.loads(res.text),'$..payTypeDesc')[0] #支付方式
   businessTime =split_n(jsonpath.jsonpath(json.loads(res.text),'$..businessTime')[0]) #營業時間
   result = (message + [price,payType,businessTime])
   r = ','.join([str(i) for i in result])+',\n'
   c.write(r)

   def get_page():
   url = 'https://app-api.chargerlink.com/spot/searchSpot'
   head = {
   "device": "client=android&cityName=%E5%8C%97%E4%BA%AC%E5%B8%82&cityCode=110106&lng=116.32154281224254&device_id=8A261C9D60ACEBDED7CD3706C92DD68E&ver=3.7.7&lat=39.895024107858724&network=WIFI&os_version=19",
   "appId": "20171010",
   "timestamp": "1532342711477",
   "signature": "36daaa33e7b0d5d29ac9c64a2ce6c4cf",
   "forcecheck": "1",
   "Content-Type": "application/x-www-form-urlencoded",
   "Content-Length": "68",
   "Host": "app-api.chargerlink.com",
   "Connection": "Keep-Alive",
   "User-Agent": "okhttp/3.2.0"
   }

   data = { "userFilter[operateType]": 2, "cityCode": 110000, "sort": 1, "page": 1, "limit": 10, } response = requests.post(url, data=data, headers=head) # 獲取數據 data = response.json() total = (data["pager"]["total"]) page_Size = (data["pager"]["pageSize"]) totalPage = (data['pager']["totalPage"]) print('當前共有{total}個充電樁，每頁展現{page_Size}個，共{totalPage}頁'.format(total=total,page_Size=page_Size,totalPage=totalPage))if __name__ == '__main__': get_page() start = int(input("親,請輸入您要獲取的開始頁:")) end = int(input("親,請輸入您要獲取的結束頁:")) for i in range(start,end+1): request(i)--------------------- 做者：愛python的王三金 來源：CSDN 原文：https://blog.csdn.net/qq_37275405/article/details/81181439 版權聲明：本文爲博主原創文章，轉載請附上博文連接！