打開方式及經常使用選項html
一、打開瀏覽器,F12打開控制檯,找到Network選項卡python
二、控制檯經常使用選項c++
一、Network: 抓取網絡數據包web
一、ALL: 抓取全部的網絡數據包算法
二、XHR:抓取異步加載的網絡數據包json
三、JS : 抓取全部的JS文件api
二、Sources: 格式化輸出並打斷點調試JavaScript代碼,助於分析爬蟲中一些參數瀏覽器
三、Console: 交互模式,可對JavaScript中的代碼進行測試cookie
三、抓取具體網絡數據包後網絡
一、單擊左側網絡數據包地址,進入數據包詳情,查看右側
二、右側:
一、Headers: 整個請求信息 General、Response Headers、Request Headers、Query String、Form Data
二、Preview: 對響應內容進行預覽
三、Response:響應內容
適用於Post類型請求的網站,比例:網易翻譯
response = requests.post(url,data=data,headers=headers)
# data :post的數據(Form表單數據,字典格式)
請求方式的特色
GET請求 : 參數在URL地址中有顯示
POST請求: Form表單提交數據
一、目標
破解有道翻譯接口,抓取翻譯結果
# 結果展現
請輸入要翻譯的詞語: elephant
翻譯結果: 大象
**************************
請輸入要翻譯的詞語: 喵喵叫
翻譯結果: mews
二、實現步驟
具體實現
一、開啓F12抓包,找到Form表單數據以下:
i: 喵喵叫
from: AUTO
to: AUTO
smartresult: dict
client: fanyideskweb
salt: 15614112641250
sign: 94008208919faa19bd531acde36aac5d
ts: 1561411264125
bv: f4d62a2579ebb44874d7ef93ba47e822
doctype: json
version: 2.1
keyfrom: fanyi.web
action: FY_BY_REALTlME
二、在頁面中多翻譯幾個單詞,觀察Form表單數據變化,咱們發現每次查詢單詞,只有salt、sign和ts的值會變化。其餘的值都不會改變。
salt: 15614112641250
sign: 94008208919faa19bd531acde36aac5d
ts: 1561411264125
三、salt、sign和ts的值通常爲本地js文件加密,刷新頁面,找到js文件並分析JS代碼
方法1:Network - JS選項 - 打開一個js文件,左下角格式化符號{},ctrl+F搜索關鍵詞salt,若是存在關鍵詞則找到了加密的js文件。
方法2:控制檯右上角 - Search - 搜索salt - 跳轉到Sources,查看文件 - 把js文件代碼複製到在線js格式化網站,格式化代碼,變成有縮進的可讀代碼。
四、打開JS文件,分析js加密算法,用Python實現
# ts : 通過分析爲13位的時間戳,字符串類型
js代碼實現: r = "" + (new Date).getTime()
python實現: str(int(time.time()*1000))
# salt
js代碼實現: r + parseInt(10 * Math.random(), 10);
python實現: ts + str(random.randint(0,9))
# sign(設置斷點調試,來查看 e 的值,發現 e 爲要翻譯的單詞)
js代碼實現: n.md5("fanyideskweb" + e + salt + "n%A-rKaT5fb[Gy?;N5@Tj")
python實現:
from hashlib import md5
s = md5()
s.update("fanyideskweb" + e + salt + "n%A-rKaT5fb[Gy?;N5@Tj".encode())
sign = s.hexdigest()
python中正則處理header和formdata
一、pycharm進入方法 :Ctrl + r ,選中 Regex
二、處理headers和formdata
(.*): (.*)
"$1": "$2",
三、點擊 Replace All
五、代碼實現
import requests import time import random from hashlib import md5 class YdSpider(object): def __init__(self): # url必定爲F12抓到的 headers -> General -> Request URL self.url = 'http://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule' self.headers = { # 檢查項最高 - 3個 "Cookie": "OUTFOX_SEARCH_USER_ID=970246104@10.169.0.83; OUTFOX_SEARCH_USER_ID_NCOO=570559528.1224236; _nt\ es_nnid=96bc13a2f5ce64962adfd6a278467214,1551873108952; JSESSIONID=aaae9i7plXPlKaJH_gkYw; td_cookie=1844\ 6744072941336803; SESSION_FROM_COOKIE=unknown; ___rl__test__cookies=1565689460872", "Referer": "http://fanyi.youdao.com/", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.\ 0.3809.100 Safari/537.36", } # 獲取,ts、salt、sign def get_salt_sign_ts(self, word): ts = str(int(time.time() * 1000)) # ts salt = ts + str(random.randint(0, 9)) # salt string = "fanyideskweb" + word + salt + "n%A-rKaT5fb[Gy?;N5@Tj" # sign s = md5() s.update(string.encode('utf-8')) sign = s.hexdigest() return salt, sign, ts def attack_yd(self, word): # 1. 先拿到salt,sign,ts salt, sign, ts = self.get_salt_sign_ts(word) # 2. 定義form表單數據爲字典: data={} # 檢查了salt sign data = { "i": word, "from": "AUTO", "to": "AUTO", "smartresult": "dict", "client": "fanyideskweb", "salt": salt, "sign": sign, "ts": ts, "bv": "7e3150ecbdf9de52dc355751b074cf60", "doctype": "json", "version": "2.1", "keyfrom": "fanyi.web", "action": "FY_BY_REALTlME", } # 3. 直接發請求:requests.post(url,data=data,headers=xxx) res = requests.post(url=self.url, data=data, headers=self.headers) # res.json() 將json格式的字符串轉爲python數據類型 html = res.json() result = html['translateResult'][0][0]['tgt'] print(result) # html:{'translateResult': [[{'tgt': '你好', 'src': 'hello'}]], 'errorCode': 0, 'type': 'en2zh-CHS', # 'smartResult': {'entries': ['', 'n. 表示問候, 驚奇或喚起注意時的用語\r\n', 'int. 喂;哈羅\r\n', 'n. # (Hello)人名;(法)埃洛\r\n'], 'type': 1}} def main(self): word = input('請輸入要翻譯的單詞:') # 輸入翻譯單詞 self.attack_yd(word) if __name__ == '__main__': spider = YdSpider() spider.main()
破解百度翻譯接口,抓取翻譯結果數據
實現步驟
一、F12抓包,先翻譯幾個單詞,查看異步數據包,找到翻譯結果的json的地址,觀察查詢參數
一、POST地址: https://fanyi.baidu.com/v2transapi
二、Form表單數據(屢次抓取在變的字段)
from: zh
to: en
sign: 54706.276099 #這個是如何生成的?
token: a927248ae7146c842bb4a94457ca35ee # 基本固定,但也想辦法獲取
二、抓取相關JS文件
右上角 - 搜索 - sign: - 找到具體JS文件(index_c8a141d.js) - 格式化輸出
三、在JS中尋找sign的生成代碼
一、在格式化輸出的JS代碼中搜索: sign: 找到以下JS代碼:sign: m(a),
二、經過設置斷點,找到m(a)函數的位置,即生成sign的具體函數
1. a 爲要翻譯的單詞
2. 鼠標移動到 m(a) 位置處,點擊上方可進入具體m(a)函數代碼塊
四、生成sign的m(a)函數具體代碼以下(在一個大的define中),咱們複製了裏面的3個函數在下面
function a(r) { if (Array.isArray(r)) { for (var o = 0, t = Array(r.length); o < r.length; o++) t[o] = r[o]; return t } return Array.from(r) } function n(r, o) { for (var t = 0; t < o.length - 2; t += 3) { var a = o.charAt(t + 2); a = a >= "a" ? a.charCodeAt(0) - 87 : Number(a), a = "+" === o.charAt(t + 1) ? r >>> a : r << a, r = "+" === o.charAt(t) ? r + a & 4294967295 : r ^ a } return r } function e(r) { var o = r.match(/[\uD800-\uDBFF][\uDC00-\uDFFF]/g); if (null === o) { var t = r.length; t > 30 && (r = "" + r.substr(0, 10) + r.substr(Math.floor(t / 2) - 5, 10) + r.substr(-10, 10)) } else { for (var e = r.split(/[\uD800-\uDBFF][\uDC00-\uDFFF]/), C = 0, h = e.length, f = []; h > C; C++) "" !== e[C] && f.push.apply(f, a(e[C].split(""))), C !== h - 1 && f.push(o[C]); var g = f.length; g > 30 && (r = f.slice(0, 10).join("") + f.slice(Math.floor(g / 2) - 5, Math.floor(g / 2) + 5).join("") + f.slice(-10).join("")) } // var u = void 0 // , l = "" + String.fromCharCode(103) + String.fromCharCode(116) + String.fromCharCode(107); // u = null !== i ? i : (i = window[l] || "") || ""; // String.fromCharCode(103)="g" // String.fromCharCode(103)="t" // String.fromCharCode(103)="k" // window.gtk也就是從網頁源碼中取值 // 進入函數斷點調試 var u = '320305.131321201' for (var d = u.split("."), m = Number(d[0]) || 0, s = Number(d[1]) || 0, S = [], c = 0, v = 0; v < r.length; v++) { var A = r.charCodeAt(v); 128 > A ? S[c++] = A : (2048 > A ? S[c++] = A >> 6 | 192 : (55296 === (64512 & A) && v + 1 < r.length && 56320 === (64512 & r.charCodeAt(v + 1)) ? (A = 65536 + ((1023 & A) << 10) + (1023 & r.charCodeAt(++v)), S[c++] = A >> 18 | 240, S[c++] = A >> 12 & 63 | 128) : S[c++] = A >> 12 | 224, S[c++] = A >> 6 & 63 | 128), S[c++] = 63 & A | 128) } for (var p = m, F = "" + String.fromCharCode(43) + String.fromCharCode(45) + String.fromCharCode(97) + ("" + String.fromCharCode(94) + String.fromCharCode(43) + String.fromCharCode(54)), D = "" + String.fromCharCode(43) + String.fromCharCode(45) + String.fromCharCode(51) + ("" + String.fromCharCode(94) + String.fromCharCode(43) + String.fromCharCode(98)) + ("" + String.fromCharCode(43) + String.fromCharCode(45) + String.fromCharCode(102)), b = 0; b < S.length; b++) p += S[b], p = n(p, F); return p = n(p, D), p ^= s, 0 > p && (p = (2147483647 & p) + 2147483648), p %= 1e6, p.toString() + "." + (p ^ m) }
五、由於js代碼太複雜,咱們很難用js復現算法,所以咱們採用第二種方案,直接將這段代碼寫入本地js文件,利用pyexecjs模塊執行js代碼進行調試
import execjs with open('translate.js', 'r') as f: js_data = f.read() # 建立對象 js_obj = execjs.compile(js_data) sign = js_obj.eval('e("monkey")') # 把monkey調入e函數 print(sign) # 546500.833013
六、獲取token
在js中 token=window.common.token,在響應(響應即網頁源碼)中想辦法獲取此值,百度翻譯的網頁地址爲token_url = 'https://fanyi.baidu.com/?aldtype=16047'。用正則去匹配 regex: "token: '(.*?)'"
代碼實現
import requests import re import execjs class BaiduTranslateSpider(object): def __init__(self): self.token_url = 'https://fanyi.baidu.com/?aldtype=16047' self.post_url = 'https://fanyi.baidu.com/v2transapi' self.headers = { 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3', # 'accept-encoding': 'gzip, deflate, br', 'accept-language': 'zh-CN,zh;q=0.9', 'cache-control': 'no-cache', 'cookie': 'BAIDUID=52920E829C1F64EE98183B703F4E37A9:FG=1; BIDUPSID=52920E829C1F64EE98183B703F4E37A9; PSTM=1562657403; to_lang_often=%5B%7B%22value%22%3A%22en%22%2C%22text%22%3A%22%u82F1%u8BED%22%7D%2C%7B%22value%22%3A%22zh%22%2C%22text%22%3A%22%u4E2D%u6587%22%7D%5D; REALTIME_TRANS_SWITCH=1; FANYI_WORD_SWITCH=1; HISTORY_SWITCH=1; SOUND_SPD_SWITCH=1; SOUND_PREFER_SWITCH=1; delPer=0; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; BCLID=6890774803653935935; BDSFRCVID=4XAsJeCCxG3DLCbwbJrKDGwjNA0UN_I3KhXZ3J; H_BDCLCKID_SF=tRk8oIDaJCvSe6r1MtQ_M4F_qxby26nUQ5neaJ5n0-nnhnL4W46bqJKFLtozKMoI3C7fotJJ5nololIRy6CKjjb-jaDqJ5n3bTnjstcS2RREHJrg-trSMDCShGRGWlO9WDTm_D_KfxnkOnc6qJj0-jjXqqo8K5Ljaa5n-pPKKRAaqD04bPbZL4DdMa7HLtAO3mkjbnczfn02OP5P5lJ_e-4syPRG2xRnWIvrKfA-b4ncjRcTehoM3xI8LNj405OTt2LEoDPMJKIbMI_rMbbfhKC3hqJfaI62aKDs_RCMBhcqEIL4eJOIb6_w5gcq0T_HttjtXR0atn7ZSMbSj4Qo5pK95p38bxnDK2rQLb5zah5nhMJS3j7JDMP0-4rJhxby523i5J6vQpnJ8hQ3DRoWXPIqbN7P-p5Z5mAqKl0MLIOkbC_6j5DWDTvLeU7J-n8XbI60XRj85-ohHJrFMtQ_q4tehHRMBUo9WDTm_DoTttt5fUj6qJj855jXqqo8KMtHJaFf-pPKKRAashnzWjrkqqOQ5pj-WnQr3mkjbn5yfn02OpjPX6joht4syPRG2xRnWIvrKfA-b4ncjRcTehoM3xI8LNj405OTt2LEoC0XtIDhMDvPMCTSMt_HMxrKetJyaR0JhpjbWJ5TEPnjDUOdLPDW-46HBM3xbKQw5CJGBf7zhpvdWhC5y6ISKx-_J68Dtf5; ZD_ENTRY=baidu; PSINO=2; H_PS_PSSID=26525_1444_21095_29578_29521_28518_29098_29568_28830_29221_26350_29459; locale=zh; Hm_lvt_64ecd82404c51e03dc91cb9e8c025574=1563426293,1563996067; from_lang_often=%5B%7B%22value%22%3A%22zh%22%2C%22text%22%3A%22%u4E2D%u6587%22%7D%2C%7B%22value%22%3A%22en%22%2C%22text%22%3A%22%u82F1%u8BED%22%7D%5D; Hm_lpvt_64ecd82404c51e03dc91cb9e8c025574=1563999768; yjs_js_security_passport=2706b5b03983b8fa12fe756b8e4a08b98fb43022_1563999769_js', 'pragma': 'no-cache', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36',} # 獲取token和gtk def get_token(self): # 定義請求頭 r = requests.get(self.token_url, headers=self.headers) token = re.findall(r"token: '(.*?)'", r.text) window_gtk = re.findall(r"window.*?gtk = '(.*?)';</script>", r.text) if token: return token[0], window_gtk[0] # 獲取sign def get_sign(self, word, gtk): with open('translate.js', 'r') as f: js_data = f.read() exec_object = execjs.compile(js_data) sign = exec_object.eval('e("{}","{}")'.format(word, gtk)) return sign # 主函數 def main(self, word, fro, to): token, gtk = self.get_token() sign = self.get_sign(word, gtk) # 找到form表單數據以下,sign和token須要想辦法獲取 form_data = { 'from': fro, 'to': to, 'query': word, 'transtype': 'realtime', 'simple_means_flag': '3', 'sign': sign, 'token': token } r = requests.post(self.post_url, data=form_data, headers=self.headers) print(r.json()['trans_result']['data'][0]['dst']) if __name__ == '__main__': spider = BaiduTranslateSpider() choice = input('1. 英譯漢 2. 漢譯英 : ') word = input('請輸入要翻譯的單詞:') if choice == '1': fro, to = 'en', 'zh' elif choice == '2': fro, to = 'zh', 'en' spider.main(word, fro, to)