HttpClient 模擬登錄知乎

時間 2019-11-19

標籤 httpclient 模擬登錄欄目系統網絡简体版

原文原文鏈接

最近作爬蟲相關工做，咱們平時用HttpWebRequest 比較多，每個Url都要建立一個HttpWebRequest實例，html

並且有些網站驗證比較複雜，在登錄及後續抓取數據的時候，每次請求須要把上次的Cookie傳遞給此次請求。git

記得這篇博客(http://www.cnblogs.com/dudu/archive/2013/03/05/httpclient.html)結尾,dudu總結了：github

HttpClient最不同凡響的地方是同一個HttpClient實例能夠發出屢次請求，每次請求是能夠是徹底不一樣的URL。web

而一個HttpWebRequest實例對應於一個Url的一次請求。這纔是HttpClient與HttpWebRequest的最大區別所在。json

那麼爲何不用HttpClient呢？cookie

源碼地址：https://github.com/zzhi/Spider4Netapp

本着學習的目的，那我就拿知乎練習一下，看看HttpClient好用否？ide

1，分析登錄頁：https://www.zhihu.com/#signin工具

根據上圖設置 DefaultRequestHeaderspost

 HttpClient h = new HttpClient(
            new HttpClientHandler
            {
              //CookieContainer = cookies,
              AutomaticDecompression = DecompressionMethods.GZip //防止返回的json亂碼
                                       | DecompressionMethods.Deflate
            });
            
            h.DefaultRequestHeaders.Add("UserAgent", Configs.ChromeAgent);
            h.DefaultRequestHeaders.Add("Accept-Language", "zh-CN,zh;q=0.8,en-US;q=0.6,en;q=0.4");
            h.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch");
            h.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
            //1.首頁
            var response = await h.GetAsync(index);
            string content = await response.Content.ReadAsStringAsync();
            if (response.IsSuccessStatusCode)
            {
                //獲取隱藏的input值
                HtmlDocument doc = new HtmlDocument();
                doc.LoadHtml(content);
                var xsdf = DocumentHelper.GetInputValue(doc, "_xsrf");//登陸須要
                nameValue["_xsrf"] = xsdf;
            }
            else
            {
                return null;
            }

2，分析登錄頁:https://www.zhihu.com/login/phone_num(我這裏是手機和密碼登陸):

分析：除了要設置DefaultRequestHeaders，還需獲取_xsrf的值<input type="hidden" name="_xsrf" value="3bc639713d3f8bb899009a7dfa37f9d2"/>，

這裏還需注意：1, Content-Type: application/x-www-form-urlencoded; charset=UTF-8 該如何設置? 參考地址：http://ronaldrosiernet.azurewebsites.net/Blog/2013/12/07/posting_urlencoded_key_values_with_httpclient

2, 登陸返回的JSON結果是亂碼，該如何處理? 參考地址：http://stackoverflow.com/questions/9242472/retrieve-json-data-with-httpclient

//2.登錄
            h.DefaultRequestHeaders.Clear();
            h.DefaultRequestHeaders.Add("UserAgent", Configs.ChromeAgent);
            h.DefaultRequestHeaders.Add("X-Requested-With", "XMLHttpRequest");
            h.DefaultRequestHeaders.Add("Origin", index);
            h.DefaultRequestHeaders.Add("Accept-Language", "zh-CN,zh;q=0.8,en-US;q=0.6,en;q=0.4");
            h.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch");
            h.DefaultRequestHeaders.Add("Accept", "*/*");
            //post參數
            nameValue["password"] = PassWord;
            nameValue["captcha_type"] = "cn";
            nameValue["remember_me"] = "true";
            nameValue["phone_num"] = Phone;
            StringBuilder sb = new StringBuilder();
            foreach (var key in nameValue.AllKeys)
            {
                sb.AppendFormat("{0}={1}&", key, nameValue[key]);
            }
            var str = sb.ToString().TrimEnd('&');
            var request = new HttpRequestMessage(HttpMethod.Post, login);
            var requestContent = str;
            request.Content = new StringContent(requestContent, Encoding.UTF8, "application/x-www-form-urlencoded");

            response = await h.SendAsync(request);
            content = await response.Content.ReadAsStringAsync();
            var dic = DocumentHelper.JsonToDic(content);

            if (dic.ContainsKey("msg"))
            {
                if (dic["msg"] != "登錄成功")//登陸過於頻繁，請稍等重試;errcode:100030
                {
                    Console.WriteLine(dic["msg"]);
                    return null;
                }
            }

3. 登陸成功後，後面就由你們隨便折騰了。這裏獲取登錄後的首頁信息吧。

//3.抓取首頁
            h.DefaultRequestHeaders.Clear();
            h.DefaultRequestHeaders.Add("UserAgent", Configs.ChromeAgent);
            h.DefaultRequestHeaders.Add("Accept-Language", "zh-CN,zh;q=0.8,en-US;q=0.6,en;q=0.4");
            h.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, sdch");
            h.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
            response = await h.GetAsync(index);
            content = await response.Content.ReadAsStringAsync();