網絡爬蟲之頁面解析

時間 2020-06-22

原文原文鏈接

<div class="output_wrapper" id="output_wrapper_id" style="font-size: 16px; color: rgb(62, 62, 62); line-height: 1.6; word-spacing: 0px; letter-spacing: 0px; font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif; background-image: linear-gradient(90deg, rgba(50, 0, 0, 0.05) 3%, rgba(0, 0, 0, 0) 3%), linear-gradient(360deg, rgba(50, 0, 0, 0.05) 3%, rgba(0, 0, 0, 0) 3%); background-size: 20px 20px; background-position: center center;"><figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://imgkr.cn-bj.ufileos.com/84359b07-75b5-4bdd-ab44-93f8161a3e53.png" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure> <blockquote style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; border-left-color: rgb(131, 175, 155); margin-top: 1.2em; margin-bottom: 1.2em; padding-right: 1em; padding-left: 1em; border-left-width: 6px; color: rgb(119, 119, 119); quotes: none; background: rgb(240, 240, 240);"> <p style="margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; text-align: start; white-space: normal; text-size-adjust: auto; font-size: 15px; font-family: -apple-system-font, BlinkMacSystemFont, 'Helvetica Neue', 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei UI', 'Microsoft YaHei', Arial, sans-serif; color: rgb(119, 119, 119); line-height: 1.75em;">做者：玩世不恭的Coder 時間：2020-03-13 說明：本文爲原創文章，未經容許不可轉載，轉載前請聯繫濤耶 </blockquote> <h1 id="h" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.5em 0px; font-weight: bold; margin-top: -0.46em; margin-bottom: 0.1em; border-bottom: 2px solid rgb(198, 196, 196); box-sizing: border-box;">網絡爬蟲之頁面解析</h1> <p class="toc" id="tocid_0" style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em; margin-left: 25px;"><a href="#h-1" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">前言</a><a href="#hbeautifulsoup" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">1、Beautiful Soup就該這樣使用</a><a href="#h-2" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">節點選擇</a><a href="#h-3" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">數據提取</a><a href="#hbeautifulsoup-1" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">Beautiful Soup小結</a><a href="#hxpath" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">2、XPath解析頁面</a><a href="#h-4" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">節點選擇</a><a href="#h-5" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">數據提取</a><a href="#hxpath-1" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">XPath小結</a><a href="#hpyquery" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">3、pyquery入門使用</a><a href="#h-6" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">節點選擇</a><a href="#h-7" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">數據提取</a><a href="#hpyquery-1" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">pyquery小結</a><a href="#h-8" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">4、騰訊招聘網解析實戰</a><a href="#h-9" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">網頁分析：</a><a href="#h-10" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">案例源碼</a><a href="#h-11" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">總結</a> <h2 id="h-1" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 10px auto; height: 40px; background-color: rgb(251, 251, 251); border-bottom: 1px solid rgb(246, 246, 246); overflow: hidden; box-sizing: border-box;"><span style="margin: 0px; padding: 0px; margin-left: -10px; display: inline-block; width: auto; height: 40px; background-color: rgb(33, 33, 34); border-bottom-right-radius: 100px; color: rgb(255, 255, 255); padding-right: 30px; padding-left: 30px; line-height: 40px; font-size: 16px;">前言</h2> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">With the rapid development of the Internet，愈來愈多的信息充斥着各大網絡平臺。正如<a href="https://baike.baidu.com/item/%E6%AD%BB%E4%BA%A1%E7%AC%94%E8%AE%B0/25476?fr=aladdin" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">《死亡筆記》</a>中L·Lawliet這一角色所提到的<a href="https://baike.baidu.com/item/大數定律/410082?fr=aladdin" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">大數定律</a>，在衆多繁雜的數據中必然存在着某種規律，偶然中必然包含着某種必然的發生。無論是咱們提到的大數定律，仍是最近火熱的大數據亦或其餘領域都離不開大量而又幹淨數據的支持，爲此，網絡爬蟲可以知足咱們的需求，即在互聯網上按照咱們的意願來爬取咱們任何想要獲得的信息，以便咱們分析出其中的必然規律，進而作出正確的決策。一樣，在咱們平時上網的過程當中，無時無刻可見爬蟲的影子，好比咱們廣爲熟知的「度娘」就是其中一個大型而又名副其實的「蜘蛛王」（SPIDER KING）。而要想寫出一個強大的爬蟲程序，則離不開熟練的對各類網絡頁面的解析，這篇文章將給讀者介紹如何在Python中使用各大主流的頁面解析工具。 <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">經常使用的解析方式主要有正則、Beautiful Soup、XPath、pyquery，本文主要是講解後三種工具的使用，而對正則表達式的使用這裏不作講解，對正則有興趣瞭解的讀者能夠跳轉：<a href="https://baike.baidu.com/item/%E6%AD%A3%E5%88%99%E8%A1%A8%E8%BE%BE%E5%BC%8F/1700215?fr=aladdin" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">正則表達式</a> <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.zh.html#contents-children" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">Beautiful Soup</a>的使用</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><a href="https://www.w3.org/TR/xpath/all/" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">XPath</a>的使用</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><a href="https://pythonhosted.org/pyquery/" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">pyquery</a>的使用</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">Beautiful Soup、XPath、pyquery解析<a href="https://hr.tencent.com/" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">騰訊招聘網</a>案例</li> </ul> <h2 id="hbeautifulsoup" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 10px auto; height: 40px; background-color: rgb(251, 251, 251); border-bottom: 1px solid rgb(246, 246, 246); overflow: hidden; box-sizing: border-box;"><span style="margin: 0px; padding: 0px; margin-left: -10px; display: inline-block; width: auto; height: 40px; background-color: rgb(33, 33, 34); border-bottom-right-radius: 100px; color: rgb(255, 255, 255); padding-right: 30px; padding-left: 30px; line-height: 40px; font-size: 16px;">1、Beautiful Soup就該這樣使用</h2> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">Beautiful Soup是Python爬蟲中針對HTML、XML的其中一個解析工具，熟練的使用之能夠很方便的提取頁面中咱們想要的數據。此外，在Beautiful Soup中，爲咱們提供瞭如下四種解析器： <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">標準庫，<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">soup = BeautifulSoup(content, "html.parser")</code></li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">lxml解析器，<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">soup = BeautifulSoup(content, "lxml")</code></li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">xml解析器，<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">soup = BeautifulSoup(content, "xml")</code></li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">html5lib解析器，<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">soup = BeautifulSoup(content, "html5lib")</code></li> </ul> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">在以上四種解析庫中，lxml解析具備解析速度快兼容錯能力強的merits，綜合考慮之下性能也較爲不錯，因此本文主要使用的是lxml解析器，下面咱們主要拿百度首頁的html來具體講解下Beautiful Soup應該如何使用，在此以前，咱們且看一下這段小爬蟲： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="python language-python hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">from bs4 import BeautifulSoup import requests if __name__ == "__main__":     response = requests.get("https://www.baidu.com")     encoding = response.apparent_encoding     response.encoding = encoding     print(BeautifulSoup(response.text, "lxml")) </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">代碼解讀： <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">首先咱們使用python中requests請求庫對百度首先進行get請求，而後獲取其編碼並將相應修改成對應的編碼格式以防止亂碼的可能性，最後經過BeautifluSoup對相應轉化爲其能解析的形式，使用的解析器是lxml。 <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">response = requests.get("https://www.baidu.com")</code>，requests請求百度連接</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">encoding = response.apparent_encoding</code>，獲取頁面編碼格式</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">response.encoding = encoding</code>，修改請求編碼爲頁面對應的編碼格式，以免亂碼</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">print(BeautifulSoup(response.text, "lxml"))</code>，使用lxml解析器來對百度首頁html進行解析並打印結果</li> </ul> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">打印後的結果以下所示： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="html language-html hljs xml" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;"><!DOCTYPE html> <html> <head><meta content="text/html;charset=utf-8" http-equiv="content-type"/><meta content="IE=Edge" http-equiv="X-UA-Compatible"/><meta content="always" name="referrer"/><link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/><title>百度一下，你就知道</title></head> <body link="#0000cc"> <div id="wrapper"> <div id="head"> <div class="head_wrapper"> <div class="s_form"> <div class="s_form_wrapper"> <div id="lg"> <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/>  # ... ...此處省略若干輸出 </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">從上述代碼中，咱們能夠看見打印出的內容過於雜亂無章，爲了使得解析後的頁面與咱們而言更加的清晰直觀，咱們可使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">prettify()</code>方法來對其進行標準的縮進操做，爲了方便講解，濤耶對結果進行適當的刪除，只留下有價值的內容，源碼及輸出以下： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="python language-python hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">bd_soup = BeautifulSoup(response.text, "lxml") print(bd_soup.prettify()) </code></pre> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="html language-html hljs xml" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;"><html>  <head>   <title>    百度一下，你就知道   </title>  </head>  <body link="#0000cc">   <div id="wrapper">    <div id="head">     <div class="head_wrapper">      <div class="s_form">       <div class="s_form_wrapper">        <div id="lg">         <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/>        </div>       </div>      </div>      # ... ...此處省略若干輸出    </div>   </div>  </body> </html> </code></pre> <h3 id="h-2" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">節點選擇</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">在Beautiful Soup中，咱們能夠很方便的選擇想要獲得的節點，只須要在<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup</code>對象中使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.</code>的方式便可得到想要的對象，其內置api豐富，徹底足夠咱們實際的使用，其使用規則以下： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="python language-python hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">bd_title_bj = bd_soup.title         # 獲取html中的title內容 bd_title_bj_name = bd_soup.title.name   # .name獲取對應節點的名稱 bd_title_name = bd_soup.title.string     bd_title_parent_bj_name = bd_soup.title.parent.name # .parent獲取父節點 bd_image_bj = bd_soup.img   # .img獲取img節點 bd_image_bj_dic = bd_soup.img.attrs # .attrs獲取屬性值 bd_image_all = bd_soup.find_all("img")  # 經過find_all查找對應的指定的全部節點 bd_image_idlg = bd_soup.find("div", id="lg")     # 經過class屬性查找節點 </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">上述代碼中解析的結果對應打印以下，你們對應輸出理解其意便可： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs coffeescript" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;"><title>百度一下，你就知道</title> title 百度一下，你就知道 head <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> {'hidefocus': 'true', 'src': '//www.baidu.com/img/bd_logo1.png', 'width': '270', 'height': '129'} [<img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/>, <img src="//www.baidu.com/img/gs.gif"/>] <div id="lg"> <img height="129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> </div> </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">代碼解讀： <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.title</code>，正如前面所說，Beautiful Soup能夠很簡單的解析對應的頁面，只須要使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.</code>的方式進行選擇節點便可，該行代碼正是得到百度首頁html的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">title</code>節點內容</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.title.name</code>，使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.name</code>的形式便可獲取節點的名稱</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.title.string</code>，使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.string</code>的形式便可得到節點當中的內容，這句代碼就是獲取百度首頁的title節點的內容，即瀏覽器導航條中所顯示的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">百度一下，你就知道</code></li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.title.parent.name</code>，使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.parent</code>能夠該節點的父節點，通俗地講就是該節點所對應的上一層節點，而後使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.name</code>獲取父節點名稱</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.img</code>，如<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.title</code>同樣，該代碼獲取的是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點，只不過須要注意的是：在上面html中咱們能夠看見總共有兩個<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點，而若是使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.img</code>的話默認是獲取html中的第一個<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點，而不是全部</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.img.attrs</code>，獲取<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點中全部的屬性及屬性內容，該代碼輸出的結果是一個鍵值對的字典格式，因此以後咱們只須要經過字典的操做來獲取屬性所對應的內容便可。好比<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.img.attrs.get("src")</code>和<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.img.attrs["src"]</code>的方式來獲取<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點所對應的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">src</code>屬性的內容，即圖片連接</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.find_all("img")</code>，在上述中的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.img</code>操做默認只能獲取第一個<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點，而要想獲取html中全部的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點，咱們須要使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.find_all("img")</code>方法，所返回的是一個列表格式，列表內容爲全部的選擇的節點</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup.find("div", id="lg")</code>，在實際運用中，咱們每每會選擇指定的節點，這個時候咱們可使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.find()</code>方法，裏面可傳入所需查找節點的屬性，這裏須要注意的是：在傳入<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">class</code>屬性的時候其中的寫法是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.find("div", class_="XXX")</code>的方式。因此該行代碼表示的是獲取<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">id</code>屬性爲<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">lg</code>的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">div</code>節點，此外，在上面的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.find_all()</code>一樣可使用該方法來獲取指定屬性所對應的全部節點</li> </ul> <h3 id="h-3" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">數據提取</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">在上一小節節點選擇咱們講到了部分數據提取的方法，然而，Beautiful Soup的強大之處還不止步於此。接下來咱們繼續揭開其神祕的面紗。（注意：如下只是進行經常使用的一些api，如如有更高的需求可查看官方文檔） <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">.get_text()</li> </ul> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">獲取對象中全部的文本內容（即咱們在頁面中肉眼可見的文本）： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs ini" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">all_content = bd_soup.get_text() </code></pre> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs javascript" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;"> 百度一下，你就知道                     新聞 hao123 地圖 視頻 貼吧  登陸  document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登陸</a>');                  更多產品       關於百度 About Baidu  ©2017 Baidu 使用百度前必讀  意見反饋 京ICP證030173號  </code></pre> <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">.strings，.stripped_strings</li> </ul> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs bash" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">print(type(bd_soup.strings)) # <class 'generator'> </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.strings</code>用於提取<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_soup</code>對象中全部的內容，而從上面的輸出結果咱們能夠看出<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.strings</code>的類型是一個生成器，對此可使用循環來提取出其中的內容。可是咱們在使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.strings</code>的過程當中會發現提取出來的內容有不少的空格以及換行，對此咱們可使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.stripped_strings</code>方法來解決該問題，用法以下： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs perl" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">for each in bd_soup.stripped_strings:     print(each) </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">輸出結果： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">百度一下，你就知道 新聞 hao123 地圖 視頻 貼吧 登陸 更多產品 關於百度 About Baidu ©2017 Baidu 使用百度前必讀 意見反饋 京ICP證030173號 </code></pre> <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">.parent，.children，.parents</li> </ul> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.parent</code>能夠選擇該節點的父節點，<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.children</code>能夠選擇該節點的孩子節點，<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.parents</code>選擇該節點全部的上層節點，返回的是生成器類型，各用法以下： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs lua" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">bd_div_bj = bd_soup.find("div", id="u1") print(type(bd_div_bj.parent)) print("*" * 50) for child in bd_div_bj.children:     print(child) print("*" * 50) for parent in bd_div_bj.parents:     print(parent.name) </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">結果輸出： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs xml" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;"><class 'bs4.element.Tag'> ************************************************** <a class="mnav" href="http://news.baidu.com" name="tj_trnews">新聞</a> <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a> <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地圖</a> <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">視頻</a> <a class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">貼吧</a> ************************************************** div div div body html </code></pre> <h3 id="hbeautifulsoup-1" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">Beautiful Soup小結</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">Beautiful Soup主要的用法就是以上一些，還有其餘一些操做在實際開發過程當中使用的並非不少，這裏不作過多的講解了，因此總體來說Beautiful Soup的使用仍是比較簡單的，其餘一些操做可見官方文檔：<a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.zh.html#contents-children" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.zh.html#contents-children</a> <h2 id="hxpath" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 10px auto; height: 40px; background-color: rgb(251, 251, 251); border-bottom: 1px solid rgb(246, 246, 246); overflow: hidden; box-sizing: border-box;"><span style="margin: 0px; padding: 0px; margin-left: -10px; display: inline-block; width: auto; height: 40px; background-color: rgb(33, 33, 34); border-bottom-right-radius: 100px; color: rgb(255, 255, 255); padding-right: 30px; padding-left: 30px; line-height: 40px; font-size: 16px;">2、XPath解析頁面</h2> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">XPath全稱是XML Path Language，它既能夠用來解析XML，也能夠用來解析HTML。在上一部分已經講解了Beautiful Soup的一些常見的騷操做，在這裏，咱們繼續來看看XPath的使用，瞧一瞧XPath的功能到底有多麼的強大以至於受到了很多開發者的青睞。同Beautiful Soup同樣，在XPath中提供了很是簡潔的節點選擇的方法，Beautiful Soup主要是經過<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.</code>的方式來進行子節點或者子孫節點的選擇，而在XPath中則主要經過<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">/</code>的方式來選擇節點。除此以外，在XPath中還提供了大量的內置函數來處理各個數據之間的匹配關係。 <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">首先，咱們先來看看XPath常見的節點匹配規則： <table style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; display: table; width: 100%; text-align: left;"> <thead style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"> <tr style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; border-width: 1px 0px 0px; border-right-style: initial; border-bottom-style: initial; border-left-style: initial; border-right-color: initial; border-bottom-color: initial; border-left-color: initial; border-image: initial; border-top-style: solid; border-top-color: rgb(204, 204, 204); background-color: white;"> <th style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; font-weight: bold; background-color: rgb(240, 240, 240); text-align: center;">表達式</th> <th style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; font-weight: bold; background-color: rgb(240, 240, 240); text-align: center;">解釋說明</th> </tr> </thead> <tbody style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; border: 0px;"> <tr style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; border-width: 1px 0px 0px; border-right-style: initial; border-bottom-style: initial; border-left-style: initial; border-right-color: initial; border-bottom-color: initial; border-left-color: initial; border-image: initial; border-top-style: solid; border-top-color: rgb(204, 204, 204); background-color: white;"> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">/</code></td> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;">在當前節點中選取直接子節點</td> </tr> <tr style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; border-width: 1px 0px 0px; border-right-style: initial; border-bottom-style: initial; border-left-style: initial; border-right-color: initial; border-bottom-color: initial; border-left-color: initial; border-image: initial; border-top-style: solid; border-top-color: rgb(204, 204, 204); background-color: white;"> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">//</code></td> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;">在當前節點中選取子孫節點</td> </tr> <tr style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; border-width: 1px 0px 0px; border-right-style: initial; border-bottom-style: initial; border-left-style: initial; border-right-color: initial; border-bottom-color: initial; border-left-color: initial; border-image: initial; border-top-style: solid; border-top-color: rgb(204, 204, 204); background-color: white;"> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.</code></td> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;">選取當前節點</td> </tr> <tr style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; border-width: 1px 0px 0px; border-right-style: initial; border-bottom-style: initial; border-left-style: initial; border-right-color: initial; border-bottom-color: initial; border-left-color: initial; border-image: initial; border-top-style: solid; border-top-color: rgb(204, 204, 204); background-color: white;"> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">..</code></td> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;">選取當前節點的父節點</td> </tr> <tr style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; border-width: 1px 0px 0px; border-right-style: initial; border-bottom-style: initial; border-left-style: initial; border-right-color: initial; border-bottom-color: initial; border-left-color: initial; border-image: initial; border-top-style: solid; border-top-color: rgb(204, 204, 204); background-color: white;"> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">@</code></td> <td style="color: inherit; line-height: inherit; margin: 0px; font-size: 1em; border: 1px solid rgb(204, 204, 204); padding: 0.5em 1em; text-align: center;">指定屬性（id、class……）</td> </tr> </tbody> </table> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">下面咱們繼續拿上面的百度首頁的HTML來說解下XPath的使用。 <h3 id="h-4" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">節點選擇</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">要想正常使用Xpath，咱們首先須要正確導入對應的模塊，在此咱們通常使用的是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">lxml</code>，操做示例以下： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs coffeescript" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">from lxml import etree import requests import html if __name__ == "__main__":     response = requests.get("https://www.baidu.com")     encoding = response.apparent_encoding     response.encoding = encoding     print(response.text)     bd_bj = etree.HTML(response.text)     bd_html = etree.tostring(bd_bj).decode("utf-8")     print(html.unescape(bd_html)) </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">1~9行代碼如Beautiful Soup一致，下面對以後的代碼進行解釋： <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">etree.HTML(response.text)</code>，使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">etree</code>模塊中的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">HTML</code>類來對百度html<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">(response.text)</code>進行初始化以構造XPath解析對象，返回的類型爲<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);"><element html="" at="" 0x1aba86b1c08="" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"></element></code></li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">etree.tostring(bd_html_elem).decode("utf-8")</code>，將上述的對象轉化爲字符串類型且編碼爲<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">utf-8</code></li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">html.unescape(bd_html)</code>，使用HTML5標準定義的規則將<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_html</code>轉換成對應的unicode字符。</li> </ul> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">打印出的結果如Beautiful Soup使用時一致，這裏就再也不顯示了，不知道的讀者可回翻。既然咱們已經獲得了Xpath可解析的對象<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">(bd_bj)</code>，下面咱們就須要針對這個對象來選擇節點了，在上面咱們也已經提到了，XPath主要是經過<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">/</code>的方式來提取節點，請看下面Xpath中節點選擇的一些常見操做： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs makefile" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">all_bj = bd_bj.xpath("//*")             # 選取全部節點 img_bj = bd_bj.xpath("//img")           # 選取指定名稱的節點 p_a_zj_bj = bd_bj.xpath("//p/a")        # 選取直接節點 p_a_all_bj = bd_bj.xpath("//p//a")      # 選取全部節點 head_bj = bd_bj.xpath("//title/..")     # 選取父節點 </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">結果以下： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs json" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">[<Element html at 0x14d6a6d1c88>, <Element head at 0x14d6a6e4408>, <Element meta at 0x14d6a6e4448>, <Element meta at 0x14d6a6e4488>, <Element meta at 0x14d6a6e44c8>, <Element link at 0x14d6a6e4548>, <Element title at 0x14d6a6e4588>, <Element body at 0x14d6a6e45c8>, <Element div at 0x14d6a6e4608>, <Element div at 0x14d6a6e4508>, <Element div at 0x14d6a6e4648>, <Element div at 0x14d6a6e4688>, ......] [<Element img at 0x14d6a6e4748>, <Element img at 0x14d6a6e4ec8>] [<Element a at 0x14d6a6e4d88>, <Element a at 0x14d6a6e4dc8>, <Element a at 0x14d6a6e4e48>, <Element a at 0x14d6a6e4e88>] [<Element a at 0x14d6a6e4d88>, <Element a at 0x14d6a6e4dc8>, <Element a at 0x14d6a6e4e48>, <Element a at 0x14d6a6e4e88>] [<Element head at 0x14d6a6e4408>] </code></pre> <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">all_bj = bd_bj.xpath("//*")</code>，使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">//</code>能夠選擇當前節點<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">(html)</code>下的全部子孫節點，且以一個列表的形式來返回，列表元素經過<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_bj</code>同樣是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">element</code>對象，下面的返回類型一致</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img_bj = bd_bj.xpath("//img")</code>，選取當前節點下指定名稱的節點，這裏建議與Beautiful Soup的使用相比較可加強記憶，Beautiful Soup是經過<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.find_all("img")</code>的形式</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">p_a_zj_bj = bd_bj.xpath("//p/a")</code>，選取當前節點下的全部<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">p</code>節點下的直接子<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">a</code>節點，這裏須要注意的是」直接「，若是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">a</code>不是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">p</code>節點的直接子節點則選取失敗</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">p_a_all_bj = bd_bj.xpath("//p//a")</code> ，選取當前節點下的全部<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">p</code>節點下的全部子孫<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">a</code>節點，這裏須要注意的是」全部「，注意與上一個操做進行區分</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">head_bj = bd_bj.xpath("//title/..")</code>，選取當前節點下的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">title</code>節點的父節點，即<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">head</code>節點</li> </ul> <h3 id="h-5" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">數據提取</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">在瞭解如何選擇指定的節點以後，咱們就須要提取節點中所包含的數據了，具體提取請看下面的示例： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs ini" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">img_href_ls = bd_bj.xpath("//img/@src") img_href = bd_bj.xpath("//div[@id='lg']/img[@hidefocus='true']/@src") a_content_ls = bd_bj.xpath("//a//text()") a_news_content = bd_bj.xpath("//a[@class='mnav' and @name='tj_trnews']/text()") </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">輸出結果： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs json" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">['//www.baidu.com/img/bd_logo1.png', '//www.baidu.com/img/gs.gif'] ['//www.baidu.com/img/bd_logo1.png'] ['新聞', 'hao123', '地圖', '視頻', '貼吧', '登陸', '更多產品', '關於百度', 'About Baidu', '使用百度前必讀', '意見反饋'] ['新聞'] </code></pre> <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img_href_ls = bd_bj.xpath("//img/@src")</code>，該代碼先選取了當前節點下的全部<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點，而後將全部<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">src</code>屬性值選取出來，返回的是一個列表形式</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img_href = bd_bj.xpath("//div[@id='lg']/img[@hidefocus='true']/@src")</code>，該代碼首先選取了當前節點下全部<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">id</code>屬性值爲<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">lg</code>的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">div</code>，而後繼續選取<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">div</code>節點下的直接子<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">img</code>節點（<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">hidefoucus=true</code>），最後選取其中的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">src</code>屬性值</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">a_content_ls = bd_bj.xpath("//a//text()")</code>，選取當前節點全部的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">a</code>節點的所遇文本內容</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;"><code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">a_news_content = bd_bj.xpath("//a[@class='mnav' and @name='tj_trnews']/text()")</code>，多屬性選擇，在xpath中能夠指定知足多個屬性的節點，只須要<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">and</code>便可</li> </ul> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">Tips：讀者在閱讀的過程當中注意將代碼和輸出的結果仔細對應起來，只要理解其中的意思在加上適當的訓練也就不難記憶了。 <h3 id="hxpath-1" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">XPath小結</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">耐心看完了XPath的使用方法以後，聰明的讀者應該不難發現，其實Beautiful Soup和XPath的本質和思路上基本相同，只要咱們在閱讀XPath用法的同時在腦殼中不斷的思考，相信聰明的你閱讀至此已經可以基本掌握了XPath用法。 <h2 id="hpyquery" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 10px auto; height: 40px; background-color: rgb(251, 251, 251); border-bottom: 1px solid rgb(246, 246, 246); overflow: hidden; box-sizing: border-box;"><span style="margin: 0px; padding: 0px; margin-left: -10px; display: inline-block; width: auto; height: 40px; background-color: rgb(33, 33, 34); border-bottom-right-radius: 100px; color: rgb(255, 255, 255); padding-right: 30px; padding-left: 30px; line-height: 40px; font-size: 16px;">3、pyquery入門使用</h2> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">對於pyquery，官方的解釋以下： <blockquote style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; border-left-color: rgb(131, 175, 155); margin-top: 1.2em; margin-bottom: 1.2em; padding-right: 1em; padding-left: 1em; border-left-width: 6px; color: rgb(119, 119, 119); quotes: none; background: rgb(240, 240, 240);"> <p style="margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; text-align: start; white-space: normal; text-size-adjust: auto; font-size: 15px; font-family: -apple-system-font, BlinkMacSystemFont, 'Helvetica Neue', 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei UI', 'Microsoft YaHei', Arial, sans-serif; color: rgb(119, 119, 119); line-height: 1.75em;">pyquery allows you to make jquery queries on xml documents. The API is as much as possible the similar to jquery. pyquery uses lxml for fast xml and html manipulation. This is not (or at least not yet) a library to produce or interact with javascript code. I just liked the jquery API and I missed it in python so I told myself 「Hey let’s make jquery in python」. This is the result. It can be used for many purposes, one idea that I might try in the future is to use it for templating with pure http templates that you modify using pyquery. I can also be used for web scrapping or for theming applications with Deliverance. The project is being actively developped on a git repository on Github. I have the policy of giving push access to anyone who wants it and then to review what he does. So if you want to contribute just email me. Please report bugs on the github issue tracker. </blockquote> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">在網頁解析過程當中，除了強大的Beautiful Soup和XPath以外，還有qyquery的存在，qyquery一樣受到了很多「蜘蛛er」的歡迎，下面咱們來介紹下qyquery的使用。 <h3 id="h-6" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">節點選擇</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">與Beautiful Soup和XPath明顯不一樣的是，在qyquery中，通常存在着三種解析方式，一種是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">requests</code>請求連接以後把html進行傳遞，一種是將<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">url</code>直接進行傳遞，還有一種是直接傳遞本地<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">html</code>文件路徑便可，讀者在實際使用的過程當中根據本身的習慣來編碼便可，下面咱們來看下這三種方式的表達： <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs perl" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">import requests from pyquery import PyQuery as pq bd_html = requests.get("https://www.baidu.com").text bd_url = "https://www.baidu.com" bd_path = "./bd.html" # 使用html參數進行傳遞 def way1(html):     return pq(html) # 使用url參數進行傳遞 def way2(url):     return pq(url=url) def way3(path):     return pq(filename=path) print(type(way1(html=bd_html))) print(type(way2(url=bd_url))) print(type(way3(path=bd_path))) # <class 'pyquery.pyquery.PyQuery'> # <class 'pyquery.pyquery.PyQuery'> # <class 'pyquery.pyquery.PyQuery'> </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">從上面三種得到解析對象方法的代碼中咱們能夠明顯看見均可以獲得同樣的解析對象，接下來咱們只要利用這個對象來對頁面進行解析從而提取出咱們想要獲得的有效信息便可，在qyquery中通常使用的是CSS選擇器來選取。下面咱們仍然使用百度首頁來說解pyquery的使用，在這裏咱們假設解析對象爲<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_bj</code>。 <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs makefile" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">response = requests.get("https://www.baidu.com") response.encoding = "utf-8" bd_bj = pq(response.text) bd_title = bd_bj("title") bd_img_ls = bd_bj("img") bd_img_ls2 = bd_bj.find("img") bd_mnav = bd_bj(".mnav") bd_img = bd_bj("#u1 a") bd_a_video = bd_bj("#u1 .mnav") # <title>百度一下，你就知道</title> # <img hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270" height="129"/> <img src="//www.baidu.com/img/gs.gif"/>  # ...... # 輸出結果較長，讀者可自行運行 </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">正如上面代碼所示，pyquery在進行節點提取的時候一般有三種方式，一種是直接提取出節點名便可提取出整個節點，固然這種方式你也可使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">find</code>方法，這種提取節點的方式是不加任何屬性限定的，因此提取出的節點每每會含有多個，因此咱們可使用循環<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.items()</code>來進行操做；一種是提取出含有特定<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">class</code>屬性的節點，這種形式採用的是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.+class屬性值</code>；還有一種是提取含有特定<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">id</code>屬性的節點，這種形式採用的是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">#+id屬性值</code>。熟悉<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">CSS</code>的讀者應該不難理解以上提取節點的方法，正是在<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">CSS</code>中提取節點而後對其進行樣式操做的方法。上述三種方式您也能夠像提取<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">bd_a_video</code>同樣混合使用 <h3 id="h-7" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">數據提取</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">在實際解析網頁的過程當中，三種解析方式基本上大同小異，爲了讀者認識<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">pyquery</code>的數據提取的操做以及博主往後的查閱，在這裏簡單的介紹下 <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="hljs vbnet" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">img_src1 = bd_bj("img").attr("src") # //www.baidu.com/img/bd_logo1.png img_src2 = bd_bj("img").attr.src    # //www.baidu.com/img/bd_logo1.png for each in bd_bj.find("img").items():     print(each.attr("src")) print(bd_bj("title").text())    # 百度一下，你就知道 </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">如上一二行代碼所示，提取節點屬性咱們能夠有兩種方式，這裏拿<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">src</code>屬性來進行說明，一種是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.attr("src")</code>，另一種是<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.attr.src</code>，讀者根據本身的習慣來操做便可，這裏須要注意的是：在節點提取小結中咱們說了在不限制屬性的狀況下是提取出全部知足條件的節點，因此在這種狀況下提取出的屬性是第一個節點屬性。要想提取全部的節點的屬性，咱們能夠如四五行代碼那樣使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.items()</code>而後進行遍歷，最後和以前同樣提取各個節點屬性便可。<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">qyquery</code>提取節點中文本內容如第七行代碼那樣直接使用<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">.text()</code>便可。 <h3 id="hpyquery-1" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">pyquery小結</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">pyquery解析如Beautiful Soup和XPath思想一致，因此這了只是簡單的介紹了下，想要進一步瞭解的讀者可查閱官方文檔在加之熟練操做便可。 <h2 id="h-8" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 10px auto; height: 40px; background-color: rgb(251, 251, 251); border-bottom: 1px solid rgb(246, 246, 246); overflow: hidden; box-sizing: border-box;"><span style="margin: 0px; padding: 0px; margin-left: -10px; display: inline-block; width: auto; height: 40px; background-color: rgb(33, 33, 34); border-bottom-right-radius: 100px; color: rgb(255, 255, 255); padding-right: 30px; padding-left: 30px; line-height: 40px; font-size: 16px;">4、騰訊招聘網解析實戰</h2> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">經過上述對Beautiful Soup、XPath以及pyquery的介紹，認真閱讀過的讀者想必已經有了必定的基礎，下面咱們經過一個簡單的實戰案例來強化一下三種解析方式的操做。這次解析的網站爲騰訊招聘網，網址url：<a href="https://hr.tencent.com/" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">https://hr.tencent.com/</a>，其社會招聘網首頁以下所示： <img src="https://gss0.baidu.com/-vo3dSag_xI4khGko9WTAnF6hhy/zhidao/pic/item/aa64034f78f0f7369adfccd70755b319eac413c6.jpg" width="500" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">這次咱們的任務就是分別利用上述三種解析工具來接下該網站下的社會招聘中的全部數據。 <h3 id="h-9" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">網頁分析：</h3> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">經過該網站的社會招聘的首頁，咱們能夠發現以下三條主要信息： <ul style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; padding-left: 32px; list-style-type: disc;"> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">首頁url鏈接爲<a href="https://hr.tencent.com/position.php" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">https://hr.tencent.com/position.php</a></li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">一共有288頁的數據，每頁10個職位，總職位共計2871</li> <li style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; margin-bottom: 0.5em;">數據字段有五個，分別爲：職位名稱、職位類別、招聘人數、工做地點、職位發佈時間</li> </ul> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">既然咱們解析的是該網站下全部職位數據，再者咱們停留在第一頁也沒有發現其餘有價值的信息，不如進入第二頁看看，這時咱們能夠發現網站的url連接有了一個比較明顯的變化，即原連接在用戶端提交了一個<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">start</code>參數，此時連接爲<a href="https://hr.tencent.com/position.php?&start=10#a" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration: none; color: rgb(30, 107, 184); overflow-wrap: break-word;">https://hr.tencent.com/position.php?&start=10#a</a>，陸續打開後面的頁面咱們不難發現其規律：每一頁提交的<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">start</code>參數以10位公差進行逐步遞增。以後，咱們使用谷歌開發者工具來審查該網頁，咱們能夠發現全站皆爲靜態頁面，這位咱們解析省下了很多麻煩，咱們須要的數據就靜態的放置在<code style="font-size: inherit; line-height: inherit; overflow-wrap: break-word; padding: 2px 4px; border-radius: 4px; margin: 0px 2px; color: rgb(233, 105, 0); background: rgb(248, 248, 248);">table</code>標籤內，以下所示： <img src="https://gss0.baidu.com/94o3dSag_xI4khGko9WTAnF6hhy/zhidao/pic/item/e850352ac65c1038f3b24f14bf119313b17e891c.jpg" width="500" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">下面咱們具體來分別使用以上三種工具來解析該站全部職位數據。 <h3 id="h-10" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 20px auto 5px; border-top: 1px solid rgb(221, 221, 221); box-sizing: border-box;"><span style="color: inherit; margin: 0px; padding: 0px; margin-top: -1px; padding-top: 6px; padding-right: 5px; padding-left: 5px; font-size: 17px; border-top: 2px solid rgb(33, 33, 34); display: inline-block; line-height: 1.1;">案例源碼</h3> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="python language-python hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important; display: -webkit-box !important;">import requests from bs4 import BeautifulSoup from lxml import etree from pyquery import PyQuery as pq import itertools import pandas as pd class TencentPosition():     """     功能： 定義初始變量     參數：         start： 起始數據     """     def __init__(self, start):         self.url = "https://hr.tencent.com/position.php?&start={}#a".format(start)         self.headers = {             "Host": "hr.tencent.com",             "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36",         }         self.file_path = "./TencentPosition.csv"     """     功能： 請求目標頁面     參數：         url： 目標連接         headers： 請求頭     返回：         html，頁面源碼     """     def get_page(self, url, headers):          res = requests.get(url, headers=headers)         try:             if res.status_code == 200:                 return res.text             else:                 return self.get_page(url, headers=headers)         except RequestException as e:             return self.get_page(url, headers=headers)     """     功能： Beautiful Soup解析頁面     參數：         html： 請求頁面源碼     """     def soup_analysis(self, html):         soup = BeautifulSoup(html, "lxml")         tr_list = soup.find("table", class_="tablelist").find_all("tr")         for tr in tr_list[1:-1]:             position_info = [td_data for td_data in tr.stripped_strings]             self.settle_data(position_info=position_info)     """     功能： xpath解析頁面     參數：         html： 請求頁面源碼     """     def xpath_analysis(self, html):         result = etree.HTML(html)         tr_list = result.xpath("//table[@class='tablelist']//tr")         for tr in tr_list[1:-1]:             position_info = tr.xpath("./td//text()")             self.settle_data(position_info=position_info)     """     功能： pyquery解析頁面     參數：         html： 請求頁面源碼     """     def pyquery_analysis(self, html):         result = pq(html)         tr_list = result.find(".tablelist").find("tr")         for tr in itertools.islice(tr_list.items(), 1, 11):             position_info = [td.text() for td in tr.find("td").items()]             self.settle_data(position_info=position_info)     """     功能： 職位數據整合     參數：         position_info： 字段數據列表     """     def settle_data(self, position_info):         position_data = {                 "職位名稱": position_info[0].replace("\xa0", " "),  # replace替換\xa0字符防止轉碼error                 "職位類別": position_info[1],                 "招聘人數": position_info[2],                 "工做地點": position_info[3],                 "發佈時間": position_info[-1],             }         print(position_data)         self.save_data(self.file_path, position_data)     """     功能： 數據保存     參數：         file_path： 文件保存路徑         position_data： 職位數據     """     def save_data(self, file_path, position_data):         df = pd.DataFrame([position_data])         try:             df.to_csv(file_path, header=False, index=False, mode="a+", encoding="gbk")  # 數據轉碼並換行存儲         except:             pass if __name__ == "__main__":     for page, index in enumerate(range(287)):         print("正在爬取第{}頁的職位數據:".format(page+1))         tp = TencentPosition(start=(index*10))         tp_html = tp.get_page(url=tp.url, headers=tp.headers)         tp.pyquery_analysis(html=tp_html)         print("\n") </code></pre> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">部分結果以下： <img src="https://gss0.baidu.com/94o3dSag_xI4khGko9WTAnF6hhy/zhidao/pic/item/d4628535e5dde7115832c220aaefce1b9c166100.jpg" width="500" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"> <h2 id="h-11" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; font-weight: bold; margin: 10px auto; height: 40px; background-color: rgb(251, 251, 251); border-bottom: 1px solid rgb(246, 246, 246); overflow: hidden; box-sizing: border-box;"><span style="margin: 0px; padding: 0px; margin-left: -10px; display: inline-block; width: auto; height: 40px; background-color: rgb(33, 33, 34); border-bottom-right-radius: 100px; color: rgb(255, 255, 255); padding-right: 30px; padding-left: 30px; line-height: 40px; font-size: 16px;">總結</h2> <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">在本篇文章中，首先咱們分別介紹了Beautiful Soup、XPath、pyquery的常見操做，以後經過使用該三種解析工具來爬取騰訊招聘網中全部的職位招聘數據，從而進一步讓讀者有一個更加深入的認識。該案例中，因爲本篇文章重點在於網站頁面的解析方法，因此未使用多線程、多進程，爬取全部的數據爬取的時間在一兩分鐘，在以後的文章中有時間的話會再次介紹多線程多進程的使用，案例中的解析方式都已介紹過，因此讀者閱讀源碼便可，若是對爬蟲頁面解析還有疑問的朋友歡迎在聯繫濤耶或者在下方留言，公衆號：【玩世不恭的Coder】，一個專一技術的分享區。 <p style="color: inherit; margin: 0px; padding: 0px; box-sizing: border-box; margin-bottom: 16px; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 15px; text-align: start; white-space: normal; text-size-adjust: auto; line-height: 1.75em;">注意：本文章中全部的內容皆爲在實際開發中常見的一些操做，並不是全部，想要進一步提高技術能力的讀者務必請閱讀官方文檔。 <figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="https://imgkr.cn-bj.ufileos.com/74117058-06b5-447a-a790-fdb6b259c6b9.png" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure></div>javascript

原文出處：https://www.cnblogs.com/LiT-26647879-510087153/p/12487958.htmlphp