網絡爬蟲_BeautifulSoup庫入門

時間 2019-11-16

原文原文鏈接

1、Beautiful Soup庫的基本元素

1.Beautiful Soup庫的理解

　　　　　　　　　 　　<p>..</p>: 標籤：Tag
			<p> p指的是標籤的名字， Name成對出現
			<p class="title">...</p>
			屬性Attributes0個或多個（由鍵值對構成）

2.Beautiful Soup庫的引用

            引用方式
			from bs4 import BeautifulSoup
			
			使用方法
			from bs4 import BeautifulSoup
			1. soup = BeautifulSoup("<html>data<html>", "html.parser")
			2. soup = BeautifulSoup(open("D://demo.html"), "html.parser")
			BeautifulSoup對應一個HTML/XML文檔的所有內容。

4.Beautiful Soup庫解析器

soup = BeautifulSoup('<html>data</html>'，'html.parser')

分類　　html

			解析器 					使用方法				 		    	  條件
			bs4的HTML解析器 		BeautifulSoup(mk,'html.parser')	 	        安裝bs4庫
			lxml的HTML解析器 		BeautifulSoup(mk,'lxml') 					pip install lxml
			lxml的XML解析器 		BeautifulSoup(mk,'xml') 					pip install lxml
			html5lib的解析器 		BeautifulSoup(mk,'html5lib')　              pip install html5lib

5.Beautiful Soup類基本元素

<p class=「title」> … </p>

分類　　html5

    基本元素 			說明
	Tag		        標籤，最基本的信息組織單元，分別用<>和</>標明開頭和結尾
	Name 		    標籤的名字，<p>…</p>的名字是'p'，格式：<tag>.name
	Attributes 	    標籤的屬性，字典形式組織，格式：<tag>.attrs
	NavigableString 標籤內非屬性字符串，<>…</>中字符串，格式：<tag>.string
	Comment 	    標籤內字符串的註釋部分，一種特殊的Comment類型

6.具體使用代碼

1. 回顧demo.htmlpython

	import requests
	url2 = "http://python123.io/ws/demo.html"
	r = requests.get(url2)
	demo = r.text
	print(demo)
	打印輸出
		# print(demo)
			<html><head><title>This is a python demo page</title></head>
			<body>
			<p class="title"><b>The demo python introduces several python courses.</b></p>
			<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
			<a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p>
			</body></html>

2. Tag 標籤ide

Tag 標籤，最基本的信息組織單元，分別用<>和</>標明開頭和結尾
任何存在於HTML語法中的標籤均可以用soup.<tag>訪問得到
當HTML文檔中存在多個相同<tag>對應內容時，soup.<tag>返回第一個
編碼

from bs4 import BeautifulSoup
import requests

url = "http://python123.io/ws/demo.html"
r = requests.get(url)
demo = r.text
soup = BeautifulSoup(demo, "html.parser")
print(soup.title)
tag = soup.a
print(tag)
	打印輸出
		# print(soup.title)
			<title>This is a python demo page</title>
		# print(tag)
			<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>

3. Tag的name（名字）url

Name 標籤的名字，<p>…</p>的名字是'p'，格式：<tag>.name
每一個<tag>都有本身的名字，經過<tag>.name獲取，字符串類型spa

from bs4 import BeautifulSoup
		import requests
		url2 = "http://python123.io/ws/demo.html"
		r = requests.get(url2)
		demo = r.text
		soup = BeautifulSoup(demo, "html.parser")
		print(soup.a.name)
		print(soup.a.parent.name)
		print(soup.a.parent.parent.name)
	
	打印輸出
		#	print(soup.a.name)
				'a'
		# print(soup.a.parent.name)
				'p'
		# print(soup.a.parent.parent.name)
				'body'

4. Tag的attrs（屬性）code

Attributes 標籤的屬性，字典形式組織，格式：<tag>.attrs
一個<tag>能夠有0或多個屬性，字典類型xml

from bs4 import BeautifulSoup
import requests
url2 = "http://python123.io/ws/demo.html"
r = requests.get(url2)
demo = r.text
soup = BeautifulSoup(demo, "html.parser")
tag = soup.a
print(tag.attrs)
print(tag.attrs['class'])
print(tag.attrs['href'])
print(type(tag.attrs))
print(type(tag))
打印輸出
			# print(tag.attrs)
				{'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}
			# print(tag.attrs['class'])
				['py1']
			#	print(tag.attrs['href'])
				http://www.icourse163.org/course/BIT-268001
			# print(type(tag.attrs))
				<class 'dict'>
		  # print(type(tag))
				<class 'bs4.element.Tag'>

5. Tag的NavigableString　htm

NavigableString 標籤內非屬性字符串，<>…</>中字符串，格式：<tag>.string
NavigableString能夠跨越多個層次

from bs4 import BeautifulSoup
import requests
url2 = "http://python123.io/ws/demo.html"
r = requests.get(url2)
demo = r.text
soup = BeautifulSoup(demo, "html.parser")
soup.a
print(soup.a.string)
print(soup.p)
print(soup.p.string)
print(type(soup.p.string))
打印輸出
		#	print(soup.a.string)
			Basic Python
		  # print(soup.p)
		  	<p class="title"><b>The demo python introduces several python courses.</b></p>
		  # print(soup.p.string)
		  	The demo python introduces several python courses.
		  # print(type(soup.p.string))
				<class 'bs4.element.NavigableString'>

6. Tag的Comment

Comment 標籤內字符串的註釋部分，一種特殊的Comment類型
Comment是一種特殊類型

from bs4 import BeautifulSoup
import requests
demo = "<b><!--This is a commet--></b><p>This is not a comment</p>"
newsoup = BeautifulSoup(demo, "html.parser")
print(newsoup.b.string)
print(newsoup.p.string)
print(type(newsoup.p.string))
	
	打印輸出
		# print(newsoup.b.string)
			This is a commet
	  # print(newsoup.p.string)
	  	This is not a comment
	  # print(type(newsoup.p.string))
			<class 'bs4.element.NavigableString'>

2、基於bs4庫的HTML內容遍歷方法

1.回顧demo.html

import requests
url = "http://python123.io/ws/demo.html"
r = requests.get(url)
demo = r.text
print(demo)

輸出的demo內容（HTML基本格式）

<html>
          <head>
                   <title>This is a python demo page</title>
          </head>
          <body>
                   <p class="title">
                         <b>The demo python introduces several python courses.</b>
                   </p>
                   <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
                          <a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and  
                           <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>. 
                   </p>
            </body>
</html>

2.標籤樹的三種遍歷方法

3. 標籤樹的下行遍歷

3.1) 下行遍歷的屬性 BeautifulSoup類型是標籤樹的根節點

   屬性                                              說明
.contents                       子節點的列表，將<tag>全部兒子節點存入列表
.children                       子節點的迭代類型，與.contents相似，用於循環遍歷兒子節點
.descendants                    子孫節點的迭代類型，包含全部子孫節點，用於循環遍歷

3.2）下行遍歷代碼

import requests
from bs4 import BeautifulSoup
url = "http://python123.io/ws/demo.html"
r = requests.get(url)
demo = r.text
soup = BeautifulSoup(demo, "html.parser")
print(soup.head)
print(soup.head.contents)
print(soup.body.contents)
print(len(soup.body.contents)) # 打印孩子節點的個數
print(soup.body.contents[1]) # 打印第一個孩子節點
打印輸出
# print(soup.head)
<head><title>This is a python demo page</title></head>

# print(soup.head.contents)
[<title>This is a python demo page</title>]

# print(soup.body.contents)
['\n', <p class="title"><b>The demo python introduces several python courses.</b></p>, '\n', <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>, '\n']

# print(len(soup.body.contents))
5

# print(soup.body.contents[1]) 
<p class="title"><b>The demo python introduces several python courses.</b></p>

3.3）循環遍歷

# 遍歷兒子節點
for child in soup.body.children:
            print(child)

# 遍歷子孫節點
for child in soup.body.descendants:
            print(child)

4. 標籤樹的上行遍歷　

4.1) 上行遍歷的屬性

  屬性                               說明
.parent                    節點的父親標籤
.parents                   節點先輩標籤的迭代類型，用於循環遍歷先輩節點

4.2）上行遍歷代碼

import requests
from bs4 import BeautifulSoup
url = "http://python123.io/ws/demo.html"
r = requests.get(url)
demo = r.text
soup = BeautifulSoup(demo, "html.parser")
print(soup.title.parent)
print(soup.html.parent) # html 是HTML文本的最高級標籤，因此其父親是他本身
print(soup.parent) # soup是一種特殊的標籤，soup的父親爲空

打印輸出
# print(soup.title.parent)
<head><title>This is a python demo page</title></head>

# print(soup.html.parent)
<html><head><title>This is a python demo page</title></head>
<body>
<p class="title"><b>The demo python introduces several python courses.</b></p>
<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>
</body></html>

# print(soup.parent)
None

4.3) 循環遍歷遍歷全部先輩節點，包括soup自己，因此要區別判斷

for parent in soup.a.parents:
            if parent is None:
                        print(parent)
            else
                        print(parent.name)

5. 標籤樹的平行遍歷　

5.1) 平行遍歷屬性

     屬性                                     說明
.next_sibling                返回按照HTML文本順序的下一個平行節點標籤
.previous_sibling          返回按照HTML文本順序的上一個平行節點標籤
.next_siblings              迭代類型，返回按照HTML文本順序的後續全部平行節點標籤
.previous_sibli             迭代類型，返回按照HTML文本順序的前續全部平行節點標籤

5.2) 平行遍歷解釋

　　平行遍歷發生在同一個父節點下的各節點間

5.3) 平行遍歷代碼

import requests
from bs4 import BeautifulSoup
url = "http://python123.io/ws/demo.html"
r = requests.get(url)
demo = r.text
soup = BeautifulSoup(demo, "html.parser")
print(soup.a.next_sibling)
print(soup.a.next_sibling.next_sibling)
print(soup.a.previous_sibling)

打印輸出
# print(soup.a.next_sibling) 
and 
# print(soup.a.next_sibling.next_sibling)
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>
# print(soup.a.previous_sibling)
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:

5.4) 循環遍歷

# 遍歷後續節點
for sibling in soup.a.next_sibling:
            print(sibling)
# 遍歷前序節點
for sibling in soup.a.previous_sibling:
            print(sibling)

6. 標籤樹的三種遍歷總結

3、基於bs4庫的HTML格式輸出

1. bs4庫的prettify()方法

.prettify()爲HTML文本<>及其內容增長更加'\n'
.prettify()可用於標籤，方法：<tag>.prettify()

讓HTML內容更加「友好」的顯示

 1 import requests
 2 from bs4 import BeautifulSoup
 3 url = "http://python123.io/ws/demo.html"
 4 r = requests.get(url)
 5 demo = r.text
 6 soup = BeautifulSoup(demo, "html.parser")
 7 print(demo)
 8 print(soup.prettify())
 9 print(soup.a.prettify())
10 
11 打印輸出
12 #print(demo)
13 <html><head><title>This is a python demo page</title></head>
14 <body>
15 <p class="title"><b>The demo python introduces several python courses.</b></p>
16 <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
17 <a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p>
18 </body></html>
19 
20 #print(soup.prettify())
21 <html>
22  <head>
23   <title>
24    This is a python demo page
25   </title>
26  </head>
27  <body>
28   <p class="title">
29    <b>
30     The demo python introduces several python courses.
31    </b>
32   </p>
33   <p class="course">
34    Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
35    <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
36     Basic Python
37    </a>
38    and
39    <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
40     Advanced Python
41    </a>
42    .
43   </p>
44  </body>
45 </html>
46 
47 #print(soup.a.prettify())
48 <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
49  Basic Python
50 </a>

prettify()方法

2. bs4庫的編碼

　　bs4庫將任何HTML輸入都變成utf‐8編碼，Python 3.x默認支持編碼是utf‐8,解析無障礙。

from bs4 import BeautifulSoup
soup = BeautifulSoup("<p>中文</p>","html.parser")
print(soup.p.string)
print(soup.p.prettify())

打印輸出
# print(soup.p.string)
中文

# print(soup.p.prettify())
<p>
 中文
</p>

RRR