<p>..</p>: 標籤:Tag <p> p指的是標籤的名字, Name成對出現 <p class="title">...</p> 屬性Attributes0個或多個(由鍵值對構成)
引用方式 from bs4 import BeautifulSoup 使用方法 from bs4 import BeautifulSoup 1. soup = BeautifulSoup("<html>data<html>", "html.parser") 2. soup = BeautifulSoup(open("D://demo.html"), "html.parser") BeautifulSoup對應一個HTML/XML文檔的所有內容。
soup = BeautifulSoup('<html>data</html>','html.parser')
分類 html
解析器 使用方法 條件 bs4的HTML解析器 BeautifulSoup(mk,'html.parser') 安裝bs4庫 lxml的HTML解析器 BeautifulSoup(mk,'lxml') pip install lxml lxml的XML解析器 BeautifulSoup(mk,'xml') pip install lxml html5lib的解析器 BeautifulSoup(mk,'html5lib') pip install html5lib
<p class=「title」> … </p>
分類 html5
基本元素 說明 Tag 標籤,最基本的信息組織單元,分別用<>和</>標明開頭和結尾 Name 標籤的名字,<p>…</p>的名字是'p',格式:<tag>.name Attributes 標籤的屬性,字典形式組織,格式:<tag>.attrs NavigableString 標籤內非屬性字符串,<>…</>中字符串,格式:<tag>.string Comment 標籤內字符串的註釋部分,一種特殊的Comment類型
1. 回顧demo.htmlpython
import requests url2 = "http://python123.io/ws/demo.html" r = requests.get(url2) demo = r.text print(demo) 打印輸出 # print(demo) <html><head><title>This is a python demo page</title></head> <body> <p class="title"><b>The demo python introduces several python courses.</b></p> <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses: <a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p> </body></html>
2. Tag 標籤ide
Tag 標籤,最基本的信息組織單元,分別用<>和</>標明開頭和結尾
任何存在於HTML語法中的標籤均可以用soup.<tag>訪問得到
當HTML文檔中存在多個相同<tag>對應內容時,soup.<tag>返回第一個
編碼
from bs4 import BeautifulSoup import requests url = "http://python123.io/ws/demo.html" r = requests.get(url) demo = r.text soup = BeautifulSoup(demo, "html.parser") print(soup.title) tag = soup.a print(tag) 打印輸出 # print(soup.title) <title>This is a python demo page</title> # print(tag) <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
3. Tag的name(名字)url
Name 標籤的名字,<p>…</p>的名字是'p',格式:<tag>.name
每一個<tag>都有本身的名字,經過<tag>.name獲取,字符串類型spa
from bs4 import BeautifulSoup import requests url2 = "http://python123.io/ws/demo.html" r = requests.get(url2) demo = r.text soup = BeautifulSoup(demo, "html.parser") print(soup.a.name) print(soup.a.parent.name) print(soup.a.parent.parent.name) 打印輸出 # print(soup.a.name) 'a' # print(soup.a.parent.name) 'p' # print(soup.a.parent.parent.name) 'body'
4. Tag的attrs(屬性)code
Attributes 標籤的屬性,字典形式組織,格式:<tag>.attrs
一個<tag>能夠有0或多個屬性,字典類型xml
from bs4 import BeautifulSoup import requests url2 = "http://python123.io/ws/demo.html" r = requests.get(url2) demo = r.text soup = BeautifulSoup(demo, "html.parser") tag = soup.a print(tag.attrs) print(tag.attrs['class']) print(tag.attrs['href']) print(type(tag.attrs)) print(type(tag)) 打印輸出 # print(tag.attrs) {'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'} # print(tag.attrs['class']) ['py1'] # print(tag.attrs['href']) http://www.icourse163.org/course/BIT-268001 # print(type(tag.attrs)) <class 'dict'> # print(type(tag)) <class 'bs4.element.Tag'>
5. Tag的NavigableString htm
NavigableString 標籤內非屬性字符串,<>…</>中字符串,格式:<tag>.string
NavigableString能夠跨越多個層次
from bs4 import BeautifulSoup import requests url2 = "http://python123.io/ws/demo.html" r = requests.get(url2) demo = r.text soup = BeautifulSoup(demo, "html.parser") soup.a print(soup.a.string) print(soup.p) print(soup.p.string) print(type(soup.p.string)) 打印輸出 # print(soup.a.string) Basic Python # print(soup.p) <p class="title"><b>The demo python introduces several python courses.</b></p> # print(soup.p.string) The demo python introduces several python courses. # print(type(soup.p.string)) <class 'bs4.element.NavigableString'>
6. Tag的Comment
Comment 標籤內字符串的註釋部分,一種特殊的Comment類型
Comment是一種特殊類型
from bs4 import BeautifulSoup import requests demo = "<b><!--This is a commet--></b><p>This is not a comment</p>" newsoup = BeautifulSoup(demo, "html.parser") print(newsoup.b.string) print(newsoup.p.string) print(type(newsoup.p.string)) 打印輸出 # print(newsoup.b.string) This is a commet # print(newsoup.p.string) This is not a comment # print(type(newsoup.p.string)) <class 'bs4.element.NavigableString'>
import requests url = "http://python123.io/ws/demo.html" r = requests.get(url) demo = r.text print(demo)
輸出的demo內容(HTML基本格式)
<html> <head> <title>This is a python demo page</title> </head> <body> <p class="title"> <b>The demo python introduces several python courses.</b> </p> <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses: <a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>. </p> </body> </html>
3.1) 下行遍歷的屬性 BeautifulSoup類型是標籤樹的根節點
屬性 說明 .contents 子節點的列表,將<tag>全部兒子節點存入列表 .children 子節點的迭代類型,與.contents相似,用於循環遍歷兒子節點 .descendants 子孫節點的迭代類型,包含全部子孫節點,用於循環遍歷
3.2)下行遍歷代碼
import requests from bs4 import BeautifulSoup url = "http://python123.io/ws/demo.html" r = requests.get(url) demo = r.text soup = BeautifulSoup(demo, "html.parser") print(soup.head) print(soup.head.contents) print(soup.body.contents) print(len(soup.body.contents)) # 打印孩子節點的個數 print(soup.body.contents[1]) # 打印第一個孩子節點 打印輸出 # print(soup.head) <head><title>This is a python demo page</title></head> # print(soup.head.contents) [<title>This is a python demo page</title>] # print(soup.body.contents) ['\n', <p class="title"><b>The demo python introduces several python courses.</b></p>, '\n', <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses: <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>, '\n'] # print(len(soup.body.contents)) 5 # print(soup.body.contents[1]) <p class="title"><b>The demo python introduces several python courses.</b></p>
3.3) 循環遍歷
# 遍歷兒子節點 for child in soup.body.children: print(child) # 遍歷子孫節點 for child in soup.body.descendants: print(child)
4.1) 上行遍歷的屬性
屬性 說明 .parent 節點的父親標籤 .parents 節點先輩標籤的迭代類型,用於循環遍歷先輩節點
4.2)上行遍歷代碼
import requests from bs4 import BeautifulSoup url = "http://python123.io/ws/demo.html" r = requests.get(url) demo = r.text soup = BeautifulSoup(demo, "html.parser") print(soup.title.parent) print(soup.html.parent) # html 是HTML文本的最高級標籤,因此其父親是他本身 print(soup.parent) # soup是一種特殊的標籤,soup的父親爲空 打印輸出 # print(soup.title.parent) <head><title>This is a python demo page</title></head> # print(soup.html.parent) <html><head><title>This is a python demo page</title></head> <body> <p class="title"><b>The demo python introduces several python courses.</b></p> <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses: <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p> </body></html> # print(soup.parent) None
4.3) 循環遍歷 遍歷全部先輩節點,包括soup自己,因此要區別判斷
for parent in soup.a.parents: if parent is None: print(parent) else print(parent.name)
5.1) 平行遍歷屬性
屬性 說明 .next_sibling 返回按照HTML文本順序的下一個平行節點標籤 .previous_sibling 返回按照HTML文本順序的上一個平行節點標籤 .next_siblings 迭代類型,返回按照HTML文本順序的後續全部平行節點標籤 .previous_sibli 迭代類型,返回按照HTML文本順序的前續全部平行節點標籤
5.2) 平行遍歷解釋
平行遍歷發生在同一個父節點下的各節點間
5.3) 平行遍歷代碼
import requests from bs4 import BeautifulSoup url = "http://python123.io/ws/demo.html" r = requests.get(url) demo = r.text soup = BeautifulSoup(demo, "html.parser") print(soup.a.next_sibling) print(soup.a.next_sibling.next_sibling) print(soup.a.previous_sibling) 打印輸出 # print(soup.a.next_sibling) and # print(soup.a.next_sibling.next_sibling) <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a> # print(soup.a.previous_sibling) Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
5.4) 循環遍歷
# 遍歷後續節點 for sibling in soup.a.next_sibling: print(sibling) # 遍歷前序節點 for sibling in soup.a.previous_sibling: print(sibling)
.prettify()爲HTML文本<>及其內容增長更加'\n'
.prettify()可用於標籤,方法:<tag>.prettify()
讓HTML內容更加「友好」的顯示
1 import requests 2 from bs4 import BeautifulSoup 3 url = "http://python123.io/ws/demo.html" 4 r = requests.get(url) 5 demo = r.text 6 soup = BeautifulSoup(demo, "html.parser") 7 print(demo) 8 print(soup.prettify()) 9 print(soup.a.prettify()) 10 11 打印輸出 12 #print(demo) 13 <html><head><title>This is a python demo page</title></head> 14 <body> 15 <p class="title"><b>The demo python introduces several python courses.</b></p> 16 <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses: 17 <a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p> 18 </body></html> 19 20 #print(soup.prettify()) 21 <html> 22 <head> 23 <title> 24 This is a python demo page 25 </title> 26 </head> 27 <body> 28 <p class="title"> 29 <b> 30 The demo python introduces several python courses. 31 </b> 32 </p> 33 <p class="course"> 34 Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses: 35 <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1"> 36 Basic Python 37 </a> 38 and 39 <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2"> 40 Advanced Python 41 </a> 42 . 43 </p> 44 </body> 45 </html> 46 47 #print(soup.a.prettify()) 48 <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1"> 49 Basic Python 50 </a>
bs4庫將任何HTML輸入都變成utf‐8編碼,Python 3.x默認支持編碼是utf‐8,解析無障礙。
from bs4 import BeautifulSoup soup = BeautifulSoup("<p>中文</p>","html.parser") print(soup.p.string) print(soup.p.prettify()) 打印輸出 # print(soup.p.string) 中文 # print(soup.p.prettify()) <p> 中文 </p>
RRR