Beautiful Soup庫

安裝

若是安裝了anaconda的集成庫，則不須要再次安裝
若是沒有，則使用pip指令安裝
pip install beautifulsoup4 -i https://pypi.tuna.tsinghua.edu.cn/simple/

查看源代碼

方法一：

右鍵網頁，查看源點

方法二：

使用requests庫

import requests
from bs4 import BeautifulSoup
kv = {'user-agent' : 'Mozilla/5.0'}
url = "https://blog.csdn.net/qq_43321732"
try:
    r = requests.get(url, headers = kv)
    r.raise_for_status()
    demo = r.text
except:
    print("爬取失敗")
soup = BeautifulSoup(demo, "html.parser")
print(soup.prettify())

使用Beautiful Soup庫

Beautiful Soup庫的基本元素

打印標籤

import requests
from bs4 import BeautifulSoup
kv = {'user-agent' : 'Mozilla/5.0'}
url = "https://ac.nowcoder.com/acm/contest/5666"
try:
    r = requests.get(url)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    demo = r.text
except:
    print("爬取失敗")
soup = BeautifulSoup(demo, "html.parser")
print(soup.title)

打印a標籤

import requests
from bs4 import BeautifulSoup
kv = {'user-agent' : 'Mozilla/5.0'}
url = "https://ac.nowcoder.com/acm/contest/5666"
try:
    r = requests.get(url)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    demo = r.text
except:
    print("爬取失敗")
soup = BeautifulSoup(demo, "html.parser")
tag = soup.a
print(tag)

a標籤屬性操做

import requests
from bs4 import BeautifulSoup
kv = {'user-agent' : 'Mozilla/5.0'}
url = "https://ac.nowcoder.com/acm/contest/5666"
try:
    r = requests.get(url)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    demo = r.text
except:
    print("爬取失敗")
soup = BeautifulSoup(demo, "html.parser")
tag = soup.a
print(tag.attrs)
print(tag.attrs['class'])
#連接屬性
print(tag.attrs['href'])
#標籤屬性類型(字典類型)
print(type(tag.attrs))
print(type(tag))

html的基本格式

遍歷方式

下行遍歷

import requests
from bs4 import BeautifulSoup
kv = {'user-agent' : 'Mozilla/5.0'}
url = "https://ac.nowcoder.com/acm/contest/5666"
try:
    r = requests.get(url)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    demo = r.text
except:
    print("爬取失敗")
soup = BeautifulSoup(demo, "html.parser")
print(soup.head)
print(soup.head.contents)
print(soup.body.contents)
print(len(soup.head.contents))
print(soup.head.contents[1])

上行遍歷

平行遍歷

平行遍歷必須發生在同一個父親節點下

更友好的顯示html

prettify方法