【pyhon】nvshens圖片批量下載爬蟲1.01

# nvshens圖片批量下載爬蟲1.01
# 原先版本在遇到網絡故障時回下載不全，這回更改了模式使得下載不成就從新下載，直到所有下載完畢
from bs4 import BeautifulSoup
import requests
import time
import urllib.request

user_agent='Mozilla/4.0 (compatible;MEIE 5.5;windows NT)'
headers={'User-Agent':user_agent}

# 找到的圖片
pictures=[]

# 不斷追逐，直到結束
def crawl(url):
    print("爬取頁面"+url);

    try:
        rsp=requests.get(url,headers=headers)
        soup= BeautifulSoup(rsp.text,'html.parser',from_encoding='utf-8')
        nextUrl="none";

        for divs in soup.find_all(class_="gallery_wrapper"):
            # 把找到的圖片放到數組裏去
            for img in divs.find_all('img'):
                print(img.get("src"))
                pictures.append(img.get("src"))

            #找下一頁
            for link in divs.find_all('a',class_='a1'):
                if link.string=='下一頁' and link.get("href").find('.html')!=-1:
                    nextUrl='https://www.nvshens.com'+link.get("href");

        #if len(pictures)>10:
        #    downloadPics()

        #time.sleep(5)

        if nextUrl!="none":
            print("前往下一頁");
            crawl(nextUrl)
        else:
            print('爬取結束，開始下載...')
            downloadPics()
            print('下載結束，nvshens圖片批量下載爬蟲工做完畢.')
    except Exception as e:
        print("發生異常。從新爬行")# 無論怎麼出現的異常，就讓它一直爬到底
        crawl(url)

# 下載圖片到本地
def downloadPics():
    while(len(pictures)>0):
        pic=pictures.pop()

        name=pic.split('/')[-1]

        try:
            rsp=urllib.request.urlopen(pic)
            img=rsp.read()
            with open(name,'wb') as f:
                f.write(img)
            print('圖片'+pic+'下載完成')
        except Exception as e:
            print('圖片'+pic+'下載異常,塞回重試')
            pictures.append(pic);

# Kickoff Start
# https://www.nvshens.com/g/22268/
crawl('https://www.nvshens.com/g/22210/')

輸出：

C:\Users\horn1\Desktop\python\7>python downloadall.py
爬取頁面https://www.nvshens.com/g/22210/
C:\Users\horn1\AppData\Local\Programs\Python\Python36\lib\site-packages\bs4\__init__.py:146: UserWarning: You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored.
  warnings.warn("You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored.")
https://img.onvshen.com:85/gallery/23789/22210/s/0.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/001.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/002.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/003.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/004.jpg
前往下一頁
爬取頁面https://www.nvshens.com/g/22210/2.html
https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/006.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/007.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/008.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/009.jpg
前往下一頁
爬取頁面https://www.nvshens.com/g/22210/3.html
https://img.onvshen.com:85/gallery/23789/22210/s/010.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/011.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/012.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/013.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/014.jpg
前往下一頁
爬取頁面https://www.nvshens.com/g/22210/4.html
https://img.onvshen.com:85/gallery/23789/22210/s/015.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/016.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/017.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/018.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg
前往下一頁
爬取頁面https://www.nvshens.com/g/22210/5.html
https://img.onvshen.com:85/gallery/23789/22210/s/020.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/021.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/022.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/023.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/024.jpg
前往下一頁
爬取頁面https://www.nvshens.com/g/22210/6.html
https://img.onvshen.com:85/gallery/23789/22210/s/025.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/026.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/027.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/028.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/029.jpg
前往下一頁
爬取頁面https://www.nvshens.com/g/22210/7.html
https://img.onvshen.com:85/gallery/23789/22210/s/030.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/031.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/032.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/033.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/034.jpg
前往下一頁
爬取頁面https://www.nvshens.com/g/22210/8.html
https://img.onvshen.com:85/gallery/23789/22210/s/035.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/036.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/037.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/038.jpg
https://img.onvshen.com:85/gallery/23789/22210/s/039.jpg
爬取結束，開始下載...
圖片https://img.onvshen.com:85/gallery/23789/22210/s/039.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/039.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/038.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/038.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/038.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/037.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/036.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/036.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/035.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/034.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/033.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/032.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/031.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/030.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/029.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/028.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/028.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/027.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/027.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/026.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/025.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/025.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/025.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/024.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/024.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/024.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/023.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/022.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/022.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/021.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/020.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/019.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/018.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/018.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/018.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/017.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/016.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/015.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/014.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/013.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/012.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/011.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/010.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/009.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/008.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/008.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/008.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/007.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/006.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/005.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/004.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/003.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/002.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/001.jpg下載完成
圖片https://img.onvshen.com:85/gallery/23789/22210/s/0.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/0.jpg下載異常,塞回重試
圖片https://img.onvshen.com:85/gallery/23789/22210/s/0.jpg下載完成
下載結束，nvshens圖片批量下載爬蟲工做完畢.

：）