如何Python下載大文件？

時間 2019-11-13

原文原文鏈接

我想用python腳本下載不少文件，可是常常就有那麼幾個出錯，寫了個error handling，跳了過去，可是把出錯的連接保存了一下。python

轉過天來，研究了一下出的什麼錯。git

一個報錯以下:github

PS C:\temp> python .\DownloadFromList.py
Downloading https://github.com/Unity-Technologies/ScriptableRenderPipeline/archive/master.zip
Traceback (most recent call last):
   File ".\DownloadFromList.py", line 20, in <module>
     r = requests.get(url)
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 512, in request
     resp = self.send(prep, **send_kwargs)
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 644, in send
     history = [resp for resp in gen] if allow_redirects else []
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 644, in <listcomp>
     history = [resp for resp in gen] if allow_redirects else []
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 222, in resolve_redirects
     **adapter_kwargs
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 662, in send
     r.content
   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\models.py", line 827, in content
     self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
MemoryErrorapi

During handling of the above exception, another exception occurred:session

Traceback (most recent call last):
File ".\DownloadFromList.py", line 28, in <module>
print("Error happened:", e.message)
AttributeError: 'MemoryError' object has no attribute 'message'
PS C:\temp> app

上網搜索了一下, 找到了解決方案. post

爲了防止這個參考資料的網頁消失(之前常常發生的), 因此我就直接把代碼抄過來放在這裏, 備用（抄襲，嗯，註明了出處就能夠光明正大的抄襲）.url

使用requestspa

def download_file(url):ip
local_filename = url.split('/')[-1]
# NOTE the stream=True parameter
r = requests.get(url, stream=True)
with open(local_filename, 'wb') as f:
    for chunk in r.iter_content(chunk_size=1024): 
        if chunk: # filter out keep-alive new chunks
            f.write(chunk)
            f.flush()
return local_filename

使用urllib2

file = urllib2.urlopen('url')
with open('filename','w') as f:
    while True:
        tmp = file.read(1024)
        if not tmp:
            break 
        f.write(tmp)

參考資料

==================

https://ox0spy.github.io/post/python/python-download-large-file-without-out-of-memory/

參考資料所援引的代碼來自下面的兩個連接。

http://stackoverflow.com/questions/16694907/how-to-download-large-file-in-python-with-requests-py

http://stackoverflow.com/questions/27053028/how-to-download-large-file-without-memoryerror-in-python