Last Updated on 2021-03-04 by Clay
Introduction
Sometimes when we downloading a file via crawler, maybe network fluctuations or other work interrupt us …… but we don’t have to download the file again.
If we can resume our downloading in breakpoint, we can resume the download progress, continue to finished it.
The basic idea is that we made a request to the website in advance and got the size of the file. Then we check the file under our download path, if it exists, then we will check where it is currently downloaded, and then start downloading from the part that has not been downloaded.
The way I downloaded it was through the built-in requests of Python.
My code
The code is very simple, I will explain it step-by-step.
import os
import time
import requests
First we import the module we need. If you have no requests module if your environment, please use the following command to download it:
pip3 install requestsAfter installing, we try to use the following code to download a file.
startTime = time.time()
url = 'https://img.ltn.com.tw/Upload/news/600/2019/11/10/phpK7nZ7J.jpg'
fileName = 'test.jpg'
content_size = int(requests.get(url, stream=True).headers['Content-Length'])
The downloaded URL is an example of a typhoon picture of a news article. If you paste the website with the url, you should see the picture directly.
- fileName is our picture name
- content_size is the size of file
if os.path.exists(fileName):
temp_size = os.path.getsize(fileName)
else:
temp_size = 0
This step is to check whether there is the picture in the current directory. If it exists, get the current size of the picture; if it does not exist, the size is set ot 0, which means that we have never downloaded this file.
print('Temp:', temp_size)
print('Total:', content_size)
headers = {'Range': 'bytes=%d-' % temp_size}
r = requests.get(url, stream=True, headers=headers)
print('[File size]: %0.2f MB' % (content_size/1024/1024))
print('Status:', r.status_code)
Output:
Temp: 0
Total: 66825
[File size]: 0.06 MB
Status: 206The Range in file Header is very important, it means how many bytes we start downloading. Also note that if the Status return value is 200, it means that this website does not support us to specify which bytes to start downloading from.
If it is 206, it means that we can specify it.
with open(fileName, 'ab') as file:
for data in r.iter_content(chunk_size=1024):
file.write(data)
temp_size += len(data)
print('\r' + '[Download progress]:[%s%s]%.2f%%;' % (
'█' * int(temp_size * 20 / content_size), ' '*(20-int(temp_size*20/content_size)),
float(temp_size/content_size*100)), end='')
print('\n' + 'Donwload finished!')
print('Download time:%.2f s' % (time.time()-startTime))
Output:
[Download progress]:[████████████████████]100.00%
Donwload finished!
Download time:0.56 sSince we have not downloaded this picture here, we can see that we have downloaded this picture directly. You might as well take a look at this picture after downloading it.
Next, we delete the image test.jpg and change the code to the following:
with open(fileName, 'ab') as file:
for data in r.iter_content(chunk_size=1024):
file.write(data)
temp_size += len(data)
print('\r' + '[Download progress]:[%s%s]%.2f%%;' % (
'█' * int(temp_size * 20 / content_size), ' '*(20-int(temp_size*20/content_size)),
float(temp_size/content_size*100)), end='')
break
Output:
[Download progress]:[ ]1.53%;
Donwload finished!
Download time:0.51 sThis time we used break to jump out of the program in the first write, so the download was not successful.
Our picture now looks like this:
Yes, you read it right, it looks like a failed picture.
Then we remove the break and download it again.
Our results this time:
Temp: 1024
Total: 66825
[File size]: 0.06 MB
Status: 206
[Download progress]:[████████████████████]100.00%
Donwload finished!
Download time:0.58 sOpen the picture and take a look. Have you completed the part that was not downloaded this time and successfully downloaded the complete picture?
Complete code
# -*- coding: utf-8 -*-
import os
import time
import requests
startTime = time.time()
url = 'https://img.ltn.com.tw/Upload/news/600/2019/11/10/phpK7nZ7J.jpg'
fileName = 'test.jpg'
content_size = int(requests.get(url, stream=True).headers['Content-Length'])
if os.path.exists(fileName):
temp_size = os.path.getsize(fileName)
else:
temp_size = 0
print('Temp:', temp_size)
print('Total:', content_size)
headers = {'Range': 'bytes=%d-' % temp_size}
r = requests.get(url, stream=True, headers=headers)
print('[File size]: %0.2f MB' % (content_size/1024/1024))
print('Status:', r.status_code)
with open(fileName, 'ab') as file:
for data in r.iter_content(chunk_size=1024):
file.write(data)
temp_size += len(data)
print('\r' + '[Download progress]:[%s%s]%.2f%%;' % (
'█' * int(temp_size * 20 / content_size), ' '*(20-int(temp_size*20/content_size)),
float(temp_size/content_size*100)), end='')
print('\n' + 'Donwload finished!')
print('Download time:%.2f s' % (time.time()-startTime))