Last Updated on 2024-07-27 by Clay
When we write Python code, sometimes we want to visualize the ‘progress bar’ to help us understand where our program is currently executing. For this need, instead of writing the progress bar ourselves, we can consider using a well-known progress bar module in Python—tqdm.
You can also visit the official website of tqdm: https://tqdm.github.io/ to read more detailed instructions.
In fact, tqdm also means ‘progress bar‘ in Arabic. By installing this Python package, we can use tqdm to display progress bars in any iterable.
Now, let’s start introducing how to use tqdm!
How to Use tqdm
Since this module is not built-in with Python, we can install it using the pip command if we are using it for the first time:
pip3 install tqdmAfter installation, let’s look at a sample code:
import time
from tqdm import tqdm
for i in tqdm(range(1000)):
time.sleep(0.01)Output:
100%|██████████| 1000/1000 [00:10<00:00, 95.76it/s]
We will see the progress bar gradually filling up. The beginning naturally is the progress bar we long for. The ‘1000/1000’ is actually ‘current iteration count / total iteration count’, ’00:10<00:00′ is ‘current execution time / estimated remaining time’, and ‘95.76it/s’ is the iterations per second. Sometimes you might see it reversed as ‘s/it’, which means the time taken for each iteration in seconds.
Additionally, the time.sleep() function makes the program pause for 0.01 seconds each time it runs here, a total of 1000 times. If we don’t set a pause, we might see the progress bar executed by tqdm finish instantly, which would not simulate the actual execution situation. You can test it out for yourself.
The tqdm() is effective for all iterable objects, for example, it can also be used for a simple list.
import time
from tqdm import tqdm
arr = [1, 2, 3, 4, 5]
for i in tqdm(arr):
time.sleep(1)
Output:
100%|█████████████████████████████████████████████| 5/5 [00:05<00:00, 1.00s/it]
Additionally, tqdm also provides a trange() function to directly iterate for a specified length, which can be thought of as equivalent to tqdm(range(x)).
import time
from tqdm import trange
for i in trange(1000):
time.sleep(0.01)
Output:
100%|██████████| 1000/1000 [00:10<00:00, 95.72it/s]
This is the basic usage of tqdm, very straightforward.
Advanced Techniques for tqdm
Next, I will mention some handy tips:
- Change progress bar length
- Change unit
- Change progress bar format
- Change progress bar description
- Change progress bar color
- Use .update() to customize when and how much to update
Change Progress Bar Length
tqdm will automatically detect the screen width and adapt to just the right width of the progress bar, but we can also manually set it:
for i in tqdm(1000, ncols=100):
time.sleep(0.01)
Output:
Above, ncols=None is the default situation without setting, and below, ncols=100 is with the specified width.
Change Unit
This function is not very commonly used, but when doing deep learning training, we often want to change the unit of progress from iterations to more intuitive epochs or steps. We can set the unit parameter for this purpose.
for i in tqdm(range(1000), unit="epoch"):
time.sleep(0.01)
Output:
100%|██████████| 1000/1000 [00:10<00:00, 98.27epoch/s]
Change Progress Bar Format
The bar_format in tqdm allows us to customize the display format of the progress bar. Here are the format tokens we can use:
- {l_bar}: Left part of the progress bar, including description and the bar
- {bar}: The progress bar itself
- {r_bar}: Right part of the progress bar, including counter, time estimation, etc.
- {n}: Current progress count
- {n_fmt}: Formatted current progress count
- {total}: Total length of the progress bar
- {total_fmt}: Formatted total length of the progress bar
- {elapsed}: Elapsed time
- {elapsed_s}: Elapsed time (seconds)
- {elapsed_td}: Elapsed time (timedelta format)
- {remaining}: Remaining time
- {remaining_s}: Remaining time (seconds)
- {remaining_td}: Remaining time (timedelta format)
- {percentage}: Completion percentage
- {rate}: Processing rate per second
- {rate_fmt}: Formatted processing rate per second
- {rate_inv}: Time taken for each item
- {rate_inv_fmt}: Formatted time taken for each item
- {postfix}: Postfix string for the progress bar
For example, I like a clean progress bar format, so I can write it like this:
for i in tqdm(range(100), bar_format="{l_bar}{bar}| {n_fmt}/{total_fmt}"):
time.sleep(0.01)
Output:
100%|███████████████| 100/100
Change Progress Bar Description
We can also add some descriptions to the tqdm progress bar. Let’s first look at the simplest way to set the description:
for i in tqdm(range(1000), desc="Processing..."):
time.sleep(0.01)
Output:
Processing...: 100%|███████████| 1000/1000 [00:10<00:00, 98.77it/s]
This is very useful in nested loops or deep structure loops, at least we can print out which progress bar it is, such as epoch-1, epoch-2, …, epoch-n.
Another handy trick is that we can also change the progress bar description during iteration! This allows for more immediate grasp of the data processing status.
import time
import random
from tqdm import tqdm
# Initialize counters
good = 0
bad = 0
# Create progress bar
pbar = tqdm(range(1000), desc="Starting")
# Simulate a loop with dynamic desc update
for _ in pbar:
# Simulate work
time.sleep(0.01)
# Randomly choose True or False
if random.choice([True, True, False]):
good += 1
else:
bad += 1
# Update progress bar description
pbar.set_description(f"Good: {good} Bad: {bad}")
# Close progress bar
pbar.close()
Output:
Good: 650 Bad: 350: 100%|████████████████| 1000/1000 [00:10<00:00, 98.17it/s]
In the process, I randomly select True or False, and use the .set_description() method of the progress bar instance pbar to update the current processing status.
This way, if there is any information that needs to be seen immediately during the iteration, it can be directly displayed.
Change Progress Bar Color
When running the progress bar, if the default color is too inconspicuous, we can also set the color for the colour parameter. However, this feature is not available in early versions of tqdm and is supported in versions like 4.38.0 and above.
for i in tqdm(range(1000), colour="green"):
time.sleep(0.01)
Output:
Use .update() to Customize When and How Much to Update
Additionally, tqdm can not only be used in for loops, but we can also choose when to update the progress bar and how much to advance the iteration progress. Here is a simple example:
times = 0
progress = tqdm(total=1000)
while times < 1000:
progress.update(1)
times += 1
sleep(0.01)
Output:
100%|██████████| 1000/1000 [00:10<00:00, 95.68it/s]
It looks like the same function, but this time we can choose when to call progress.update(), and also choose the amount to update().
Lastly, if you are using Google Colab, Kaggle Notebook, Jupyter Notebook, etc., the usage of tqdm should employ the notebook package provided:
from tqdm.notebook import tqdmCreate Your Own Progress Bar
Of course, besides using the tqdm module to implement our own progress bar, we can also manually write our own progress bar!
Here is a simple example for reference:
from time import sleep
curr = 0
total = 1000
total_block_num = 50
for n in range(total):
curr += 1
block_num = int(curr * total_block_num / total)
space_num = total_block_num - block_num
print(f"\r[{curr / total * 100:.2f}%]: [{'█' * block_num}{' ' * space_num}]", end="")
sleep(0.01)
Output:
[100.00%]:[██████████████████████████████████████████████████];
Advanced Challenge
Next, I also tried to write the above method into a function that can take an iterable object to generate a progress bar, and also set a color parameter like tqdm.
from time import sleep
import sys
import itertools
def progress_bar(iterable, total=None, total_block_num=50, color="default"):
if total is None:
total = len(iterable)
color_codes = {
"default": "\033[0m", # Default
"red": "\033[91m", # Red
"green": "\033[92m", # Green
"yellow": "\033[93m", # Yellow
"blue": "\033[94m", # Blue
"magenta": "\033[95m", # Magenta
"cyan": "\033[96m", # Cyan
"white": "\033[97m", # White
}
if color not in color_codes:
color = "default"
color_code = color_codes[color]
for curr, item in enumerate(iterable, 1):
block_num = int(curr * total_block_num / total)
space_num = total_block_num - block_num
progress_str = f"\r{color_code}[{float(curr / total * 100):.2f}%]: [{'█' * block_num}{' ' * space_num}]\033[0m"
print(progress_str, end="")
yield item
print() # Move to the next line after completion
if __name__ == "__main__":
for _ in progress_bar(range(1000), color="cyan"):
sleep(0.01)
Output:
If you’re interested, why not try writing your own progress bar? My current progress bar implementation is still quite rigid and can’t freely choose update times and quantities like tqdm can — but this is an area we can improve on later.
Try challenging yourself to write it as a class, or allow users to update the progress using an update() function?
When I have some free time, I’d like to take on this challenge too.
Would you like me to explain or elaborate on any part of this translation?