Problem
Today I was testing the simpletransformers that is a well-known python deep learning package. But on my notebook, my training was stuck and not continue.
This is a weird problem and it is not cause by myself. The same code I can work on another notebook.
After searching on the Internet, I found there is a GitHub issue that have the same problem with me. One reply mentioned a strange solution, I did and simpletransformers did work.
Solution
First, try to use CPU instead of GPU. If CPU can train, that means the GPU environment may have some problem, recommend to check again.
But if the problem occurred on CPU training, maybe you can try the following parameters:
model_args = ClassificationArgs(
use_multiprocessing=False,
use_multiprocessing_for_evaluation=False,
)
os.environ["TOKENIZERS_PARALLELISM"] = "false"
To put it simply, just disable all multiprocessing. Maybe simpletransformers have a dead lock problem at the multi-processing operations.
You can try this solution. If I encounter a similar problem in the future but can solve it in a different way, I will also update it here.
References
- https://github.com/ThilinaRajapakse/simpletransformers/issues/638
- https://docs.python.org/3/library/multiprocessing.html
- https://simpletransformers.ai/