[Paper Reading] Lifting the Curse of Multilinguality by Pre-training Modular Transformers
Last Updated on 2024-08-19 by Clay
Cross-lingual Modular (X-Mod) is an interesting language model architecture that modularizes the parameters for different languages as Module Units, allowing the model to use separate parameters when fine-tuning for a new language, thereby (comparatively) avoiding the problem of catastrophic forgetting.
Read More »[Paper Reading] Lifting the Curse of Multilinguality by Pre-training Modular Transformers