Last Updated on 2024-08-19 by Clay Cross-lingual Modular (X-Mod) is an interesting language model architecture that modularizes the parameters for different languages as Module Units, allowing the model to use separate parameters when fine-tuning for a new language, thereby (comparatively) avoiding the problem of catastrophic forgetting. The main reason I looked into this paper is … Continue reading [Paper Reading] Lifting the Curse of Multilinguality by Pre-training Modular Transformers