Qwen1.5-MoE-A2.7B: A Small MoE Model with only 2.7B Activated Parameters yet Matching the Performance of State-of-the-Art 7B models

https://www.marktechpost.com/2024/03/29/alibaba-releases-qwen1-5-moe-a2-7b-a-small-moe-model-with-only-2-7b-activated-parameters-yet-matching-the-performance-of-state-of-the-art-7b-models-like-mistral-7b/Open link