Aerial View Object Classification (AVOC) has started to adopt deep learning approaches with significant success in recent years, but limited to optical data. On the other hand, Synthetic Aperture Radar (SAR) has wild aerial view related applications in the remote sensing field. However, SAR has received far less attention due to the special characteristics of the SAR data, which is the long-tailed distribution of the aerial view objects that increases the difficulty of classification. In this paper, we present a two-branch framework, including the cascading expert branch and paralleling expert branch, to tackle the long-tailed distribution of the dataset. Our proposed multi-expert architecture achieves 24.675% and 26.029% in the development phase and testing phase, respectively, in the NTIRE 2021 Multimodal Aerial View Object Classification Challenge Track 1. The proposed method is proved to possess the effectiveness (top-tier performance among 157 participants) and efficiency (i.e., a lightweight architecture) for the AVOC task.