Abstract Rare microbial lineages, such as members of the candidate phyla radiation (CPR) bacteria and Diapherotrites, Parvarchaeota, Aenigmarchaeota, Nanoarchaeota, and Nanohaloarchaeota (DPANN) archaea, are increasingly recognized as key components of microbial communities in natural systems. Yet, their global distribution, biogeographic patterns, and broader role in shaping microbial community structure across diverse ecosystems remain poorly characterized. Here, we analyzed 2860 metagenomes spanning nine ecosystems using a curated reference database and a bias-aware taxonomic filtering approach to quantify the richness, relative abundance, and structural influence of low-abundance microbial taxa on community structure across a wide range of ecosystems. Our findings reveal that rare taxa, primarily CPR and DPANN, disproportionately shape microbial community dissimilarities across global ecosystems. We observed that the richness of these two groups, that drives community structure variation, increases with latitude, peaking in temperate regions, thereby contrasting classical latitudinal diversity patterns and suggesting unique biogeographic drivers. CPR and DPANN were predominantly enriched in free-living environments, particularly groundwater and soil, then in host-associated habitats, consistent with niche specialization shaped by environmental filtering and dispersal constraints. These findings challenge abundance-centric assumptions in microbial ecology and highlight the need to integrate low-abundance taxa into macroecological frameworks. Fully resolving their ecological functions, however, will require targeted experimental and multi-omics investigations.