Prochlorococcus and Synechococcus are the two most abundant and widespread phytoplankton in the global ocean. To better understand the factors controlling their biogeography, a reference database of the high-resolution taxonomic marker petB, encoding cytochrome b6, was used to recruit reads out of 109 metagenomes from the Tara Oceans expedition. An unsuspected novel genetic diversity was unveiled within both genera, even for the most abundant and well-characterized clades, and 136 divergent petB sequences were successfully assembled from metagenomic reads, significantly enriching the reference database. We then defined Ecologically Significant Taxonomic Units (ESTUs)—that is, organisms belonging to the same clade and occupying a common oceanic niche. Three major ESTU assemblages were identified along the cruise transect for Prochlorococcus and eight for Synechococcus. Although Prochlorococcus HLIIIA and HLIVA ESTUs codominated in iron-depleted areas of the Pacific Ocean, CRD1 and the yet-to-be cultured EnvB were the prevalent Synechococcus clades in this area, with three different CRD1 and EnvB ESTUs occupying distinct ecological niches with regard to iron availability and temperature. Sharp community shifts were also observed over short geographic distances—for example, around the Marquesas Islands or between southern Indian and Atlantic Oceans—pointing to a tight correlation between ESTU assemblages and specific physico-chemical parameters. Together, this study demonstrates that there is a previously overlooked, ecologically meaningful, fine-scale diversity within some currently defined picocyanobacterial ecotypes, bringing novel insights into the ecology, diversity, and biology of the two most abundant phototrophs on Earth.