Skip to main content Skip to secondary navigation
Journal Article

Mathematical constraints on F_ST: biallelic markers in arbitrarily many populations

 

FST is one of the most widely used statistics in population genetics. Recent mathematical studies have identified constraints that challenge interpretations of FST as a measure with potential to range from 0 for genetically similar populations to 1 for divergent populations. We generalize results obtained for population pairs to arbitrarily many populations, characterizing the mathematical relationship between FST, the frequency M of the more frequent allele at a polymorphic biallelic marker, and the number of subpopulations K. We show that for fixed KFST has a peculiar constraint as a function of M, with a maximum of 1 only if M = i/K, for integers i with ⌈K/2⌉ ≤ i− 1. For fixed M, as K grows large, the range of FST becomes the closed or half-open unit interval. For fixed K, however, some M < (− 1)/K always exists at which the upper bound on FST lies below 2 x 21/2 − 2 ≈ 0.8284. We use coalescent simulations to show that under weak migration, FST depends strongly on M when K is small, but not when K is large. Finally, examining data on human genetic variation, we use our results to explain the generally smaller FST values between pairs of continents relative to global FST values. We discuss implications for the interpretation and use of FST.

Author(s)
Nicolas Alcala
Noah A. Rosenberg
Journal Name
Genetics
Publication Date
July 1, 2017
DOI
10.1534/genetics.116.199141