Evolutionary theory suggests that lifespan-associated alleles should be purged from the gene pool, and yet decades of GWAS and model organism studies have shown they persist. Here, we address one potential explanation, the idea that the alleles that regulate lifespan do so only in certain contexts. We exposed thousands of outbred Drosophila to a standard and a high sugar diet. We then sequenced over 10,000 individuals and track genome-wide allele frequency changes over time, as these populations aged. We mapped thousands of lifespan-altering alleles, some associated with early vs late life tradeoffs, late-onset effects, and genotype-by-environment interactions. We find that lifespan-reducing alleles are most likely to be recently derived, have stronger effects on a high-sugar diet, consistent with the hypothesis that historically neutral or beneficial alleles can become detrimental in novel conditions. We also show that the gene midway, a regulator of lipid storage and ortholog of the lifespan-associated gene DGAT1 in mice, also regulates lifespan in Drosophila. Our results provide insight into the highly polygenic and context-dependent genetic architecture of lifespan, as well as the evolutionary processes that shape this key trait.