We investigated the relationship between lipid binding glycoprotein apolipoprotein E (apoE; gene APOE) polymorphisms (ε4 allele carriers versus no carriers = ε4+/ε4−) and trace elements (TEs) (e.g., (methyl)mercury, arsenic, lead, cadmium, selenium, manganese, copper, and zinc) in mothers (N = 223) and their new-borns (N = 213) exposed to potentially toxic metal(loid)s from seafood consumption. The apoE isoform encoded by the ε4 allele is believed to have beneficial effects in early life but represents a risk factor for age-associated diseases. Under certain conditions ε4 carriers are more susceptible to oxidative stress and metal(loid) toxicity. DNA from Croatian pregnant women (N = 223, third trimester) and their new-borns (N = 176), was genotyped for APOE by TaqMan® SNP assay – rs429358 and rs7412. Seafood intake data and TE levels in maternal urine, milk, hair, peripheral venous blood, mixed cord blood, and new-borns’ urine were available from previous studies. We compared TEs between ε4+ and ε4− carriers using Mann-Whitney U tests and applied multiple linear regression models to analyse the TE’s dependence on the presence of allele ε4 (genotypes ε3/ε4, ε4/ε4) in combination with other explanatory variables. We identified 17% (n = 37) and 20% (n = 35) ε4 allele carriers in mothers and new-borns, respectively. The Mann-Whitney U test showed that mothers with the ε4 allele had significantly higher mean levels of (methyl)mercury in peripheral venous blood, cord blood, and hair; arsenic in urine and cord blood; and selenium in peripheral venous blood and plasma. However, taking confounders into account, only the maternal plasma selenium remained statistically significant in the linear regression models (ε4 carriers vs non-carriers: 62.6 vs 54.9 ng/mL, p