Rare genomic Copy Number Variants (CNVs) are major contributors to neurodevelopmental disorder. The vast majority of pathogenic CNVs reported back to patients are ultra-rare and their quantitative effects on traits such as intelligence are undocumented. Here, we identified all CNVs ≥ 50 kilobase in 24,092 individuals from unselected and autism cohorts. We developed statistical models to estimate the effect-size of CNVs on intelligence based on their coding and non-coding characteristics. Measures of intolerance to haploinsufficiency best explained the effect of any deletion or duplication on general intelligence. There was no heterogeneity across unselected and autism cohorts. Validation was performed using an intraclass concordance and showed that model estimates of general intelligence were 78% accurate with mean effect-sizes previously published for 47 CNVs. Inheritance data on 27,766 CNVs showed that deletions and duplications with the same large effect-size on intelligence occur de novo at the same frequency. Our first outline for the effect sizes of all coding genes on intelligence suggests that around 10,000 genes affect this trait.