Artificial neural networks can emulate the human vision because of their spike-based operation by employing memristors as synapses. Here, Seo et al. integrate synaptic and optical sensing functions in a single heterostructure, which enables accurate and energy-efficient recognition of colored patterns.