Indentation size effect (ISE), in general, is observed in shallow indentation tests, which is manifested as an increase (NISE) or decrease (RISE) in hardness with penetration depth decreases. The complicated origin of ISE is dependent on multiple factors, which can be divided into two types, both strain gradient and external factors. It's very difficult to judge which factor is the source for a certain ISE phenomenon. Further, studies on ISE phenomena under the joint of multiple factors are fewer. To date, the classical Nix-Gao model based on strain gradient has been widely recognized, but it's sometimes not suited for all ISE. And most current analytical models on ISE are not yet mature. Therefore, more mechanistic models on ISE are required, especially for RISE. Herein, we critically reviewed several mechanistic models on ISE, focusing on the exploration of RISE, and developed an analytical model incorporating external factors, i.e. surface undulation and indenter tip irregularity, to account for NISE and RISE. We examined the two external factors on the origin responsible for ISE in light of 3D numerical simulations and experimental observations, meanwhile, the combined effect of both factors on ISE was also considered. The effect of both external factors on ISE can be equivalent to a depth deviation δ. The corresponding hardness deviation caused by δ can be theoretically eliminated through correcting the depth deviation. More importantly, only after eliminating the effect of external factors on ISE, one can determine whether the ISE results from strain gradient plasticity. Finally, this paper was summarized and outlooked.