In this work, we propose a long-term UAV detection and tracking system from RGB-Thermal (RGB-T) sequences. The system consists of a high resolution daylight visible camera and a thermal camera mounted on a UAV (airborne), for the detection of flying intruders. The framework is composed of the detection and tracking modules. The primary detection module based on the YOLOv4 method is optimized for small UAV detection and works both on the RGB and Thermal domains. To alleviate the issue of temporarily losing the intruder, we employ a discriminative correlation filter based object tracker, which is initialized with the output of the detection module and tracks the target at a higher speed. The dimensionality reduction is applied to the features for tracking to improve the performance. Meanwhile, we utilize the infrared signal as a spatial regularization term of the tracker to suppress the boundary effects that stem from circular convolution, leading to a more robust appearance model and tracking performance. The tracker is efficiently optimized via the Alternating Direction Method of Multiplier (ADMM). We evaluate our method on multiple visual and thermal tracking benchmarks, as well as field tests with a prototype platform. The experimental results demonstrate that our system can achieve accurate, robust and continuous detection and tracking of UAVs under complex circumstances.
