Loom malfunctions are the main cause of faulty fabric production. A fabric inspection system is a specialized computer vision system used to detect fabric defects for quality assurance. In this paper, a deep-learning algorithm was developed for an on-loom fabric defect inspection system by combining the techniques of image pre-processing, fabric motif determination, candidate defect map generation, and convolutional neural networks (CNNs). A novel pairwise-potential activation layer was introduced to a CNN, leading to high accuracy of defect segmentation on fabrics with intricate features and imbalanced dataset. The average precision and recall of detecting defects in the existing images reached, respectively, over 90% and 80% at the pixel level and the accuracy on counting the number of defects from a publicly available dataset exceeded 98%.