Over a dozen methods have been developed to infer gene regulatory networks (GRNs) from single-cell RNA-seq data. An experimentalist seeking to analyze a new dataset faces a daunting task in selecting an appropriate inference method since there are no widely accepted ground-truth datasets for assessing algorithm accuracy and the criteria for evaluation and comparison of methods are varied. We have developed BEELINE, a comprehensive evaluation of state-of-the-art algorithms for inferring GRNs from single-cell transcriptomic data [1]. BEELINE incorporates 12 diverse algorithms for GRN inference. It provides an easy-to-use and uniform interface to each method in the form of a Docker image. BEELINE implements several measures for estimating and comparing the accuracy, stability, and efficiency of these algorithms. Thus, BEELINE facilitates reproducible, rigorous, and extensible evaluations of GRN inference methods. We selected (a) synthetic networks with predictable cellular trajectories, (b) literature-curated Boolean models, and (c) diverse transcriptional regulatory and functional interaction networks to serve as the ground truth for evaluating the accuracy of GRN inference algorithms. We developed a strategy to simulate single-cell gene expression data from the first two types of networks. We used multiple experimental single-cell RNA-seq datasets in conjunction with the third type of network. Our evaluations suggest that the area under the precision-recall curve and early precision of these algorithms are moderate. Techniques that do not require pseudotime-ordered cells are generally more accurate. Based on these results, we present recommendations to end users of GRN inference methods. Finally, we discuss the potential of supervised algorithms for GRN inference.
