Background Current worldwide clinical practice guidelines recommend acetaminophen as the first option for the treatment of acute low back pain. However, there is no concrete evidence regarding whether acetaminophen or nonsteroidal anti-inflammatory drugs (NSAIDs) is more effective for treating acute low back pain (LBP) in Japan. The present study aimed to investigate whether acetaminophen treatment for acute musculoskeletal pain was comparable with loxoprofen (a traditional NSAID in Japan) treatment. Methods Of the 140 patients with acute LBP who visited out-patient hospitals, 127 were considered eligible and were randomly allocated to a group taking acetaminophen or one taking loxoprofen. As primary outcome measure, pain intensity was measured using a 0–10-numeric rating scale (NRS). Moreover, pain disability, pain catastrophizing, anxiety, depression, and quality of life, as well as adverse events, were assessed as secondary outcomes. The primary outcome was tested with a noninferiority margin (0.84 on changes in pain-NRS), and the secondary outcomes were compared using conventional statistical methods at week 2 and week 4. Results Seventy patients completed the study (acetaminophen: 35, loxoprofen: 35). The dropout rates showed no significant difference between the two medication-groups. We found that the mean differences of changes in pain-NRS from baseline to week 2 or 4 between the two medication groups were not statistically beyond the noninferiority margin (mean [95% confidence interval]: −0.51 [−1.70, 0.67], at week 2 and −0.80 [−2.08, 0.48] at week 4). There were no consistent differences between the two medication groups in terms of secondary outcomes. Conclusions The results suggest that acetaminophen has comparable analgesic effects on acute LBP, based on at least a noninferiority margin, compared with loxoprofen at 4 weeks. Acetaminophen seems to be a reasonable first-line option for patients with acute LBP in Japan.