Nonadiabatic holonomic quantum computation (NHQC) provides an essential way to construct robust and high-fidelity quantum gates due to its geometric features. However, NHQC is more sensitive to decay and dephasing errors than a conventional dynamical gate since it requires an ancillary intermediate state. Here, we utilize the Hamiltonian reverse engineering technique to study the influence of intermediate-state decoherence on NHQC gate fidelity, and propose schemes to construct an arbitrary single-qubit holonomic gate and nontrivial two-qubit holonomic gate with high fidelity and robustness to the decoherence. Although the proposed method is generic and can be applied to many experimental platforms, such as superconducting qubits, trapped ions, and quantum dots, here we take a nitrogen-vacancy center as an example to show that the gate fidelity can be significantly enhanced from 89% to 99.6% in contrast to recent experimental NHQC schemes [Phys. Rev. Lett. 119, 140503 (2017)0031-900710.1103/PhysRevLett.119.140503; Nat. Photonics 11, 309 (2017)1749-488510.1038/nphoton.2017.40; Opt. Lett. 43, 2380 (2018)0146-959210.1364/OL.43.002380], and the robustness against decoherence can also be significantly improved. All in all, our scheme provides a promising way for fault-tolerant geometric quantum computation.