Due to strong zero-phonon line emission, narrow inhomogeneous broadening, and stable optical transition frequencies, the quantum system consisting of negatively charged silicon-vacancy (SiV) centers in diamond is highly expected to develop universal quantum computation. We propose to implement quantum computation for the first time using SiV centers placed in a one-dimensional phononic waveguide, for which quantum gates are realized in a nonadiabatic geometric way and protected by dynamical decoupling (DD). The scheme has the feature of geometric quantum computation that is robust to control errors and the advantage of DD that is insensitive to environmental impact. Furthermore, the encoding of qubits in long-lifetime ground states of silicon-vacancy centers can reduce the effect of spontaneous emission. Numerical simulations demonstrate the practicability of the SiV center system for quantum computation and the robustness improvement of quantum gates by DD pulses. This scheme may provide a promising path toward high-fidelity geometric quantum computation in solid-state systems.