We perform polarization-resolved magneto-optical measurements on single InAsP quantum dots embedded in an InP nanowire. In order to determine all elements of the electron and hole $g$-factor tensors, we measure in magnetic field with different orientations. The results of these measurements are in good agreement with a model based on exchange terms and Zeeman interaction. In our experiment, polarization analysis delivers a powerful tool that not only significantly increases the precision of the measurements, but also enables us to probe the exciton spin state evolution in magnetic fields. We propose a disentangling scheme of heavy-hole exciton spins enabling a measurement of the electron spin $T_2$ time.