Water, in ultra-deep layers of the earth and in layers receiving abnormally high heat, can exist in a supercritical state. Supercritical water (SCW) can participate in the transformations of organic compounds not only as a solvent but also as a reactant, influencing petroleum formation and the evolution of sedimentary organic matter. Here, we carried out hydrous pyrolysis experiments in both SCW and water vapor (WV) using two organic-rich marine rocks under closed conditions, to quantitatively evaluate the generation potential of hydrocarbons in a supercritical state and to clarify the effect of water phase on hydrous pyrolysis experiments. The results showed that SCW promoted gaseous and liquid hydrocarbon generation and facilitated the cracking of aliphatic hydrocarbons. For gaseous hydrocarbons, the action of SCW became stronger as the temperature increased. For liquid hydrocarbons, the peak yields of bitumen were enhanced by the SCW, and the temperature corresponding to peak yield in SCW was lower than that in WV. These results were attributed to the supply of hydrogen and oxygen from SCW for the petroleum formation and cracking. The δ13C and δD values of gases were also influenced by SCW. However, these values obtained in SCW did not always become positive as their yields increased. Generally, methane (CH4) was enriched in 12C and 1H. The influence of SCW on the isotopic fractionation of ethane (C2H6) and propane (C3H8) was more complex. The water phase is an important factor affecting the experimental results of hydrous pyrolysis. From these findings, it can be concluded that SCW allowed for increased conversion of sedimentary organic matter to gaseous and liquid hydrocarbons in ultra-deep layers and layers affected by volcanic-hydrothermal activity.