Bolt or self-tapping screw connections are extensively employed in steel-timber composite structures (STC). To determine the lateral resistance of the connection between natural wood and thin-walled steel (0.8–3 mm) with small diameter self-tapping screws (d less than 6 mm), 95 single-shear connections were designed and tested under monotonic and cyclic loads. The influence factor such as timber species, screw size, and steel thickness are selected to evaluate the connection performance. In comparison to the STC bolt connection, the failure of this connection is the screw fracture and pullout failure. The failure mode is primarily determined by the constraint on the screw rotation and the effect of axial and friction resistance (rope-effect). Additionally, the lateral resistance of two major failure modes is theoretically analyzed using the Johansen yield theory. The relevant contents of Eurocode 5 and GB/T 50005–2017 dowel-type connections are analyzed and verified. Preliminary conclusions can be drawn, with the constraint on screw weakens, the rope-effect increases. The design method can provide guidance for engineering requirement.