We study experimentally the dynamics of vertical-cavity surface-emitting lasers (VCSELs) with polarization-rotated (PR) optical feedback, such that the natural lasing polarization of a VCSEL is rotated by 90 deg and then is reinjected into the laser. We observe noisy, square-wave-like polarization switchings with periodicity slightly longer than twice the delay time, which degrade to (or alternate with) bursts of irregular oscillations. We present results of simulations that are in good agreement with the observations. The simulations demonstrate that close to threshold the regular switching is very sensitive to noise, while well above threshold is less affected by the noise strength. The frequency splitting between the two polarizations plays a key role in the switching regularity, and we identify wide parameter regions where deterministic and robust switching can be observed.