We propose a generalization of the encryption system based on double random phase encoding (DRPE) and a joint transform correlator (JTC), from the Fourier domain to the fractional Fourier domain (FrFD) by using the fractional Fourier operators, such as the fractional Fourier transform (FrFT), fractional traslation, fractional convolution and fractional correlation. Image encryption systems based on a JTC architecture in the FrFD usually produce low quality decrypted images. In this work, we present two approaches to improve the quality of the decrypted images, which are based on nonlinear processing applied to the encrypted function (that contains the joint fractional power spectrum, JFPS) and the nonzero-order JTC in the FrFD. When the two approaches are combined, the quality of the decrypted image is higher. In addition to the advantages introduced by the implementation of the DRPE using a JTC, we demonstrate that the proposed encryption system in the FrFD preserves the shift-invariance property of the JTC-based encryption system in the Fourier domain, with respect to the lateral displacement of both the key random mask in the decryption process and the retrieval of the primary image. The feasibility of this encryption system is verified and analyzed by computer simulations.