A new optical security system for image encryption based on a nonlinear joint transform correlator (JTC) in the Fresnel domain (FrD) is proposed. The proposal of the encryption process is a lensless optical system that produces a real encrypted image and is a simplified version of some previous JTC-based encryption systems. We use a random complex mask as the key in the nonlinear system for the purpose of increasing the security of the encrypted image. In order to retrieve the primary image in the decryption process, a nonlinear operation has to be introduced in the encrypted function. The optical decryption process is implemented through the Fresnel transform and the fractional Fourier transform. The security system proposed in this paper preserves the shift-invariance property of the JTC-based encryption system in the Fourier domain, with respect to the lateral displacement of the key random mask in the decryption process. This system shows an improved resistance to chosen-plaintext and known-plaintext attacks, as they have been proposed in the cryptanalysis of the JTC encrypting system. Numerical simulations show the validity of this new optical security system. (C) 2014 Optical Society of America