A new optical security system for image encryption based on a fractional joint transform correlator and nonlinear filtering is proposed. The position of the lens in the proposed optical encryption setup can be chosen, so that an additional key is introduced in the security system. The distributions at the input and output planes of the encryption system are related by a fractional Fourier transform (FrFT) at a given fractional order that is defined, among other parameters, by the focal length and the position of the lens in the setup; this fractional order acts as an additional key of the security system. The optical intensity of the complex distribution in the fractional Fourier domain (output plane) is captured by a CCD camera. The nonlinearity introduced in the last step of the encryption process, maintains the encrypted function as a real-valued function. The security system proposed in this work is an generalization of the encryption system based on a conventional joint transform correlator (JTC), from the Fourier domain to the fractional Fourier domain. Additional advantages of the proposed system are: new degrees of freedom for the optical setup, alleviated alignment requirement, and the introduction of an additional key (the fractional order of the FrFT) that improves security. Numerical simulations verify the validity of this new optical security system