To detail out,
My understanding of multiplicative decomposition of deformation gradient F =FeFp (see the attached figure) is
Fp indicates the shape change without inducing any stress
Fe leads to elastic stretching and rotation
Say, If I load a single crystal and unload it, we will have a residual plastic strain present. As, we removed the load, the effect of Fe would have been gone. So all the elastic effects vanish.