Large strain viscoelasticity - help with CETOL?

I am having problems with large strain viscoelasticity, where I am using Prony series to model the visco part. There is no default for the parameter CETOL, nevertheless, by reducing this tolerance or reducing the time step I am getting results that make no sense.

In my understanding, I solve the problem. Then I want to check that my results are correct, so I do convergence analysis. I refine the time step, I refine the mesh and I reduce the tolerances. Then if everything is defined properly, my solution should be better with these refinements. But the case with large strain viscoelasticity is not improving with these refinements, results are getting worse (I can compare simple cases to analytical solution). Thus my question is, how should I select CETOL and why my results are not better for tighter tolerance. Can anyone help me with this? I had this problem in ANSYS, I switched to ABAQUS but it is the same.