Are there constraints imposed on the optimization? If yes, then check that the constraints do not break the symmetry. ADF automatically preserves symmetry when the starting structure is symmetric and no Symmetry NOSYM has been specified. This symmetry preservation may conflict with constraints if the constraints are not specified according to symmetry. For example, if two interatomic distances are symmetry-equivalent and you freeze one of them, then you should also freeze the other explicitly.
Optimization in Cartesian coordinates usually needs more steps to converge compared to delocalized. If you haven't tried delocalized optimization for the system then you should do it.
Near 180-degree angles with optimization in delocalized coordinates. ADF normally does not have a problem with a near-180-degree valence angle if the initial value of the angle was larger than 175 degrees or if it is a terminal bond angle. If the initial angle was larger than 175 degrees then it gets a special treatment. For example, any torsion angle that contains the three atoms is removed or replaced with a torsion angle involving only the two end atoms of the near-180-degrees angle. If the angle defines a terminal bond then two angles in two coordinate planes are used to define the bond instead of a valence and a torsion angle. However, if the initial value of the angle was far from 180 degrees and has become close to it during optimization then optimization may become unstable, especially if this is an angle connecting large fragments. In this case, it may be useful to restart geometry optimization from the latest geometry. As a last resort, you may want to constrain the angle to a value close, but not equal, to 180 degrees.